Taming generative video models for zero-shot optical flow extraction
Authors: Seungwoo Kim
Khai Loong Aw
Klemen Kotar
Cristobal Eyzaguirre
Wanhee Lee
Yunong Liu
Jared Watrous
Stefan Stojanov
Juan Carlos Niebles
Jiajun Wu
Daniel LK Yamins
Venue: NeurIPS 2025 poster ()
Keywords: optical flow
generative models
Primary area: deep_learning

Abstract: Extracting optical flow from videos remains a core computer vision problem. Motivated by the recent success of large general-purpose models, we ask whether frozen self-supervised video models trained only to predict future frames can be prompted, without fine-tuning, to output flow. Prior attempts to read out depth or illumination from video generators required fine-tuning; that strategy is ill-suited for flow, where labeled data is scarce and synthetic datasets suffer from a sim-to-real gap. Inspired by the Counterfactual World Model (CWM) paradigm, which can obtain point-wise correspondences by injecting a small tracer perturbation into a next-frame predictor and tracking its propagation, we extend this idea to generative video models for zero-shot flow extraction. We explore several popular architectures and find that successful zero-shot flow extraction in this manner is aided by three model properties: (1) distributional prediction of future frames (avoiding blurry or noisy outputs); (2) factorized latents that treat each spatio-temporal patch independently; and (3) random-access decoding that can condition on any subset of future pixels. These properties are uniquely present in the recently introduced Local Random Access Sequence (LRAS) architecture. Building on LRAS, we propose KL-tracing: a novel test-time inference procedure that injects a localized perturbation into the first frame, rolls out the model one step, and computes the Kullback–Leibler divergence between perturbed and unperturbed predictive distributions. Without any flow-specific fine-tuning, our method is competitive with state-of-the-art, task-specific models on the real-world TAP-Vid DAVIS benchmark and the synthetic TAP-Vid Kubric. Our results show that counterfactual prompting of controllable generative video models is an effective alternative to supervised or photometric-loss methods for high-quality flow.

Taming generative video models for zero-shot optical flow extraction
Seungwoo Kim'
Khai Loong Aw*
Klemen Kotar*
Cristobal Eyzaguirre
Wanhee Lee
Yunong Liu
Jared Watrous
Stefan Stojanov
Juan Carlos Niebles
Jiajun Wu
Daniel L.K. Yamins
Stanford University
eXXa
DoDe
Figure We introduce a zero-shot test-time inference procedure called KL-tracing, which extracts robust optical flow and point tracking from generative world model on challenging in-the-wild videos. In every column, the green line links the query location in the first frame (top) to the position predicted by our method in the second frame (bottom). All clips are real-world internet videos and contain phenomena that classical, appearance-basee optical flow methods find challenging: (A) Newton's 's cradle, where both frames have four balls the middle, but the balls are different; the example involves physical reasoning. (B) Globe has challenging in-place object rotation and the query point is in the textureless ocean. (C) Dog weaving through occluding poles with large, rapid motion, including depth changes and motion blur. (D) Soccer tackle with fast, diagonal motion with motion blur and partial occlusion. (E) Windmill rotation where the repetitive blades and uniform sky make local matching challenging. These examples highlight the benefits of leveraging a powerful world model to extract optical flow for challenging real-world scene dynamics.
Abstract
Extracting optical flow from videos remains core computer vision problem. Motivated by the recent success of large general-purpose models, we whether frozen self-supervised video models trained only to predict future frames be prompted, without fine-tuning, to output flow. Prior attempts to read out depth or illumination from video generators required fine-tuning; that strategy is ill-suited for flow, where labeled data is scarce and synthetic datasets suffer from a sim-to- real gap. Inspired by the Counterfactual World Model (CWM) paradigm, which can obtain point-wise correspondences by injecting a small tracer perturbation into next-frame predictor and tracking its propagation, generative video models for zero-shot flow extraction. We explore several popular architectures and find that successful zero-shot flow extraction in this manner is
extend this idea
"Equal contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

aided by three model properties: (1) distributional prediction of future frames (avoiding blurry or noisy outputs): (2) factorized latents that treat each spatio- temporal patch independentlyy and (3) random-access decoding that can condition on any subset of future pixels. These properties are uniquely present in the recently introduced Local Random Access Sequence (LRAS) architecture. Building on LRAS, we propose KL-tracing: a novel test-time inference procedure that injects a localized perturbation into the first frame, rolls out the model one step, and computes the Kullback-Leibler divergence between perturbed and unperturbed predictive distributions. Without any flow-specific competitive with state-of-the-arg task-specific models on the real-world TAP- Vid DAVIS benchmark and the synthetic TAP- Vid Kubric. Our results show that counterfactual prompting of controllable generative video models is an effective alternative supervised photometric-loss methods for high-quality flow.
method is
Introduction
Extracting motion information (optical flow) from videos is a fundamental yet open challenge in computer vision with many applications. Due to the intractable cost of obtaining ground-truti labels from real-world videos, most supervised baselines [39 41] are trained on synthetic datasets, e.g., FhyingChairs [12,, FlyingThings (28) and Sintel [6). While invaluable for progress, these datasets cover a narrow slice of motion statistics- prreiisssperdmmiimntlyyy rigid objects, limited lighting variation, and short temporal horizons. As result, they under-represent the long-tail of real-world phenomena such as non-rigid deformation, atmospheric effects, rapid camera shake and textureless regions. Self-supervised models (22 37 that can be trained real videos attempt bridge this gap, but they often rely on task-specific heuristics, such as photometric consistency or smoothness which fail under complex lighting. occlusion, or long-range motion. Consequently, both supervised and self-supervised optical flow baselines struggle to generalize to challenging in-the-wild videos.
Inspired by successes across vision and language where large general- purpose models outperform smaller task-specific ones, we explore large-scale video models as a possible solution. Trained on massive repositories of real-world data, modern vidco models already demonstrate strong scene
understanding 5 30 277. suggesting an implicit grasp of optical flow 3 40 However, prior work extracting visual intermediates such depth and illumination from these models still required supervised fine-tuning S4 Extending that recipe to optical flow would again depend on synthetic labels, facing the same sim-to-real domain gap. This motivates our search for zero-shot procedure that can extract accurate optical flow from off-the-shelf video models without any additional training.
In fact, such a procedure has been proposed in the Counterfactual World Model (CWM) framework 3 Zero-shot optical flow is obtained by adding a small "tracer" perturbation to the source frame and tracking how a pretrained next-frame predictor propagates the tracer to the target frame (Section 3). In practice, however, because deterministic video models like CWM can only predict a single future state, it encourages predictions that average over future possibilities, yielding perceptually blurry frames that wash out the injected tracer, leading to less precise motion estimates [Section[4.]
To overcome this, we implement the perturb-and-ttn andhr--and-rrcckk method with generative video models, which generate crisp predictions as they sample from a distribution instead of regressing to a mean. This is insufficient, however, as each tate-of-thee-rt generative video model faces its own flow extraction challenges. Stable Video Diffusion (SVD) [5] produces photorealistic frames, but its conditioning is weak and -onlooclized: generation is guided by single global latent inverted from the target frame, so lixel-level edits introduce noisy differences, corrupting the extracted flow (Section 4.2). The recent Cosmos model [30] can condition generation on ground-truth patches from the target frame, but its autoregressive rollout is strictly in raster order, so these provided patches must all lie in the raster left region of the image, far less informative than the same number of patches distributed randomly across the image, thus failing to accurately reconstruct the target frame (Section
Through a systematic analysis of state-of-the-art video models, we find three key properties for accu- rate zero-shot optical flow extraction: (1) distributional prediction of future frames (avoiding blurry or noisy outputs); (2) local tokenization, which encodes each spatio-temporal patch independently; and
Project website at: httpe://hoouooilld..ghhhh..erree

---

(3) random access decoding that can condition on any subset of future pixels. These are present in the recent Local Random Access Sequence (LRAS) architecture [25), whose patch-level conditioning and random-order decoding provide much more fine-grained controllability than previous generative video models (Section (4.4). In addition, we propose a new method, Kl.-tracing. that further improves upon the probe-and-track method by using the autoregressive nature of LRAS which exposes the probability distributions for predictions. KL-tracing computes the perturbation difference in the prediction logit space, thereby bypassing noisy RGB differences resulting from sampling randomness. We achieve state-of-the results on the challenging real-world TAP-Vid DAVIS dataset and the synthetic TAP-Vid Kubric dataset relative to supervised and unsupervised flow baselines (Section (
Overall, we show how to tame generative models models-unrlly hard to control, and normally only steered with previous frames or coarse text prompts -nntptts-tnto- and Kl--tracng for extraction of precise optical flow. Our contributions are threefold: (1) We provide the first systematic study of optical flow extraction from large generative video models, highlighting failure modes of both deterministic regressors and underconstrained generative models, and discover key model properties for flow extraction; (2) we introduce KL-tracing, simple yet effective test-time inference flow procedure for tracing perturbations through the predicted distributions of future states; and (3) we tame LRAS with KL-tracing to obtain both quantitative and qualitative results of accurate flow traces on both real-world and synthetic benchmarks, as well as in-the-wild videos.
into a tool that obeys fine- tine-grinned patch conditioning
2 Related Work
Optical Flow refers to estimating the per-pixel 2D motion between pair of video frames. Most models (39 [4] are supervised on synthetic video datasets [12 28 GI, posing sim-to-real gap. i.e., predominantly rigid objects, limited lighting variation, and short temporal horizons, failing to capture the long tail of real-world phenomena such as non-rigid deformation, atmospheric effects, fast camera shakes, and fine-scale textureless regions. Alternatively, self-supervised models [23] 26 37 train on real-world videos, but often rely on task-specific architectures and heuristics such as photometric
consistency and smoothness loss [23 26 37 Regardless, these optical flow methods are designed for short frame gaps and relatively non-complex scenes with little motion dynamics.
Point Tracking follows set of points across longer time horizons. Most methods supervised 17 24], often on synthetic datasets, facing the sim-to-real gap. Alternatively, self-supervised methods rely task-specific heuristics such smoothness losses and cycle consistency [21 (5),
ie., tracking a point forward in time, then backward, should return to its original position. In contrast, use zero-shot method to extract flow from general purposs self-supervvsed video models trained on diverse, large-scale datasets, as they better learn challenging real-world dynamics (Section (6) Further, the best performing trackers on long horizon benchmarks often exploit multi-frame context, which can obscure a model's ability to resolve core frame-to-frand dynamics across variable timesteps. Therefore, we compare our zero-shot method primarily against baselines following the traditional, two-frame setting for flow.
Deterministic Video Models predict single future frame/latent [1) 16 3 2). Trained with either latent or pixel / reconstruction losses, they are implicitly optimized to output the expectation over plausible futures. Whenever the conditioning signal-a a single frame Fi. multiple frames (F-N.. F1), or a partially masked target F Emmasked can lead to multiple future possibilities, the model averages them, producing spatially blurred predictions which hurt flow estimation.
Generative Video Models. Generative models can sample different image or video generations from a probabilistic distribution. Diffusion models iteratively denoise single noise sample into generation
19 36 33 20 Autoregressive models [31 7 32 S generate outputs sequentially, modeling the joint distribution over pixels or tokens, cach element conditioned
previously generated elements. However, many of these models, c.g., Cosmos (30,, predict tokens in raster order and use a global tokenizer where each image patch is encoded with information from other patches. Hence, they lack fine-grained controllability which makes them challenging to use for flow extraction (Section 4.3) In contrast, recent class of models, Local Random Access Sequence (LRAS), has several key properties which make them suitable for optical flow extraction (Section[4.4) Applying our novel test-time inference procedure to LRAS achieves state-of-the-art results (Section 6).
Tracks from Diffusion Models. Related to our work, there has been recent interest in using the emergent motion understanding capabilities of generative video models. The focus of these
3

---

Clean prediction
frozen
Frame 2
video model
mask patches
invert noise
argmna
perturb
prediction
frazen
video model
Estimanda
Figure Test-time inference procedure for extracting flow from a pre-trained, frozen, generative video model, based on the Counterfactual World Model (CWM) paper [3]. This involves three steps: (1) Perturbation: add a small, white-colored 2D Gaussian dot perturbation to frame at the location of the point wish to track. (2) Generate model predictions conditioned on the two frames. For CWM, Cosmos, and LRAS, we provide frame I and masked patches of frame 2 (Sections 4. For Stable Video Diffusion, we provide the noised latents of both frames (Section 4.2) Estimate optical flow by computing the RGB difference between the clean and perturbed predictions.
investigations has primarily been on reading out motion information from intermediate representations of diffusion models 38 In this work, we instead rely on a different extraction procedure (namely, propagating intervention points for tracking), which we find to be more transferable to generative architectures beyond diffusion. Our approach, when paired with design decisions that maximize controllability and locality, outperforms the latest diffusion-specific methods for flow (Section
3 timt-time inference procedure and evaluation setup for optical flow
We use zero-shot procedure to extract optical flow from various pre-trained video models. For all models, we utilize the generic method template illustrated Figure based on the Counterfactual World Model (CWM) paper [3]. Step 1. Inject small perturbation. We duplicate the initial frame Fi and perturb it with small "white bump" to form Fi, i.e., Gaussian centered the query location Tg with amplitudes 255 on each RGB channel and standard deviation a equal to 2.0. Step 2. Run model twice. Both clean and perturbed initial frames frozen model, each time with a sparse mask of the second frame (H-mmakked) ). Hence, two forward passes: (F1, Fmaaskedd + F.fred) and (Fi, (Fissssekee + Fsprdd, =Fgrod) ). Step 3. Estimate optical flow. Compute the RGB difference between the predictions, porod and Foredd Then, take the arg-mas identify the location to which the perturbation was carried.
separately forwarded through the
We find that this procedure can be extended any video model that exposes this interface of next- frame prediction. For example, this works with models that allow providing a subset or masking of and also video diffusion models, by converting frames I and 2 into
patches (Sections noised latents and using them as conditioning (Section
Evaluation setup. We use TiP-Vid DAVIS 19 and Kubric [33 for evaluation. TiP-Vid DAVIS contains real-worid videos with numan-annotated flow and occlusion labels, while Kubric synthetic dataset with ground-truth -ruth labels. Metrics. We use the following metrics. (1) Average Distance (AD) (or endpoint error (EPE)), the Euclidean distance between the predicted and ground- truth flow, as well as metrics from TAP- Vid ( (2) Average Jaccard (AJ), the "true positives" divided by "true positives" plus "false positive" plus "false negatives", averaged over various thresholds; (3) < 65 the fraction of points that are within a threshold of the ground truth location, averaged over various thresholds, and (4) Occlusion Accuracy (OA) for predicting occluded points.
4 Model evaluations
We conduct several studies extracting optical flow from various video model classes, highlighting issues we found with each. We investigate deterministic models (Section 4.1), diffusion models

---

and autoregressive models (Section 4.3 Finally, we identify the key properties of generative models that aid precise flow prediction, and a model class that satisfies them (Section 4.4)
(Section 4.2
4.1 Study 1: Deterministic models produce blurry-averaged predictions.
Method: Deterministic video models such as the Counterfactual World Model (CWM) 3 40 predict
a single future state. CWM a visual foundation model that can be zero-shot prompted to perform many tasks, such predicting flow, keypoints, object segments, counterfactuals, and depth. CWM was trained with an l loss to perform masked next-frame prediction: (Fi, Fquasked + Fg It deterministically predicts a single future state F.pre as lacks a mechanism for sampling. To extract flow from CWM, we use the procedure in Section 3 with a small, red perturbation, following 3
Findings: Deterministic video models such CWM regress to the mean future state, blurring predictions. Deterministic models, such as the Counterfactual World Model (CWM), often produce blurry predictions as they model single, average future state (Figure A This hurts optical flow extraction in two ways. (1) At the location where the perturbation is carried to, the perturbation is less visible due to blurriness from prediction uncertainty. (2) At locations where the perturbation is not expected to propagate, blurriness manifests as minor RGB differences. The clean and perturbed predictions are different in these locations due to the added perturbation in the first frame (Fi). Combined, these introduce errors when using arg-max max to select the destination for the query point. To address the issue of blurry predictions from CWM, we next explore extracting flow from Stable Video Diffusion (SVD), a diffusion model that produces sharp predictions.
4.2 Study 2: Diffusion models' global latent code lacks sufficient fine-grained controllability
Method: Applying the method above (Section 3 Figure to Stable Video Diffusion [5] is challenging as generation operates on full-frame latents rather than pixels, so these models do not natively support partial frames as input. However, we need to provide information from the second frame to anchor the generation, otherwise the model will hallucinate arbitrary future frames unsuitable for flow estimation. Below, we describe how we adapt the flow extraction procedure to the latent diffusion setting, while keeping the generations anchored to the observed video. Step 1. Frame selection and perturbation. As SVD is trained with a fixed framerate and more than two frames, we first retrieve an interpolated sequence of intermediate frames bridging the query (Fi) and target (Fx) frames F= (Fi, F2...., F.). We add dot perturbation to F1, obtaining Fi. Step 2. Latent inversion and paired generation. To anchor the sampling trajectory to the original video, we apply a latent inversion technique introduced in VideoShop [10). Inversion addresses the drift in generation by finding an initial noise vector that reconstructs the input video when fed into the diffusion process. Specifically, we first obtain the latents for every frame l = {ll, la,... .......... EN } and then partially apply the inversion process, adding noise to the latents up to a fraction of the total noising steps. This is done using SVD's UNet and an inverted scheduler, resulting in noisy latents & = {l...... [N From these latents we perform two denoising passes with identical hyperparameters, but with different
conditioning frames, F1 and the perturbed Fi, obtaining two denoised latents gored and Z goed which can be decoded to produce the clean and perturbed reconstruction _Frred Step 3. Estimate optical flow (same as in Section 33. We compute the RGB difference between the clean and perturbed generations to localize the perturbation. We compare two generations, rather than the perturbed generation to the original input, in order to suppress noise from imperfect inversion and VAE artifacts, improving the accuracy and robustness of flow predictions even under lossy latent compression.
Stable Video Diffusion (SVD) lacks fine-grained controllability as it generates each frame by globally denoising coarse latent code. Thus, local perturbations in pixel space are washed out or remapped unpredictably during sampling. This stochastic, scene-level regeneration pre- vents deterministic, point-wise correspondences. As result, its clean and perturbed predictions differ in locations where the perturbation is not supposed to be carried to, resulting in noisy differences.
Findings:
Attempting to address the precision issues arising from coarse global latent, the recent DiffTrack 29 uses a one-to-one frame-to-latent mapping to avoid temporal compression. It uses query-key matching in the 3D attention blocks across frames to perform point tracking. Though this method demonstrates that strong temporal correspondence can be found in diffusion model representations, with significant improvement over existing diffusion- based methods such as DIFT [38), it still lags
5

---

behind specialized optical flow models, thus failing to close the gap between veneral-purpose video models and task-specific baselines (Table Next, we explore autoregressive models which allow us to provide actual patches of the second frame as more fine-grained conditioning.
4.3 Study 3: Raster-order autoregressive models struggle with partial-frame conditioning
Method: Cosmos (30 includes both a diffusion-basse and an autoregressive foundation model. We evaluate flow extractions from the autoregressive model which allows us to provide the actual frame patches directly as conditioning. We use the Cosmos autoregressive model with 4B parameters, which comes with 7B diffusion decoder for generating images. As the Cosmos autoregressive model does not have pointer tokens, autoregressive rollout predictions for image tokens cannot be performed in random-access order. Instead, they can only be made in raster order, i.e., from top left to bottom right of the image. Nevertheless, we make a best-effort attempt to extract flow by trying multiple settings.
Setring A. We provide the first frame and the top 10% raster tokens of the second frame: clean (Fi, E Inster F. and perturbed F2 Ea raser - F
Setting B. We provide the first frame and overwrite a random 10% of the predicted tokens dur-
ing the model's autoregressive rollout with ground truth tokens of the second frame: clean (Fi, (
F and perturbed (Fi, 12 poverwiie-rrnnoo--Uus
fipred)
Setting C. We provide both frames fully: clean (Fi, + F-frad) and perturbed (F1, F2 + Frrat)
Findings: All Cosmos evaluation settings perform poorly (Figure and Table 2
Serting A. The model prediction effectively repeats the first frame, unable propagate the perturba- tion (Figure 7A). This is because the top 10% raster patches reveal less about the second frame than random 10% patches. At the same time, revealing more patches will hurt performance; if the perturbation should be carried to a revealed patch, the model will not generate the perturbation. Setting B. This also results in the model prediction repeating the first frame, because the model gen- erates many new patches with less than 10% ground truth patches. This is much more challenging than getting all the revealed ground truth patches at the start.
Setting C. The model prediction correctly matches the target frame but does not contain the perturbation. We evaluate this setting as the model has a diffusion decoder that has the potential to (but unfortunately does not) reproduce the perturbation in Aigeed
Cosmos flow extraction is challenging due to the lack of non-local tokenizer and pointer tokens for random-access decoding order. Therefore, we next explore LRAS, which has these properties.
4.4 Study 4: Key model properties for flow extraction are present in LRAS
and we discover that flow extraction can be improved by generative models having three key properties (Table I): (1) distributional prediction of future frames (avoiding blurry or noisy outputs; which CWM Tacks); (2) local tokenizer that treats each spatio-temporal patch independently (which SVD and Cosmos lack); and (3) random-access decoding order that allows the model to condition any subset of the second frame patches (which SVD and Cosmos lack). We identify Local Random Access Sequence (LRAS), recent family of autoregressive visual foundation world models that has these properties and can be zero-shot prompted to perform many tasks, such as 3D object manipulation, novel view synthesis, and depth estimation. use its autoregressive generative video world model with 7 billion (7B) parameters.
Method: From Sections
Findings: LRAS model produces clean and perturbed predictions with minimal variations due to sampling randomness. However, these small variations result in slightly noisy difference maps, harming flow extraction (Figure 5). We address this with the KL-traccin procedure below.
KL-tracing bypasses sampling randomness
We design novel test-time inference procedure, KL-tracing, for optical flow extraction from models that predict a probability distribution at every patch of the target frame (Figure 3 KL-tracing is the same as the RGB flow extraction method in Section except for the third step. Step 3. Estimate optical flow with patchwise KL--dvergence. We use the patch-wise logits for
Method:
the clean (F=("e)) and perturbed (Figee)) predictions: {z.}} and ) (Figure 3). For every patch

---

preciction topts
verpance
poed
frem
Nesked Franez
peraro
Perturbed Trame
Perturbed pred logit
Perturted
treen
Fatimang
Figure 3: KL-tracing, our novel yet simple test-time inference procedure for extracting optical flow from controllable generative models such as LRAS. We follow the same steps for perturbation and conditioned prediction in Figure 2 but estimate optical flow by computing the KL divergence between the clean and perturbed prediction logits.
(i,j) we compute the KL-divereence: (i,(i,)) KL (2.,) (Z pet) ). The resulting KI divergence map should peak at the patch where the perturbation is carried to, giving crisp optical flow estimate (Figure 3). We take an arg-ma arg-mxx to identify the patch with the highest KL divergence to estimate optical flow. To detect if point is occluded, we simply use threshold on the KL divergence value.
(B) Stable Video Diffusion
(A) CWM
(C) Cosmos
(D) KL-LRAS
Clean prediction
Frame 1
Perturbed prediction
Perturbed Frame
Frame 2
Difference
Figure Our method, KL.ttrccing using LRAS extracts better flow than other generative video models. (A) Deterministic models, such CWM [3] often produce blurry predictions they model single, average future state. (B) Stable Video Diffusion lacks fine-grained controllability due to its coarse global latent code. Its clean and perturbed predictions differ in locations where the perturbation not supposed to be carried to. (C) The Cosmos autoregressive world model lacks fine-grained controllability
does not utilize pointers denote the position of each token, making challenging to prompt for flow extraction. (D) The LRAS model highly controllable and has minimal differences between the clean and perturbed predictions. We use KL-tracing to compute the difference logit instead RGB space, obtaining sharp flow extractions.
Findings: KL divergence of prediction logits bypasses noisy differences resulting from sampling randomness. Fach RGB generation sample drawn from the prediction distribution. Instead of sampling multiple RGB generations and averaging them to suppress the noisy differences resulting
7

---

Perturbed
Frame
Frame
RGB DIF
P3B DIT
RGB DIT N
Aug. RGB CHf
Single KL Off
Figure 5: KL-divergence of prediction distributions bypasses noisy RGB differences resulting from sampling randomness. Computing the KL divergence of the clean and perturbed prediction logits (last column) more efficient yet functionally similar computing the average RGB difference over many samples (second last column).
from sampling randomness, it is more efficient to directly use the predicted distribution. This is an important benefit autoregressive models as they directly expose the probability distribution. Empirically, observe two benefits. First, there minor differences between the clean and perturbed predictions at locations where the perturbation is not expected to propagate, because adding the perturbation affects the rest of the image due to the transformer's global attention. KI- tracing empirically results in smaller noisy differences than computing the RGB difference. Second, occasionally the added perturbation does not visually get carried over in the perturbed RGB prediction, but it still appears as elevated uncertainty in logit space, detected by KL-tracing.
KL-LRAS (ours)
SMURF
SEA-RAFT
Doduo
KI-LRAS
Jours)
SMURF
SEA-RAFT
Doduo
ODOO
2008
Figure 6: Large world models such as LRAS capture aspects of real-world dynamics that are challenging for specialized flow models relying on visual similarity or photometric loss. Specialized optical flow baselines, both supervised and self-supervised, struggle on complex, real- world scenes that are not fully captured by their training heuristics (Section(6). A deeper understanding of dynamics beyond feature/photometric similarity allows large world models such as LRAS to resolve motion the presence of homogeneous objects, motion blur, and partial occlusion, among others.

---

Large video world models capture aspects of real-world dynamics that are challenging for specialized flow models
Method: We compare KI.-tracing with supervised flow methods such as RAFT 339 and SEA-RAFT
[41], as well as unsupervised methods such as Doduo [22] and SMURF 37 (Table 3 Figure 6
We also compare our method to alternative branch of approaches, multi-frame trackers, which lypically use multiple frames of information to generate point tracks across time (Appendix C
Findings: Powerful video models capture challenging aspects of real-world dynamics. For example, a globe undergoes in-place rotation with the query point on a textureless surface (Figure D. or physical motion examples such as Newton's 's cradle (Figure O These real-world dynamics are challenging for specialized flow models which are small and not trained on diverse, large-scale video datasets. Video world models use generic training objectives, so they do not suffer from failure modes introduced by training heuristics used by specialized flow models. Supervised methods are trained synthetic datasets labeling real-world videos is expensive, and hence face sim-to-real gap where they do not observe challenging real-world dynamics. Self-supervised methods such Doduo [22] and SMURF 37 use photometric loss for training, which results in challenges in predicting flow for frame pairs with significant differences in light intensity, and also enforce strong global consistency and smoothness constraints, which introduce failure modes for complex non-uniform dynamics that contain various magnitudes of flow across the same frame pair.
CWM 3
SVD [5]
Cosmos 30
LRAS 25
Distributional prediction of future frames Non-global tokenizer Random-access decoding order
No
Yes
Yes
Yes
Yes
No
No
Yes
Yes
No
No
Yes
Table 1: Key properties of video models for precise flow extraction: (1) distributional prediction of future frames, thus avoiding blurry or noisy outputs, (2) non-global tokenizer that treats each spatio-temporal patch independently, (3) random-access decoding order that allows conditioning on any subset of second-frie fronnd-rammeeeeeo patches.
TAP-Vid Vid DAVIS Subset (3%)
Model
Endpoint Error (EPE)
LRAS RGB (5MM, 8MS, 2STD) (ours)
8,4797
LRAS KL (5MM, 8MS, 2STD) (ours)
5.0762
Stable Video Diffusion [S]
74.7990
Cosmos (top 10% raster) (5MM, 2STD, 512x512) 3D
35.4338
Cosmos (overwrite 10% during rollout) (SMM, 2STD, 512x512)
37.7552
Cosmos (provide full second frame) (5MM, 2STD, 512x512) 30
66.5521
Table 2: KL-tracing with LRAS beats other video models. MM multi-mash (average multiple random masks), MS = multi-scale (zoom into image at multiple scales).
7 Discussion
Taming generative world models with fine-grained controllability. A key finding our work is that zero-shot prompting many mainstream generative video models for visual property extraction can be challenging due to their lack of fine-grained controllability. However, a series of design features can help tame these models: (I) predicting distribution of future states rather than deterministic average, (2) encoding each spatio-temporal patch independently, and (3) having a mechanism to decode individual parts of frame in any given order. Currently, the LRAS [25] framework was the only one we identified that contained all of these properties, but they could be adopted by other generative models as well, increasing the ease with which they can be visually prompted.
Tight coupling with the base generative model. The main advantage of using generative model for flow is that it becomes easier to improve flow prediction quality on new domains. Any previously

---

DAVIS
Kubric
Method
AD
< <ong
OA f
AJ 1
OA 1
TAP-Vid First
S
RAFT 39
41.77
25.33
54.37
66.40
71.93
5.60
82.15
88.54
SEA-RAFT
43.41
20.18
58.69
66.34
75.06
6.54
84.63
89.50
W
Doduo
22
23.34
13.41
48.50
47.91
54.98
5.31
72.20
73.56
U
SMURF
30.64
27.28
44.18
59.15
65.81
6.81
80.57
87.91
DifkTrrak 29
46,90
CWM 40
15.00
23.53
26.30
76.63
28.77
11.64
41.63
84.93
LRAS with KL-tracing (ours)
44.16
11.18
65.20
74.58
65.49
5.06
81.66
87.63
Table 3: TAP-Vis Vid First:
quantitative results DAVIS and Kubric. Tracking starts when point first appears and continues to the video end, thus involving large frame gaps. LRAS with KL-tracing outperforms two-frame baselines. "S" supervised, "W" weakly supervised, "U" unsupervised.
unrepresented domain become "in-distribution" for KL-tracing, simply by extending the training data of the base generative model (e.g., LRAS). There is no need to obtain new, task-specific labels, which is particularly appealing for flow due to the intractable cost of human labeling for anything other than synthetic, computer-graphic type scenes. As modern video datasets continue to scale at rapidly increasing rate, KI.-tracing's performance
improve in lockstep.
Distillation. Compared with real-time dense flow baselines (39 or diffusion- based methods that build on feature correspondence [38 (9), KL-tracing can be slower due to the process of propagating interventions point-by-point, However, on challenging, real-world scenes, the quality of the extracted flow is notably better (Table In fact, can improve the Pareto frontier by taking the sparse labels that are extracted and distilling them into faster architecture such as SEA-RAFT, thereby using KL-tracing much more scalable alternative human labeling.
Future work. natural next step extending this test-time inference procedure extract other visual intermediates such object segments, material properties, motion affordances. By moving away from fine-tuning our world models on labeled datasets the vision community can experience a shift akin to the one seen with large language models, moving from rigid representations constructed by fine-tuning toward dynamic, task-specific and adaptable ones extracted on the fly.
paradigm of zero-shot prompting,
Overall, our results indicate that prompting controllable, ceff-supervised world models is scalable and effective alternative to supervised or photometric-los approaches for high-quality optical flow.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the following awards: Simons Foundation grant SFI-AN-NC-GB- Culminatiion-00009988---99 National Science Foundation CAREER grant 1844724, National Science Foundation Grant NCS-FR 2123963, National Science Foundation Grant RI 2211255 Office of Naval Research grant N00014-204 1-2--9 ONR MURI N00014-21-1-2800 ONR MURI N00014- 24-1-2748, and ONR MURI N00014-22- 1-224-- We also thank the Stanford HAI, Stanford Data Sciences, the Marlowe team, and the Google TPU Research Cloud team for their computing support.
References
[1] Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Mojtaba, Komeili, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, Sergio Arnaud, Abha Gejji, Ada Martin, Francois Robert Hogan, Daniel Dugas, Piotr Bojanowski, Vasil Khalidov, Patrick Labatut, Francisco Massa, Marc Szafraniec, Kapil Krishnakumar, Yong Li, Xiaodong Ma, Sarath Chandar, Franziska Meier, Yann LeCun, Michael Rabbat, and Nicolas Ballas. V-jepa 2: Self-supervised video models enable understanding, prediction and planning, 2025.
[2] Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting Feature Prediction for Learning Visual Representations from Video, February 2024. arXiv:2404.0844 [es].
[3] Daniel M. Bear, Kevin Feigelis, Honglin Chen, Wanhee Lee, Rahul Venkatesh, Klemen Kotar, Alex Durango, and Daniel L. K. Yamins. Unifying (Machine) Vision via Counterfactual World Modeling, June 2023. trXiv:2306.01822 [cs].
[4] Zhangxing Bian, Allan Jabri, Alexei A. Efros, and Andrew Owens. Learning Pixel Trajectories with Multiscale Contrastive Random Walks, April 2022. arXiv:2201.08379 [cs].
[5] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets, November 2023. arXiv:2311.15122 [cs].
[6] Daniel J. Butler, Jonas Wulff, Garrett B. Stanley, and Michael Black. A Naturalistic Open Source Movie for Optical Flow Evaluation. In David Hutchison, Takeo Kanade, Josef Kit- tler, Jon M. Kleinberg, Friedemann Mattern, John C. Mitchell, Moni Naor, Oscar Nierstrasz, C. Pandu Rangan, Bernhard Steffen, Madhu Sudan, Demetri Terzopoulos, Doug Tygar. Moshe Y. Vardi, Gerhard Weikum, Andrew Fitzgibbon, Svetlana Lazebnik. Pietro Perona, Yoichi Sato, and Cordelia Schmid, editors, Computer Vision ECCV 20/2, volume 7577, pages 61 1-625.. Springer Berlin Heidelberg, Berlin, Heidelberg, 2012. Series Title: Lecture Notes Computer Science.
[7] Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. PixeISNAIL: An Improved Autoregressive Generative Model, December 2017. arXiv:1712.09766 [es].
[8] Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial Video Generation on Complex Datasets, September 2019. arXiv:1907.000 006777 [es].
[9] Carl Doersch, Ankush Gupta, Larisa Markeeva, Adrià Recasens, Lucas Smaira, Yusuf Aytar, João Carreira, Andrew Zisserman, and Yi Yang. TAP-Vid: Benchmark for Tracking Any Point in a Video, March 2023. 1tXiv:2211.03722 [es].
[10] Xiang Fan, Anand Bhattad, and Ranjay Krishna. Videoshop: Localized semantic video editing with noise-extrapolated diffusion inversion, 2024.
[11] Ye Fang, Zeyi Sun, Shangzhan Zhang, Tong Wu, Yinghao Xu, Pan Zhang, Jiaqi Wang, Gordon Wetzstein, and Dahua Lin. Relightvid: Temporal-consistent diffusion model for video relighting. arXiv preprint arXiv:2501.16338 2025.

---

[12] Philipp Fischer, Alexey Dosovitskiy, Eddy Ilg, Philip Häusser, Caner Hazurbay, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, and Thomas Brox. FlowNet: Learning Optical Flow with Convolutional Networks, May 2015. arXiv:1504.0683. [cs].
[13] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J. Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, Thomas Kipf, Abhijit Kundu, Dmitry Lagun. Issam Laradji, Hisueh-Ti, Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang. Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi. Kubric: A scalable dataset generator, March 2022. arXiv:2203.035500 [cs].
[14] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to Control: Learning Behaviors by Latent Imagination, March 2020. arXiv: 19120019 [es].
[15] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning Latent Dynamics for Planning from Pixels, June 2019. urXiv:1811.04555 [cs].
[16] Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering Atari with Discrete World Models, February 2022. arXiv:2010.02193 [es].
[17] Adam W. Harley, Zhaoyuan Fang, and Katerina Fragkiadaki. Particle Video Revisited: Tracking Through Occlusions Using Point Trajectories, July 2022. arXiv:2204.04153 [cs].
[18] Adam W. Harley, Yang You, Xinglong Sun, Yang Zheng, Nikhil Raghuraman, Yunqi Gu, Sheldon Liang, Wen-Hsuan Chu, Achal Dave, Pavel Tokmakov, Suya You, Rares Ambrus, Katerina Fragkiadaki, and Leonidas J. Guibas. Alltracker: Efficient dense point tracking at high resolution, 2025.
[19] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models, December 2020. arXiv:2006.1 1239 [cs].
[20] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video Diffusion Models, June 2022. arXiv:2204.03450 [cs].
[21] Allan Jabri, Andrew Owens, and Alexii A. Efros. Space-Time Correspondence as Contrastive Random Walk, December 2020. arXiv:2006.1461 [cs].
[22] Zhenyu Jiang, Hanwen Jiang, and Yuke Zhu. Doduo: Learning Dense Visual Correspondence from Unsupervised Semantic-Aware Flow, September 2023. arXiv:2309. 151 10 [es].
[23] Rico Jonschkowski, Austin Stone, Jonathan T. Barron, Ariel Gordon, Kurt Konolige, and Anelia Angelova. What Matters in Unsupervised Optical Flow, August 2020. wtXiv:2006.04900 [cs].
[24] Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. CoTracker: is Better Track Together, October 2024. arXiv:2307.07635 [cs].
[25] Wanhee Lee, Klemen Kotar, Rahul Mysore Venkatesh, Jared Watrous, Honglin Chen, Khai Loong Aw, and Daniel L. K. Yamins. 3D Scene Understanding Through Local Ran- dom Access Sequence Modeling, April 2025. arXiv:2504.03875 [cs].
[26] Liang Liu, Jiangning Zhang, Ruifei He, Yong Liu, Yabiao Wang, Ying Tai, Donghao Luo, Chengjie Wang, Jilin Li, and Feiyue Huang. Learning by Analogy: Reliable Supervision from Transformations for Unsupervised Optical Flow Estimation, November 2020. arXiv:2003.13045 [cs].
[27] Neclu Madan, Andreas Moegelmose, Rajat Modi, Yogesh S. Rawat, and Thomas B. Moeslund. Foundation models for video understanding: A survey, 2024.
[28] Nikolaus Mayer, Eddy 1lg, Philip Häusser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4040-4048, June 2016. 151.:15112.021444 [cs].
12

---

[29]
Jisu Nam, Soowon Son, Dahyun Chung, Jiyoung Kim, Siyoon Jin, Junhwa Hur, and Seungryong Kim. Emergent temporal correspondences from video diffusion transformers, 2025.
[30]
NVIDIA, Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, Daniel Dworakowski, Jiaojiao Fan, Michele Fenzi, Francesco Ferroni, Sanja Fidler, Dieter Fox, Songwei Ge, Yunhao Ge, Jinwei Gu, Siddharth Gururani, Ethan He, Jiahui Huang. Jacob Huffman, Pooya Jannaty, Jingyi Jin, Scung Wook Kim, Gergely Klár, Grace Lam, Shiyi Lan, Laura Leal-Taixe, Anqi Li, Zhaoshuo Li, Chen-Hsuan Lin, Tsung- Yi Lin, Huan Ling, Ming- Yu Liu, Xian Liu, Alice Luo, Qianli Ma, Hanzi Mao, Kaichun Mo, Arsalan Mousavian, Seungjun Nah, Sriharsha Niverty, David Page, Despoina Paschalidou, Zeeshan Patel, Lindsey Pavao, Morteza Ramezanali, Fitsum Reda, Xiaowei Ren, Vasanth Rao Naik Sabavat, Ed Schmerling, Stella Shi, Bartosz Stefaniak, Shitao Tang, Lyne Tchapmi, Przemek Tredak, Wei-Cheng Tseng. Jibin Varghese, Hao Wang, Haoxiang Wang. Heng Wang, Ting-Chun Wang, Fangyin Wei, Xinyue Wei, Jay Zhangjie Wu, Jiashu Xu, Wei Yang, Lin Yen-Chen, Xiaohui Zeng, Yu Zeng, Jing Zhang, Qinsheng Zhang, Yuxuan Zhang, Qingging Zhao, and Artur Zolkowski. Cosmos World Foundation Model Platform for Physical AI, March 2025. arXiv:2501.0333 [cs].
[31] Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu. Conditional Image Generation with PixeICNN Decoders, June 2016. arXiv:1606.05328 [es].
[32] Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating Diverse High-Fidelity Images with VQ-VAE-2, June 2019. arXiv:1906.0044: .00446 [cs].
[33] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding, May 2022. arXiv:2205. 11487 [cs].
[34] Jiahao Shao, Yuanbo Yang, Hongyu Zhou, Youmin Zhang, Yujun Shen, Matteo Poggi, and Yiyi Liao. Learning temporally consistent video depth from video diffusion priors. arXiv preprint arXiv:2406.014993 2024.
[35] Ayush Shrivastava and Andrew Owens. Self-Supervised Any-Point Tracking by Contrastive Random Walks, September 2024. arXiv:2409. 16288 [cs].
[36] Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-Based Generative Modeling through Stochastic Differential Equations, February 2021. arXiv:201 1. 1.146 [es].
[37] Austin Stone, Daniel Maurer, Alper Ayvaci, Anelia Angelova, and Rico Jonschkowski. SMURF: Self-Teaching Multi-Frame Unsupervised RAFT with Full-Image Warping, May 2021. arXiv:2105.0701 [cs].
[38] Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emer- gent correspondence from image diffusion, 2023.
[39] Zachary Teed and Jia Deng. RAFT: Recurrent All-Pairs Field Transforms for Optical Flow, August 2020. 1XXv:2003.12039 [cs].
[40] Rahul Venkatesh, Honglin Chen, Kevin Feigelis, Daniel M. Bear, Khaled Jedoui, Klemen Kotar, Felix Binder, Wanhee Lee, Sherry Liu, Kevin A. Smith, Judith E. Fan, and Daniel I. K. Yamins. Counterfactual World Modeling for Physical Dynamics Understanding, December 2023. arXiv:2312.06721 [cs].
[41] Yihan Wang, Lahav Lipson, and Jia Deng. SEA-RAFT: Simple, Efficient, Accurate RAFT for Optical Flow, May 2024. arXiv:2405. 14793 [cs].
13

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract claims are supported by the results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations discussed in discussion section.
14

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
No theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Methodology is detailed and allows for reproduction of experiments using open sourced models.
Guidelines:
The answer NA means that the paper does not include experiments.
15

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
All data used is already available openly, code will be released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ 7/nips. /ttppp.cc/pblic/guideeee http://oeeuuiiissooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
16

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
Method describes exactly how reproduce results.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report standard optical flow tracking metrics on widely used datasets containing many samples.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split. initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еттог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We discuss how much compute required to perform a single point extraction a multiple of the compute of forward pass of chosen model.
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics bictps://neuripee httprr/ss../uullle/ccccccceeeee
Answer: [Yes]
Justification:
Our research is about lower level vision and does not come near any ethics
issues.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: Since this paper is low level computer vision paper there is really not yet broad system for distributing it.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
18

---

Answer:
[NA]
Justification:
No such risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification:
No new assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No New Assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
19

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
No human subjects or crow sourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: No IRB necessary.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [No]
Justification: Paper about optical flow, no LLMs used at all.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our L.MM policy fbttps://neurii: ce. / for what should or should be described.
20

---

A
Additional details on methods
Models
Perturbation
Predictions based on conditioning
Compute difference
CWM
3
White Gaussian
Frame 1, Partial Frame
RGB difference
SVD [
White Gaussian
Noised latents of Frame and
RGB difference
Cosmos
30
White Gaussian
Frame 1, Partial Frame
RGB difference
LRAS, RGB
25
White Gaussian
Frame Partial Frame
RGB difference
LRAS, KL
White Gaussian
Frame 1, Partial Frame
KL divergence of logits
Table Flow extraction for all models follows the same three-step template: (I) Perturbation of
Frame with small, white-colored Gaussian bump, (2) Conditioning the model on frames and to generate clean and perturbed predictions, and (3) Compute the difference between the clean and perturbed predictions to extract optical flow.
B Additional results
B Overwrite Random 10%
(C) Provide full Frame 2
(A) Top 10% raster
Clean prediction
Frame
Perturbed prediction
Perturbed Frame
Difference
Frame
Figure 7: All evaluation settings for Cosmos (30) result in poor flow extractions. See Section 4.3 for a more detailed explanation of each result.
C Multi-Crame trackers
We are primarily interested in evaluating model's ability to resolve motion from two frames, with methods and baselines chosen accordingly. While multi-frame trackers often achieve improved results due to aggregating and refining predictions globally, the two-frame setting directly evaluates model's core dynamics understanding, especially in more challenging real-world cases. We compare method with supervised, multi-frame trackers run under the same two-frame constraint.
21

---

DAVIS
Method
AJ
OA f
TAP-Vid First
CoTracker-v3 24
39.9
58.0
76.9
AllTracker
18
54.9
66.0
78.0
LRAS with KL-tracing (ours)
44.2
65.2
74.6
Table 5: TAP-Vid DAVIS results for two-frame trackers. Tracking starts when a point first appears and continues to the video end (large frame gaps). The supervised tracker baselines are natively multi-frame, but are evaluated in a two-frame setting for direct comparison.
The performance of multi-frame ffrmme trackers is significantly impacted when removing context from intermediate frames. Notably, we find that AllTracker [18] is more robust, due to its approach of merging two-frame, flow-type correspondence with tracker-style global refinement across frames. Despite being completely training-free, our method shows comparable performance with the latest trackers in this challenging setting (Table 5).
22


Reviews:
[{"review_id": "Rvtt4COW2V", "reviewer": "Reviewer_fTmd", "summary": "This paper investigates the potential of extracting optical flow in a zero-shot manner from frozen, large generative video world models. Motivated by the limitations of prior supervised and photometric approaches, the authors systematically evaluate deterministic, diffusion, and autoregressive generative models. They show that successful flow extraction requires distributional prediction, factorized local latents, and random-access decoding. The paper introduces KL-tracing, a test-time inference procedure that measures the Kullback-Leibler divergence between clean and perturbed model predictions to robustly localize motion and extract flow without model fine-tuning. Experimental results on real-world (TAP-Vid DAVIS) and synthetic (TAP-Vid Kubric) datasets demonstrate that KL-tracing with LRAS surpasses both classical and state-of-the-art flow models, even though the world models were not trained with flow supervision.", "questions": "The idea is now new, as it has been proposed in papers like SiamMAE and other papers. The authors are encourage to do more experiments including both ablation and comparison with other SOTA methods.", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper presents detailed and comparative results across various generative world models (deterministic, diffusion, autoregressive, and LRAS), using standard real-world and synthetic benchmarks. The results, particularly in Table 2, demonstrate strong empirical performance, with KL-LRAS outperforming both supervised and unsupervised baselines in several key metrics.\n\nDespite strong performance on TAP-Vid, the paper does not empirically demonstrate zero-shot transfer to significantly out-of-distribution domains, such as medical, aerial, or night-time video. While the theoretical basis for generalization is sound, more experimental evidence or at least critical discussion would be helpful. In addition, although the performance is good as shown in the Figure, the paper failed to compare with other tracking methods like cotracker3. We encourage the authors to do more compareison with other SOTA optical flow or tracking methods.", "was_revised": true, "final_justification": "The author addressed all my concerns in the rebuttal, I inclined to accept the paper", "rebuttal": "We thank the reviewer for the thoughtful review and constructive suggestions! Overall, we are glad that reviewers found our method \"novel\" and \"interesting\" (uSVM) and our empirical results and analyses to be \"strong\" and \"detailed,\" and a significant improvement to existing methods (fTmd, uSVM, nvPE). We address the main points of the reviewer below, highlighting transferability and additional tracking baselines:\n\n> The paper does not empirically demonstrate zero-shot transfer to significantly out-of-distribution domains.\n\nThe reviewer brings up an interesting point of transferability and generalizability. We believe that this is demonstrated, at least to some extent, in our work both quantitatively and qualitatively. Although the base LRAS model is trained on real-world videos, KL-LRAS, with no additional training, performs strongly on both real (TAP-Vid DAVIS) *and synthetic* (TAP-Vid Kubric) benchmarks (Table 2), demonstrating strong cross-domain generalization within the larger range of \"normal\" videos containing people and objects. Further, qualitative results in Figures 1, 6, and the attached video demo also show robustness on challenging, long-tail motions (e.g., motion blur or in-place rotation).\n\nTo be honest, however, we don't know if a model trained on YouTube videos will transfer to very different kinds of videos where notions of object and timescale are extremely different (like medical data). Regardless, our approach, KL tracing, works with any base predictor that is effective at its underlying prediction task, whatever the domain may be. Thus—and this is the most important point—the main advantage of our approach is that we can easily extend our method to handle any new domain merely by extending the training data of the base predictor model of which our approach operates on top. We don't need to get new labels for that domain, bypassing the intractable cost of obtaining human labels for anything other than synthetic, carefully curated computer-graphic-type scenes. As foundation models improve across domains, our method's performance scales accordingly without requiring domain-specific flow annotations.\n\nWe will make this point clearer, and also include further qualitative examples of out-of-domain \"stress tests\" such as performance on night-time videos (please note that we are unable to submit additional demos as part of this rebuttal).\n\n> The paper failed to compare with other tracking methods like cotracker3\n\nWe agree that this reference is informative and therefore include results for CoTracker-v3.\n\n| TAP-Vid DAVIS                           | AJ $\\uparrow$  | AD $\\downarrow$ | $<\\delta^x_\\text{avg}\\uparrow$ | OA $\\uparrow$   |\n|:----------------------------------------|-------:|-------:|------------:|-------:|\n| CoTracker‑v3 [1]      | 39.85  | 19.04  | 57.96       | **76.87** |\n| KL‑LRAS (ours)                          | **44.16** | **11.18** | **65.20** | 74.58 |\n\n\nWe note that the above results show CoTracker-v3 run under the same **two-frame** setting on TAP-Vid DAVIS to ensure fair comparisons with the flow and generative model approaches. The lower performance of CoTracker in this more difficult setting shows that the underlying motion estimation still remains challenging for trackers. Following the reviewer's advice, we will also add additional point tracker and flow baselines to our paper, as well as express the above point more clearly.\n\n[1] Karaev et al., 2024. \"CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos.\""}, {"review_id": "00eBbUFiGR", "reviewer": "Reviewer_uSVM", "summary": "This paper proposes KL-tracing, a novel zero-shot inference approach for extracting optical flow from pretrained generative world models. Inspired by the concept of counterfactual world models (CWMs), the authors introduce a perturb-and-track procedure based on KL-divergence in latent spaces, tracing subtle perturbations through predictions of future frames. Evaluations conducted on the challenging TAP-Vid DAVIS and Kubric benchmarks demonstrate convincing performance improvements, especially when using the LRAS generative model.", "questions": "1. Could the authors discuss more concretely what would be required to make KL-tracing work with architectures beyond LRAS? The performance on cosmos and SVD is not strong.\n\n2. How sensitive is KL-tracing to the magnitude and shape of the perturbation? Would a different form (e.g., a low-frequency mask or structured noise) work similarly?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The idea in this paper is interesting, although it takes time to digest due to the writing clarity. The use of KL-divergence in latent distributions to trace perturbations and the perspective of counterfactual world models are not commonly seen in this direction. It feels novel and offers a fresh take on optical flow to me as a reviewer.\n\n2. The empirical results on the TAP-Vid benchmarks are good and convincing. \n\n3. Despite could-be-improved writing clarity, the authors do a commendable job walking through how the counterfactual idea applies to different types of generative models (e.g., LRAS, Stable Video Diffusion, Cosmos). As a reviewer, I appreciate the effort to bridge the conceptual gap across these architectures.\n\n\nWeakness:\n1. The best performing setup relies on a specific model (LRAS), which somewhat limits the broader applicability of the method. The authors provide analysis for SVD and Cosmos, but those results are weak. I believe the evaluations of these two models should also be included in Table 2 for completeness, as that table is the clearest comparison with traditional flow baselines.\n\n2. The writing could be greatly improved. The introduction assumes the reader is already familiar with Counterfactual World Models, which may not be true for everyone. Also, the paper emphasizes LRAS architecture too heavily, despite it not being the core contribution here. Lastly, I’d suggest reorganizing the method sections more cleanly—there’s noticeable redundancy (e.g., L117–L125 repeat in L180–L191 almost verbatim).\n\nI feel the paper would benefit from a few rounds of polishing and iteration to reach a more refined state. That said, I like the core idea and I encourage the authors to keep improving this work.", "was_revised": true, "final_justification": "Most of my concerns have been addressed. However, I still think the proposed method is restricted to the certain type of model, i.e., LRAS, which limits the applicability of the proposed to broader families of video models.\n\nOverall, this is an interesting paper, but there is still room for improvement in terms of quality. I will maintain my current rating for now (positive recommendation) as an encouragement.", "rebuttal": "We thank the reviewer for the thoughtful review and constructive suggestions! Overall, we are glad that reviewers found our method \"novel\" and \"interesting\" (uSVM) and our empirical results and analyses to be \"strong\" and \"detailed,\" and a significant improvement to existing methods (fTmd, uSVM, nvPE). We address the main points of the reviewer below:\n\n> The best performing setup relies on a specific model (LRAS), which somewhat limits the broader applicability of the method.\n\n> Could the authors discuss more concretely what would be required to make KL-tracing work with architectures beyond LRAS?\n\nThe key properties needed to make KL tracing work are (1) locality (local tokenizer enables detailed perturbation control) and (2) distributional prediction (predicting entire distributions instead of a single sample). Please refer to Section 3.5 for more detail. These features are model-agnostic and can be incorporated into a generative model's architecture in various ways. While the mainstream models of today (e.g., Cosmos, SVD) may lack these properties, and among existing models LRAS happens to have them, we believe that our results highlight an alternative direction for building powerful future generative video models, since the requirements that KL tracing suggest are not overly restrictive. For clarity, we will include further discussions outlining the general properties we have identified to work particularly well for KL tracing.\n\n> I believe the evaluations these two models should also be included in Table 2 for completeness.\n\nWe agree that placing SVD and Cosmos results directly in Table 2 makes the comparison clearer, and will add full results for both models in the revision. At the moment, to help with the discussion and contextualize diffusion-based extraction, we include TAP-Vid results from DiffTrack [1], a recent SOTA paper that uses a model-specific readout on Diffusion Transformers (DiT) for zero-shot optical flow:\n\n| TAP‑Vid DAVIS                 | $<\\delta^x_\\text{avg}\\uparrow$   |\n|:-----------------------------|-----------------:|\n| SEA‑RAFT                     | 58.7 |\n| DiffTrack [1] |        46.9 |\n| KL‑LRAS (ours)               |     **65.2** |\n\nThis shows that flow extraction from diffusion models remains challenging compared to KL tracing used with LRAS.\n\n> How sensitive is KL-tracing to the magnitude and shape of the perturbation? Would a different form (e.g., a low-frequency mask or structured noise) work similarly?\n\nWe thank the reviewer for engaging with our work and proposing interesting ablations. We evaluated this using the same setting as in Table 1. For \"shape\" and \"magnitude\", we vary different parameters of our Gaussian bump perturbation such as amplitude ($\\alpha$) and standard deviation ($\\sigma$). For the low-frequency mask perturbation, we use a standard Gaussian blur kernel. We find empirically that our choice of a Gaussian \"bump\" perturbation is quite important, though the results are less sensitive to the specific parameters. However, we are open to the idea that a more focused investigation on the optimal perturbation may yield better results or a different form. Our results are below:\n\n | Pert Type | $\\alpha$ | $\\sigma$ |  EPE $\\downarrow$ |\n|:----------|----------:|--:|-----:|\n | Gaussian  |      -200 | 2 | 5.30 |\n| Gaussian  |      -100 | 2 | 4.90 |\n | Gaussian  |       100 | 2 | 5.45 |\n| Gaussian  |       200 | 2 | 5.10 |\n| Gaussian  |       255 | 1 | 6.45 |\n| Gaussian  |       255 | 3 | 5.21 |\n| Gaussian  |       255 | 4 | 6.55 |\n| Gaussian  |       255 | 5 | 7.05 |\n| Low Freq. Mask |       255 | 2 | 41.61 |\n| **(baseline) Gaussian**        |       255 | 2 |  5.08 |\n\nFinally, we greatly appreciate the concerns brought up about writing clarity. We will take the reviewer's advice and (1) include a more proper introduction of Counterfactual World Models, (2) do a heavy reorganization of the method section and (3) add more discussions directly related to our contributions.\n\n[1] Nam et al., 2025. \"Emergent Temporal Correspondences from Video Diffusion Transformers.\""}, {"review_id": "CkwKIpIuhr", "reviewer": "Reviewer_XcXT", "summary": "The paper proposes a test-time intervention-based method to obtain the motion of one point for a future timestep in a video using an off-the-shelf video world model. The method works by introducing an intervention at the source point and then tracking that intervention's impact to subsequent frames. Two variants are explored, one where the effect is measured from changes in the output frames directly, and one where the effect is measured by the effect on the predicted posterior distributions.", "questions": "Currently, I see little evidence for the claims made for the main contributions 1 and 3, and in l. 52..54: Regarding main contribution 1, in the weaknesses section, I provided a non-exhaustive list of a range of methods that already investigate optical flow/tracker extraction from large generative (video) models, raising questions about the claim of this being \"the first systematic study\" of this topic. Given that the studies referenced also find other powerful mechanisms for optical tracking in such models, l. 52..54 seem unjustified given the analyses performed in this work. Similarly, just not comparing to relevant prior art does not make the proposed method state-of-the-art as claimed in main contribution 3.\n\nBringing this work to an acceptable level would require major revisions that thoroughly address the mentioned weaknesses.", "limitations": "No. The checklist refers to Sec. 5 for limitations, but I see *no* discussion of limitations of the proposed method in that section.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 2, "significance": 1, "originality": 1, "strengths_and_weaknesses": "**Strengths**\n\nThe proposed method is simple, which makes it likely to generalize well to other models that fit the scope of base models covered by the method.\n\n\n\n**Weaknesses**\n\n1. Speed: the method only queries a single point per forward pass, which will likely make anything but the sparsest inference very costly. At the very least, an analysis of the quality/speed tradeoff compared to other methods should be provided.\n1. Insufficient comparisons: the proposed method is not the first to enable (training-free) flow/tracker estimation based on pretrained generative vision models, so comparisons to these works should be included. Specifically, Track4Gen [Jeong et al., CVPR 2025] already evaluates a wide range of settings including zero-shot-, test-time optimization-, and full finetuning-based versions. Similarly, methods like DIFT [Tang et al., NeurIPS 2023] have also already demonstrated strong tracking performance using just features from image diffusion models. There are also a range of other works that investigated such mechanisms, such as Emerging Tracking from Video Diffusion [Zhang et al., OpenReview 2024], MOFT [Xiao et al., NeurIPS 2024], COVE [Wang et al., NeurIPS 2024], DistillDIFT [Fundel et al., WACV 2025]. In addition, standard optical trackers (CoTracker3, BootsTAP, ...) should also be compared to.\n1. The claims made in l. 52..54 (also in other points of the paper, such as the abstract) are very general but not suffciently backed up by experiments: claims are made about flow extraction from generative model architectures in general, but the experiments throughout the paper only cover one family of approaches (propagating intervention points).\n1. Many of the supposed advantages of \"large world models\" over classical methods proposed in 4.5 read more like limitations of pairwise flow estimation methods, not like limitations that would apply to optical tracking methods, which are also mainstream.\n1. The proposed KL-based method can, out of the box, only be applied to a very limited range of generative video models and, importantly, does not directly apply to the prevalent model paradigm: diffusion models.\n1. The checklist contains false claims, such as:\n    a. 2: Limitations: not discussed in the section mentioned\n    b. 7: Statistical significance: claimed to be reported, yet never provided", "was_revised": true, "final_justification": "Overall, I continue to lean towards rejection after considering the rebuttal, the other reviews, and the discussion. *Iff* the improvements promised by the authors, I am not fundamentally opposed to the acceptance of the paper. One major concern is that many of my concerns relate to a major misrepresentation of the contribution and main insights of the paper in the initial submission, whose resolution cannot be verified during the rebuttal/discussion phase.\n\nHere is a list of the initially mentioned weaknesses I consider resolved (Y), partially resolved (~) or unresolved (X):\n- W1 (major, X): The authors provided a speed comparison during the rebuttal, which shows the method to be 6+ orders of magnitude slower than sota methods. I consider this a severe limitation for any practical application\n- W2 (major, Y): Mostly resolved\n- W3 (major, ~): Resolving this concern can only be done by a fundamental revision of the presentation of the results and claims made in the paper, which can not be checked during the discussion period.\n- W4 (minor, ~): Again primarily about presentation, so not verifiable\n- W5 (minor, ~): I, unlike the authors, consider applicability to readily available models an important factor. Arguments can be made in both directions\n- W6 (minor, ~): Again primarily about presentation, so not verifiable", "rebuttal": "We thank the reviewer for the thoughtful review and constructive suggestions! We address the main points below:\n\n**1. Diffusion Model Comparisons:** We appreciate the reviewer bringing up existing diffusion methods. We acknowledge that several important attempts have been made in this direction, and we will include this in our related works and discussions.\n\nAfter our submission, DiffTrack [1], a zero‑shot flow extraction procedure using a pre‑trained Diffusion Transformer, has been introduced. The authors include metrics from TAP‑Vid, and the method is shown to supersede many existing diffusion baselines, including DIFT [2]. The reviewer also brought up Track4Gen [3], which published comparable metrics that we present below (we note, however, that Track4Gen is **not** training‑free$^\\dagger$, as their \"zero-shot\" evaluations are performed after they fine-tune the base model with trajectory annotations generated using optical flow).\n\n\n| TAP‑Vid DAVIS | AJ $\\uparrow$ | $<\\delta^x_{\\text{avg}}\\uparrow$ | OA $\\uparrow$ |\n|:--------------|----:|---------------------------:|----:|\n| DIFT [2] | — | 39.7 | — |\n| Track4Gen$^\\dagger$ [3] | 40.2 | 58.9 | **78.4** |\n| DiffTrack [1] | — | 46.9 | — |\n| KL‑LRAS (ours) | **44.2** | **65.2** | 74.6 |\n\n\nWe see that on the commonly reported metric $<\\delta^x_{\\text{avg}}$, which measures the percent of points that fall within a set of error thresholds, our method outperforms SOTA diffusion baselines. Comparing with Track4Gen [3], we outperform or are competitive on all metrics, despite being completely training‑free.\n\nThe reviewer also brought up additional concurrent work; however, at the moment direct comparison is challenging due to availability of code [4, 5]; difficulty in applying to non-generated videos (as mentioned in Sec. 6 of [6]); or differences in tasks [7], such as semantic correspondence vs. optical flow. Per the reviewer's suggestion, as public artifacts of previous methods become available, we will include more evaluations for comparison.\n\nIn sum, many existing diffusion baselines have already been superseded by DiffTrack [1], against which our KL‑LRAS method shows improved results. By including the latest, highest-performing baselines, we hope to better contextualize our results. We will continue to incorporate additional comparisons as methods become publicly available.\n\n**2. Speed:** Extracting information from any large, generative video model is computationally demanding, as we discuss in Section 5. It just sort of \"comes with the territory.\" However, it is still an important direction to take in order to leverage their deep understanding of scenes and motion. In fact, among the generative models we tested, our method is notably faster:\n\n   | Method | Relative time for point tracking |\n   |:-------|---------------------------------:|\n   | LRAS with KL‑tracing (ours) | **1×** |\n   | LRAS with RGB method | 8× |\n   | Cosmos | 68× |\n   | Stable Video Diffusion | 2x |\n\nBut as discussed in our section on limitations, the natural next step to take after successful flow extraction—as is common in the literature on using general foundation models to build deployable, task-specific models—is to distill it into a faster architecture that can output dense flow in real time. The main purpose of our method therefore is to be used as a sparse *pseudo‑labeler*. This will significantly increase domain diversity for flow datasets, which have traditionally been limited to synthetic, or heavily curated scenes, without incurring the intractable cost of human labeling.\n\nSince submission, we have been actively exploring distillation strategies, where we too have been interested in understanding the effect of label sparsity. In particular, we have trained an existing, off‑the‑shelf dense flow model on a synthetic dataset with ground‑truth labels using different levels of sparsity. We find that a model trained on an extremely sparsified (0.03% of points labeled) dataset performs competitively to the same model trained on the original, dense dataset (endpoint error: 2.47 for dense, 2.63 for sparse), strongly encouraging us on the likely effectiveness of sparse distillation. Our goal is to have the distilled model ready for release with the NeurIPS camera-ready version, if accepted.\n\n**3. Exploration of different extraction procedures:**\n\n> Experiments throughout the paper only cover one family of approaches (propagating intervention points).\n\nTo our knowledge, there are two main approaches for obtaining motion information from pre-trained models: (1) representation learning / mining approaches, as highlighted by the diffusion papers the reviewer mentioned, and (2) \"propagating intervention points,\" as discussed in our paper. We acknowledge that our focus has primarily been on intervention-based approaches. This is because, if possible to execute, (2) is simpler, more generalizable, and naturally zero-shot. Also many other works have reported baselines on type (1) already — and which our results surpass.  But ultimately, we agree with the reviewer's suggestion to expand the discussion of other branches of related work by including results from the latest diffusion methods (DiffTrack, etc.) from the table above.\n\n**4. Support for Diffusion models:**\n> The proposed KL-based method ... does not directly apply to the prevalent model paradigm: diffusion models.\n\nThis is an important point.  Though diffusion models are currently mainstream, and are great at generating photorealistic scenes, they have known difficulties with precise steering. Our results give a reason to continue exploring diffusion alternatives in future generative video models. (Other recent works provide independent reasons for this conclusion from different angles.) While methods that reinforce the use of a particular class of models that are currently popular can be valuable, we hope the same can be said for work that sheds light on complementary approaches.\n\nOur results expose two fairly general properties that, if present, are likely to be sufficient for good tracing: (1) locality, and (2) full distributional predictions. These criteria, while exposing a weakness of current diffusion models, are consistent with other classes of models that are actively being explored in the field. In fact, that diffusion models have this weakness is one of the points we are trying to make — one which we believe is quite valuable for the community.\n\n**5. Comparison with tracking methods:**\n\n> Limitations [do not] apply to optical tracking methods.\n\nTo add context to this discussion, we present comparisons with CoTracker‑v3, the latest SOTA point tracker.\n\n   | TAP‑Vid DAVIS | AJ $\\uparrow$ | AD $\\downarrow$ | $<\\delta^x_{\\text{avg}}$ $\\uparrow$ | OA $\\uparrow$ |\n   |:--------------|----:|----:|---------------------------:|----:|\n   | CoTracker‑v3 [8] | 39.85 | 19.04 | 57.96 | **76.87** |\n   | KL‑LRAS (ours) | **44.16** | **11.18** | **65.20** | 74.58 |\n\nPoint trackers face similar limitations, requiring synthetic data due to labeling difficulty (as discussed explicitly in [8]). The above results show CoTracker-v3 run under the same **two-frame** setting on TAP-Vid DAVIS to ensure fair comparison. The lower performance of CoTracker in this more difficult setting shows that the underlying motion estimation still remains challenging for optical trackers. Per the reviewer's advice, we will include this, and BootsTAP results with relevant discussion.\n\n**6. Limitations:** In the discussion, we do try honestly to mention the main limitation of our approach — the computational demand. We will further highlight this clearly in future versions.\n\n**7. Statistical Significance:** The TAP‑Vid benchmarks do not include standard procedures for measuring statistical significance, and most papers do not report such results. We interpreted the corresponding checklist entry to have been met by closely following the conventions of our domain, though we appreciate the reviewer bringing this to our attention. Part of the reason why this is not as emphasized in this domain is because the datasets are fairly large, so standard error bars tend to be extremely small and a little bit beside the point. Nevertheless, we computed the standard error for the Average Distance (endpoint error) metric for each model that we ran, as well as a paired t‑test to verify that our method significantly outperforms other models. For example, a one‑sided paired t‑test shows that our method achieves significantly lower endpoint error than Cosmos (p = 6e‑08). We will report these numbers in the supplement if they are believed to be important.\n\n[1] Nam et al., 2024. \"Emergent Temporal Correspondences from Video Diffusion Transformers.\"\n\n[2] Tang et al., 2023. \"Emergent Correspondence from Image Diffusion.\"\n\n[3] Jeong et al., 2025. \"Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generation.\"\n\n[4] Wang et al., 2024. \"COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing.\"\n\n[5] Zhang et al., 2024. \"Emerging Tracking from Video Diffusion.\"\n\n[6] Xiao et al., 2024. \"Video Diffusion Models are Training-free Motion Interpreter and Controller.\"\n\n[7] Fundel et al., 2024. \"Distillation of Diffusion Features for Semantic Correspondence.\"\n\n[8] Karaev et al., 2024. \"CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos.\""}, {"review_id": "Yr2ElLDSxL", "reviewer": "Reviewer_nvPE", "summary": "This paper introduces KL-tracing, a new approach to tracking image-space perturbations, for unsupervised optical flow extraction from the LRAS model. This paper also designs several new baselines from existing video models, such as stable video diffusion and cosmos. KL-tracking-based LRAS achieves strong performance on TAP-Vid, outperforming existing unsupervised models and some flow-specific models.", "questions": "See above.", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- From reported results, KL-tracing is more effective than RGB-difference-based perturbation tracking. It significantly improves the performance.\n- Authors also design baselines from other well-known video generative models, making the comparisons more fair.\n\n### Minor Weaknesses\n- The baselines from other models (e.g. cosmos) are relatively weak. \n- The overall design only allows 2-view point-tracking in a single inference, while optical flow is usually evaluated on (semi-)dense pixels. I recommend authors to follow the standard optical flow settings and evaluate KL-tracing on Sintel[1], KITTI[2], and Spring[3].\n\n[1] Butler, Daniel J., et al. \"A naturalistic open source movie for optical flow evaluation.\" Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part VI 12. Springer Berlin Heidelberg, 2012.\n\n[2] Geiger, Andreas, Philip Lenz, and Raquel Urtasun. \"Are we ready for autonomous driving? the kitti vision benchmark suite.\" 2012 IEEE conference on computer vision and pattern recognition. IEEE, 2012.\n\n[3] Mehl, Lukas, et al. \"Spring: A high-resolution high-detail dataset and benchmark for scene flow, optical flow and stereo.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "was_revised": true, "final_justification": "My concerns are addressed. I raise my rating to \"Accept\".", "rebuttal": "We thank the reviewer for the thoughtful review and constructive suggestions! Overall, we are glad that reviewers found our method \"novel\" and \"interesting\" (uSVM) and our empirical results and analyses to be \"strong\" and \"detailed,\" and a significant improvement to existing methods (fTmd, uSVM, nvPE). We address the main points of the reviewer below:\n\n> The baselines from other models (e.g., cosmos) are relatively weak.\n\nFlow improvement is primarily measured by comparing to stronger, task-specific baselines such as SEA-RAFT (Table 2), against which our method achieves better performance. The results from Table 1 that showcase Cosmos and SVD separately demonstrate the challenge of extracting motion cues from large-scale generative models that lack controllability. We will integrate Cosmos and SVD results into the main TAP-Vid table (Table 2) in future iterations for a more comprehensive comparison. Further, we introduce a significantly stronger (though more architecture-specific) baseline, DiffTrack [1], that also utilizes Diffusion models, and note that our method still achieves higher performance.\n\n| TAP‑Vid DAVIS                 | $<\\delta^x_{\\text{avg}}\\uparrow$   |\n|:-----------------------------|-----------------:|\n| SEA‑RAFT                     | 58.7 |\n| DiffTrack [1] |        46.9 |\n| KL‑LRAS (ours)               |     **65.2** |\n\n> The overall design only allows 2-view point-tracking in a single inference, while optical flow is usually evaluated on (semi-)dense pixels. I recommend authors to follow the standard optical flow settings and evaluate KL-tracing on Sintel, KITTI, and Spring.\n\nThank you for this important suggestion. As our main focus is a generalizable flow extraction method for real-world videos, we chose TAP-Vid as our main benchmark, following a recent trend [2, 3], as we believe it better captures a model's ability to generalize to challenging, in-the-wild scenes. To further improve our coverage, we also include results on Kubric (Table 2) from the same benchmark, which is a synthetic dataset similar to Sintel or Spring. We appreciate the suggestion for additional benchmarks and will consider expanding our evaluation in future work.\n\n[1] Nam et al., 2025. \"Emergent Temporal Correspondences from Video Diffusion Transformers.\"\n\n[2] Jiang et al., 2023. \"Doduo: Learning Dense Visual Correspondence from Unsupervised Semantic-Aware Flow.\"\n\n[3] Karaev et al., 2024. \"CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos.\""}]

Author rebuttal:

Meta-review: This paper introduces KL-tracing, a prompting-based approach for extracting optical flow from frozen self-supervised video world models. Built on the LRAS architecture, the method achieves state-of-the-art results on TAP-Vid DAVIS and Kubric, offering a novel and conceptually interesting counterfactual perspective on flow estimation without fine-tuning. The simplicity of the approach and its strong empirical performance on challenging benchmarks make it a notable contribution.

While reviewers raised concerns regarding the limited applicability of the method (with strongest results on LRAS), lack of evaluation on standard optical flow datasets, and computational efficiency, the rebuttal addressed several points satisfactorily. Importantly, reviewers recognized the novelty and potential of the idea, with two maintaining positive recommendations and one increasing their score to Accept. Only one reviewer continued to lean toward rejection, primarily due to concerns about generality and speed.

On balance, the strengths in novelty, conceptual clarity, and empirical performance outweigh the limitations. The paper can be accepted.
Decision: Accept (poster)