Title: Taming generative video models for zero-shot optical flow extraction
Venue: NeurIPS 2025
Accepted: yes
Keywords: optical flow

generative models
Abstract: Extracting optical flow from videos remains a core computer vision problem. Motivated by the recent success of large general-purpose models, we ask whether frozen self-supervised video models trained only to predict future frames can be prompted, without fine-tuning, to output flow. Prior attempts to read out depth or illumination from video generators required fine-tuning; that strategy is ill-suited for flow, where labeled data is scarce and synthetic datasets suffer from a sim-to-real gap. Inspired by the Counterfactual World Model (CWM) paradigm, which can obtain point-wise correspondences by injecting a small tracer perturbation into a next-frame predictor and tracking its propagation, we extend this idea to generative video models for zero-shot flow extraction. We explore several popular architectures and find that successful zero-shot flow extraction in this manner is aided by three model properties: (1) distributional prediction of future frames (avoiding blurry or noisy outputs); (2) factorized latents that treat each spatio-temporal patch independently; and (3) random-access decoding that can condition on any subset of future pixels. These properties are uniquely present in the recently introduced Local Random Access Sequence (LRAS) architecture. Building on LRAS, we propose KL-tracing: a novel test-time inference procedure that injects a localized perturbation into the first frame, rolls out the model one step, and computes the Kullback–Leibler divergence between perturbed and unperturbed predictive distributions. Without any flow-specific fine-tuning, our method is competitive with state-of-the-art, task-specific models on the real-world TAP-Vid DAVIS benchmark and the synthetic TAP-Vid Kubric. Our results show that counterfactual prompting of controllable generative video models is an effective alternative to supervised or photometric-loss methods for high-quality flow.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces KL-tracing, a new approach to tracking image-space perturbations, for unsupervised optical flow extraction from the LRAS model. This paper also designs several new baselines from existing video models, such as stable video diffusion and cosmos. KL-tracking-based LRAS achieves strong performance on TAP-Vid, outperforming existing unsupervised models and some flow-specific models.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- From reported results, KL-tracing is more effective than RGB-difference-based perturbation tracking. It significantly improves the performance.\n- Authors also design baselines from other well-known video generative models, making the comparisons more fair.\n\n### Minor Weaknesses\n- The baselines from other models (e.g. cosmos) are relatively weak. \n- The overall design only allows 2-view point-tracking in a single inference, while optical flow is usually evaluated on (semi-)dense pixels. I recommend authors to follow the standard optical flow settings and evaluate KL-tracing on Sintel[1], KITTI[2], and Spring[3].\n\n[1] Butler, Daniel J., et al. \"A naturalistic open source movie for optical flow evaluation.\" Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part VI 12. Springer Berlin Heidelberg, 2012.\n\n[2] Geiger, Andreas, Philip Lenz, and Raquel Urtasun. \"Are we ready for autonomous driving? the kitti vision benchmark suite.\" 2012 IEEE conference on computer vision and pattern recognition. IEEE, 2012.\n\n[3] Mehl, Lukas, et al. \"Spring: A high-resolution high-detail dataset and benchmark for scene flow, optical flow and stereo.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a test-time intervention-based method to obtain the motion of one point for a future timestep in a video using an off-the-shelf video world model. The method works by introducing an intervention at the source point and then tracking that intervention's impact to subsequent frames. Two variants are explored, one where the effect is measured from changes in the output frames directly, and one where the effect is measured by the effect on the predicted posterior distributions.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThe proposed method is simple, which makes it likely to generalize well to other models that fit the scope of base models covered by the method.\n\n\n\n**Weaknesses**\n\n1. Speed: the method only queries a single point per forward pass, which will likely make anything but the sparsest inference very costly. At the very least, an analysis of the quality/speed tradeoff compared to other methods should be provided.\n1. Insufficient comparisons: the proposed method is not the first to enable (training-free) flow/tracker estimation based on pretrained generative vision models, so comparisons to these works should be included. Specifically, Track4Gen [Jeong et al., CVPR 2025] already evaluates a wide range of settings including zero-shot-, test-time optimization-, and full finetuning-based versions. Similarly, methods like DIFT [Tang et al., NeurIPS 2023] have also already demonstrated strong tracking performance using just features from image diffusion models. There are also a range of other works that investigated such mechanisms, such as Emerging Tracking from Video Diffusion [Zhang et al., OpenReview 2024], MOFT [Xiao et al., NeurIPS 2024], COVE [Wang et al., NeurIPS 2024], DistillDIFT [Fundel et al., WACV 2025]. In addition, standard optical trackers (CoTracker3, BootsTAP, ...) should also be compared to.\n1. The claims made in l. 52..54 (also in other points of the paper, such as the abstract) are very general but not suffciently backed up by experiments: claims are made about flow extraction from generative model architectures in general, but the experiments throughout the paper only cover one family of approaches (propagating intervention points).\n1. Many of the supposed advantages of \"large world models\" over classical methods proposed in 4.5 read more like limitations of pairwise flow estimation methods, not like limitations that would apply to optical tracking methods, which are also mainstream.\n1. The proposed KL-based method can, out of the box, only be applied to a very limited range of generative video models and, importantly, does not directly apply to the prevalent model paradigm: diffusion models.\n1. The checklist contains false claims, such as:\n    a. 2: Limitations: not discussed in the section mentioned\n    b. 7: Statistical significance: claimed to be reported, yet never provided\n\nQUESTIONS:\nCurrently, I see little evidence for the claims made for the main contributions 1 and 3, and in l. 52..54: Regarding main contribution 1, in the weaknesses section, I provided a non-exhaustive list of a range of methods that already investigate optical flow/tracker extraction from large generative (video) models, raising questions about the claim of this being \"the first systematic study\" of this topic. Given that the studies referenced also find other powerful mechanisms for optical tracking in such models, l. 52..54 seem unjustified given the analyses performed in this work. Similarly, just not comparing to relevant prior art does not make the proposed method state-of-the-art as claimed in main contribution 3.\n\nBringing this work to an acceptable level would require major revisions that thoroughly address the mentioned weaknesses.\n\nLIMITATIONS:\nNo. The checklist refers to Sec. 5 for limitations, but I see *no* discussion of limitations of the proposed method in that section.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 1, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes KL-tracing, a novel zero-shot inference approach for extracting optical flow from pretrained generative world models. Inspired by the concept of counterfactual world models (CWMs), the authors introduce a perturb-and-track procedure based on KL-divergence in latent spaces, tracing subtle perturbations through predictions of future frames. Evaluations conducted on the challenging TAP-Vid DAVIS and Kubric benchmarks demonstrate convincing performance improvements, especially when using the LRAS generative model.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The idea in this paper is interesting, although it takes time to digest due to the writing clarity. The use of KL-divergence in latent distributions to trace perturbations and the perspective of counterfactual world models are not commonly seen in this direction. It feels novel and offers a fresh take on optical flow to me as a reviewer.\n\n2. The empirical results on the TAP-Vid benchmarks are good and convincing. \n\n3. Despite could-be-improved writing clarity, the authors do a commendable job walking through how the counterfactual idea applies to different types of generative models (e.g., LRAS, Stable Video Diffusion, Cosmos). As a reviewer, I appreciate the effort to bridge the conceptual gap across these architectures.\n\n\nWeakness:\n1. The best performing setup relies on a specific model (LRAS), which somewhat limits the broader applicability of the method. The authors provide analysis for SVD and Cosmos, but those results are weak. I believe the evaluations of these two models should also be included in Table 2 for completeness, as that table is the clearest comparison with traditional flow baselines.\n\n2. The writing could be greatly improved. The introduction assumes the reader is already familiar with Counterfactual World Models, which may not be true for everyone. Also, the paper emphasizes LRAS architecture too heavily, despite it not being the core contribution here. Lastly, I’d suggest reorganizing the method sections more cleanly—there’s noticeable redundancy (e.g., L117–L125 repeat in L180–L191 almost verbatim).\n\nI feel the paper would benefit from a few rounds of polishing and iteration to reach a more refined state. That said, I like the core idea and I encourage the authors to keep improving this work.\n\nQUESTIONS:\n1. Could the authors discuss more concretely what would be required to make KL-tracing work with architectures beyond LRAS? The performance on cosmos and SVD is not strong.\n\n2. How sensitive is KL-tracing to the magnitude and shape of the perturbation? Would a different form (e.g., a low-frequency mask or structured noise) work similarly?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the potential of extracting optical flow in a zero-shot manner from frozen, large generative video world models. Motivated by the limitations of prior supervised and photometric approaches, the authors systematically evaluate deterministic, diffusion, and autoregressive generative models. They show that successful flow extraction requires distributional prediction, factorized local latents, and random-access decoding. The paper introduces KL-tracing, a test-time inference procedure that measures the Kullback-Leibler divergence between clean and perturbed model predictions to robustly localize motion and extract flow without model fine-tuning. Experimental results on real-world (TAP-Vid DAVIS) and synthetic (TAP-Vid Kubric) datasets demonstrate that KL-tracing with LRAS surpasses both classical and state-of-the-art flow models, even though the world models were not trained with flow supervision.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper presents detailed and comparative results across various generative world models (deterministic, diffusion, autoregressive, and LRAS), using standard real-world and synthetic benchmarks. The results, particularly in Table 2, demonstrate strong empirical performance, with KL-LRAS outperforming both supervised and unsupervised baselines in several key metrics.\n\nDespite strong performance on TAP-Vid, the paper does not empirically demonstrate zero-shot transfer to significantly out-of-distribution domains, such as medical, aerial, or night-time video. While the theoretical basis for generalization is sound, more experimental evidence or at least critical discussion would be helpful. In addition, although the performance is good as shown in the Figure, the paper failed to compare with other tracking methods like cotracker3. We encourage the authors to do more compareison with other SOTA optical flow or tracking methods.\n\nQUESTIONS:\nThe idea is now new, as it has been proposed in papers like SiamMAE and other papers. The authors are encourage to do more experiments including both ablation and comparison with other SOTA methods.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}