Self-Supervised Learning of Motion Concepts by Optimizing Counterfactuals
Authors: Stefan Stojanov
David Wendt
Seungwoo Kim
Rahul Mysore Venkatesh
Kevin Feigelis
Klemen Kotar
Khai Loong Aw
Jiajun Wu
Daniel LK Yamins
Venue: NeurIPS 2025 spotlight ()
Keywords: self-supervised learning
motion understanding
visual tracking
video foundation modeling
Primary area: applications

Abstract: Estimating motion primitives from video (e.g., optical flow and occlusion) is a critically important computer vision problem with many downstream applications, including controllable video generation and robotics. Current solutions are primarily supervised on synthetic data or require tuning of situation-specific heuristics, which inherently limits these models' capabilities in real-world contexts. A natural solution to transcend these limitations would be to deploy large-scale, self-supervised video models, which can be trained scalably on unrestricted real-world video datasets. However, despite recent progress, motion-primitive extraction from large pretrained video models remains relatively underexplored. In this work, we describe Opt-CWM, a self-supervised flow and occlusion estimation technique from a pretrained video prediction model. Opt-CWM uses ``counterfactual probes'' to extract motion information from a base video model in a zero-shot fashion. The key problem we solve is optimizing the quality of these probes, using a combination of an efficient parameterization of the space counterfactual probes, together with a novel generic sparse-prediction principle for learning the probe-generation parameters in a self-supervised fashion. Opt-CWM achieves state-of-the-art performance for motion estimation on real-world videos while requiring no labeled data.

Self-Supervised Learning of Motion Concepts by Optimizing Counterfactuals
Stefan Stojanov" Stanford University
David Wendt" Stanford University
Seungwoo Kim Stanford University
Rahul Mysore Venkatesh" Stanford University
Kevin Feigelis Stanford University
Klemen Kotar
Khai Loong Aw Stanford University
Jiajun Wa Stanford University
Stanford University
Daniel L.K. Yamins
Stanford University
Abstract
Estimating motion primitives from video (e.g., optical flow and occlusion) is a critically important computer vision problem with many downstream applica- tions, including controllable video generation and robotics. Current solutions are primarily supervised on synthetic data or require tuning of situation-specifiy heuris- tics, which inherently limits these models' capabilities in real-world contexts, natural solution to transcend these limitations would be to deploy large-scale, self- supervised video models, which can trained scalably on unrestricted real-world video datasets. However, despite recent progress, motion-primitie extraction from large pretrained video models remains relatively underexplored. In this work, we describe Opt-CWM, self-supervised flow and occlusion estimation technique from a pretrained video prediction model. Opt-CWM uses "counterfactual probes" to extract motion information from a base video model in zero-shot fashion. The key problem we solve is optimizing the quality these probes, using combination of an efficient parameterization of the space counterfactual probes, together with sparse-predictie prediction principle for learning the gebbbratnn parame- ters in self- supervisdd fashion. Opt-CWM achieves state-of of of-the-art performance for motion estimation on real-world videos while requiring no labeled data.
novel generic sparse
Introduction
Extracting "low-level" scene motion properties such as optical flow 13 40 occlusions (28), and point object tracks [19 10 is important for video understanding applications such automated video filtering [54] 55), action recognition 26 38 and motion prediction [3) Recently, scene
motion primitives have also been critical for increasing the controllability and consistency of video
generation models 15 and have gained an important role in robotics applications [43
Optical flow and occlusion are two core primitives in this domain. The most common approach to optical flow estimation uses supervised learning from labeled flow data. However, because densely annotating flow in real-world scenes is prohibitively expensive, supervised methods usually rely
synthetic data 31 32). Methods trained on synthetic data have proven to be robust real-world video 40 50] However, relying on this approach has limited flow estimation methods from taking
Eqqual contribution. Project website: https://heurooilaa..gg http:///wwoooaaaaa........papppp
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

mosking policy
FLOW
OCC! luason
Samel Merro))
Frama Guarse)
claan
dear prediidinn
Malil
ROB
BGB
and ided
1XRCOO frume?
petuubeddrppu
parturted pow.cttttooooooo
petta ingut
pertured ctoliction
Figure 1: Counterfactual probing for flow and occlusion: (A) CWMs learn to predict the next frame with temporally factored masking policy B. (B) The motion of a point can be estimated using a counterfactual probing program FLOW: the model predicts the next frame with and without a perturbation placed on the point, and the difference image between the clean and perturbed predictions reveals the estimated motion. (C) Occlusion is estimated using related probe OCC: diffuse and low magnitude difference indicates that the perturbed point has been occluded.
advantage of recent advances in self-supervised visual representation learning from massive video and inherently has to contend with a sim-to-real domain gap.
datasets 41 35 14
estimannstimation methods are typically based on photometric loss learning frame-pair feature correspondences to warp pixels from one RGB frame to corresponding locations in future frames. However, pure photometric loss is a weak constraint, in part because correspondences are often ill-defined (e.g.. objects with homogeneous textures). Existing state-of-the- art methods use various nearest neighbor or clustering procedures [21 6 or complement photometric
In contrast, self-supervised
loss with strong task- seek-specific regularizations like smoothness [23 39 Because these heuristics are
often only correct in narrow scenarios, performance is limited in cases where the heuristics fail.
In this work, we show how extract high-quality self-supervised flow and occlusion estimates without the use of such heuristics. A promising initial approach to this problem comes from Counterfactual World Modeling (CWM) [3 444, method that constructs zero-shot estimates of a variety of visual properties (flow, segments, shape, etc) from an underlying pre-trained multi-frame model (Figure D CWM begins with sparse RGB-conditioned next frame predictor 8 two-frame masked autoencoder trained with a highly asymmetric masking policy 3 This forces the model to encode scene dynamics small number of patch feature tokens that factor temporal dynamics from visual appearance. Motion properties can then be extracted from the base model in a zero-shot fashion via simple "counterfactual probes". acting as a kind of test-time inference procedure. For example, to compute flow from a given point in the first frame, a perturbation is made to the image at that point, and flow computed by comparing the difference between aGB. prediction on the perturbed (counterfactual) condition with its prediction in the original unperturbed (factual) condition (see Figure lB.. Intuitively, this corresponds to placing a visual marker on the point, predictively flowing it forward, and then analyzing where it gets "carried" in the predicted next frame.
In principle, the CWM approach circumvents the key limitation of the heuristic- based methods by replacing situation-spesct -pecific fixed heuristics (e.g., motion smoothness) with general- general-purpose purpose predictive model. The quantity of interest, in our case flow, is defined as the outcome of probing the model's predictions [44]. However, while CWM is an intriguing conceptual proposal, it has conceptual drawback that substantially limits its real-world performance: the probes that it relies on are hand-designed and can be out-o--domain in real-world video. Perturbations are often not properly "carried along" " with moving objects, resulting in suboptimal counterfactual motion extractions (Figure 2B. As result, the accuracy of the flows extracted by the originally proposed CWM method has remained inferior to wate-of-the-ar flow estimation solutions.
Here we present Opt-CWM, generic solution to this problem. Opt-CWM introduces two conceptual innovations that leverage the advantages of the CWM idea while making highly performant in real- world settings. The first of these innovations is method for parameterizing a counterfactual probe policy generator with a learnable neural network (Figure A). This network can predict situation- specific probes that take into account the appearance context (both local and global) around target points to be tracked, and thus can be less -uitt--disti than hand-coded probes. The second innovation is an approach for learning the probe generator in a principled fashion without relying

---

on any supervision from labeled data or heuristics. The main insight behind this learning procedure construct a task-agnostic generalization of the asymmetric masking principle used train the base model Ap itself. In particular, Opt-CWM connects sparse outputs of the parameterized flow prediction function to randomly initialized sparse now-conditioned next-frame predictor f and performs joint optimization (Figure 3 of both qo and the probe generator. This forces T to predict a future frame based on a present frame and sparse (putative) flow, creating an information bottleneck that generates useful gradients back on the probe generator's 's parameters.
We find that Opt-CWM achieves strong performance when compared with existing motion estimation methods (both supervised and self-supervised) that are purposely built for this task [39 37 50 33]. as well as recent adaptations of lcrge-scale self-supervised visual representations for motion estimation 233. when evaluated on real-world benchmarks [TO). The success of our approach reveals a promising direction for scalable counterfactual extraction of a variety of visual properties.
2 Related Work
Supervised flow estimation. Supervised methods like RAFT 40 50
approach optical flow as a
dense regression problem and learn from synthetic optical flow datasets 7 31. They also typically use task-specific architectures that are mailor-made for optical flow estimation, with strong inductive biases (e.g.. iterative floddfilling)) and task regularizations to ensure learning from limited training datasets. While these methods show strong performance in many contexts, their reliance on synthetic supervision and specialized architectures limits their generalizability. It is for this reason that our self-supervised Opt-CWM, which can be trained on unlimited in-the-wild videos, can outperform even supervised methods in certain key contexts.
Self-supervised flow with photometric loss. Methods for self supervupe flow learning 24 such as SMURF (39] learn dense visual correspondence by optimizing photometric loss. Because of the weakness of pure photometric loss alone as supervision, these methods rely on complex variety of heuristically chosen regularization losses (e.g., spatial smoothness of flow, among others) to achieve reasonable performance levels. Because these heuristics need to be tuned in a datasct-specifit manner, these methods have failure models in complex dynamic scenes, especially with variable and large time-framm frame gaps. In contrast to these methods, Opt-CWM does not rely on such heuristics, as the quality of the flow extraction is directly correlated with the prediction learning objective.
Augmenting self-supervised flow with visual pre-training. variety of methods augment photo-
metric loss using features derived from self supervised visual pre- tra-iiaining 2 6 [1] 23). For example, the recent state-of-the-art Doduo method 23 uses DINOv2 features as a basis on which to compute feature correspondences for downstream photometric loss. This approach allows the extension of these methods to wider video training datasets (such as Kinetics) and thereby improves performance and generalizability. However, even when backed by strong image features, photometric loss is weak constraint, requiring additional heuristic regularizers to improve performance. Opt-CWM, by avoiding scenario-specific heuristics, compares favorably to these methods.
Point tracking. Point tracking across multiple frames is a related problem to flow and occlusion estimation. The majority of solutions for point tracking are supervised [19 10) or semi- semi-supervised (55 11] and as such are further out of scope for this work. However, several recent works propose self-supervised approaches to finding temporal correspondence, typically relying on pre-trained representations [6] 21) These methods then extract point tracks through consistency objectives such as cycle consistency 66 27 or heuristics like softmax-similante [45] applied at the frame pair level. Tumanyan et al. 42] take a related approach, performing test-time optimization on individual videos using pre-trained DINO features and short-term supervision from RAFT. The current otate-of- the-art self-supervised method in this domain, GMRW (77, which is the main baseline comparison for our proposed Opt-CWM, uses cycle consistency to build tracks based on frame pair-level predictions.
Real-world motion benchmarks. The TAP-Vid benchmark 100 provides critical set of metrics for measuring the accuracy of motion-estimation systems in real-world video. This is critical for ensuring that potential advances motion estimation are tested against the challenges of real- wollworldd motion complexities, covering scenarios not encountered in synthetic benchmarks (e.g., non-rigid, highly articulated, deformable and breakable objects, fluids, inelastic collisions, animate objects, and human interactions). While originally intended for the supervised point tracking domain, recent
3

---

A
B
Learned
conditioned encoder
perturbed Imput
perturbelion predictor Mil
V
REB etc
MLP,
penateddippt
notion
token
preditted porturbzze tanuraacionn
FLOK
perturbed original
Figure 2: Parameterizing the counterfactual probe generator as an input-conditioned function. (A) Building pre-trained RGB-conditioned predictor 8 Opt-CWM uses image-conditioned perturbation prediction function óş containing a small MLPp. As illustrated in B, dg can learn to predict image-conditioned perturbations that blend naturally with the underlying scene, potentially allowing for the perturbation to be accurately carried over to the next frame prediction. But how should the parameters of ó be learned without any flow supervision labels? See Figure 3
self-supervised tracking works have begun to utilize TAP-Vid as a main benchmark for motion In this work, we also follow this practice.
estimation 23 37
3 Methods
3.1 Counterfactual World Modeling
CondiCondi Next Frame Predictor. A Countefactual World Model (CWM) is RGB- conditioned next frame predictor AGB consisting of an encoder & and decoder dee similar to VideoMAE [41]. but trained with highly asymmetric masking policy that reveals all patches of the first frame and a small fraction of patches of the second frame [3 (and see Figure (TA). Specifically, let J1, I2 E IR Hx be two frame pairs in a video, and define Ma as masking function that randomly masks some fraction, of patches in an image. Given fully visible first frame Iı and partially visible second frame (.(I). RGB trained predict Iz by minimizing
L MSE 12). where f2
T RGB (I, (Z.))).
(1)
Here we train CWM with 0.1 on publicly available video data with frame gap of 150ms. (See the supplement for more details.) The asymmetric masking training policy forces RGB to separate scene appearance-which is wholly available in the first trame-from scene dynamics, the information of which is now concentrated the sparse set visible next frame patches. other words, y RGB is "temporally factored"..
Motion Estimation With Counterfactual Probes. Because it induces strong dependence on the appearance and position of the revealed patches from Iı and Iz, temporal factoring allows the zero-shot extraction of visual structure through applying counterfactual probes: small changes to the appearance the position of visible patches. By measuring the predictor's 's response to these counterfactuals, we can easily extract useful information like object motion, segments, or shape from its representation [3]. As shown in Figure IB using the FLOW procedure, colored patch is placed on a moving object, and its motion can be determined by finding its location in the predicted frame. To track some pixel location P1 (row), 1, coli) from one frame the next, input image /1 is perturbed by adding a colored patch 6 at pixel location P1 to create the counterfactual input image 1 = 11 + Then, the next frames with and without the counterfactual perturbation are predicted:
(2) Subtracting these two predicted frames and taking an L -norm across the color channels produces the difference image A = 1řz - fzlg- Finally, the next- fext-frame pixel location Pa can be computed by finding the peak in the difference image: PO2 = arg max A. FLOW essentially a kind of test-time inference applied to the pretrained CWM base model. To extract occlusion information, the OCC procedure is identical to FLOW up to computation of the difference image (See Figure However, if a patch in the first frame gets occluded in the second frame, the response to the perturbation in the difference image A will be small in magnitude and diffuse shape. Applying simple threshold to the maximum value of A creates occlusion binary indicator.
ig = RGB (T + 6, M. = RGB (K, (,,M)) and i = RGB (a,)))

---

flow downstream proxy
FLOWg upstream goal
frame reconstruction error
U
RGB
flow estimation error (max pixel pixel distance)
(tog MSE)
Sqarie
FLOW
FLOW
T flow 1
y
1ov
frams
sparse forward flow
mim LABK 0.3
50
100
150
training epochs
Figure 3: A generic sparse-prediction principle for learning optimized counterfactuals. A) The parameterized counterfactual flow function FLOW, extracts motion from frozen RGB-conditionee predictor RGB through counterfactual perturbation (details in Figure 2 Its parameters 0 are trained using gradients from flow-conditioned predictor d that jointly trained perform next-frame framee prediction. The predictor flov can only learn to predict future frames if it is given correct flow-like vectors, form of information bottleneck that ensures useful gradients are passed back FLOWg- We thus exploit the pre-trained aAB predictor by bootstrapping using an extension of the principle of sparse next-frame prediction. (B) There tight coupling between the low-conditioned predictor flou and the learned flow estimation function FLOWg, so both pixel reconstruction (the proxy goal) and motion estimation (our real goal) simultaneously
dow-conditioned predictor y floo
improve.
3.2 Optimizing Counterfactual Perturbations
The problem with hand-designed perturbations. While the CWM approach of using fixed hand- designed probes (c.g., additive colored squares) can sometimes be effective probing motion with q AGB they are often suboptimal. First, this is because they are out of domain for the base predictor, and second, by being image- and -istionnindeeeedentt they can be unsuited the local image context. Anecdotally, this results in visually obvious failure cases, such the perturbation not moving with the object or being suppressed entirely.
Using the challenging TAP-Vid benchmark (see Section 4or more details), we empirically quantified that the original fixed hand-designed perturbations tion performance (see CWM results in [T. The main requirement for a "good" perturbation is that it is sufficiently in-distribution and image/point specific cause meaningful context-dependent changes for probing the base predictor. But how can probes be designed for this purpose? Our solution has two basic novel components: parameterizing an image-conditioned and differentiable counterfactual probe generator, and formulating a general general-purpose self-supervised loss objective for learning the probe generation policy parameters.
insufficient for self-supervised motion estima-
Parameterized Perturbations. We re-foouulate the motion extraction procedure from Section 3.1 to make it parameterized differentiable policy function and introduce the functional form of a sum of colored Gaussians as a natural perturbation class. (See Figure A
Let FLOWg: (J,. J2, P1) + & be per-pixel motion estimation function with learnable parameters that takes an image pair (21,12) and a pixel location P1 in /1 and outputs the predicted flow 62 P1- Here, 62 is the estimated second frame pixel location. The procedure FLOWa involves multiple components: the counterfactual perturbation function, (a, P1), which now produces variable counterfactual perturbations
function of the frame pair and pixel location (as opposed to a fixed perturbation 5, used in the standard CWM); the pre-trained, frozen, RGB- conditioned predictor, RGB utilized within the FLOWa program; and "softargmax" module to predict a pixel location using a differentiable approximation to the argmax function.
Gaussian Perturbations. We choose to parameterize the counterfactual perturbations as Gaussians because this function class presents natural method of forming in-domain counterfactual input images. To compute the Gaussian parameters for given counterfactual perturbation, use the
5

---

encoder of the con-conntt predictor, L RGB This outputs a sequence of feature tokens from its last transformer block, which encode global and local scene content for each patch and thus form a natural basis from which Gaussian parameters can be computed using shallow MLP. Given pixel location pI, we find its corresponding patch embedding token, 1p and use it as an input to an MLP that outputs a parameter vector which is in turn used to compute the Gaussian perturbation:
50 (I1. Ma (12). P1) = Gaussian (MLP (tp,)) )) where tp, = L
RGB enc
(I Ma. (12)))
(3)
Then, FLOW, computes the difference image, A, similar to the FLOW program, using i = 8g (I1 50, Ma)))). To make FLOW, differentiable, we use a softargmax over A to estimate f'2.
+
Softargmax Module. We follow the softargmax formulation proposed in [48]. Given difference image, A = 1/2 I2 we apply a teaperature-scaled 2D softmax and then take the expectation to
find 2a P2 ~ooommax (A/ /r) [p]] The predicted flow is then computed as P = f22 P1-
Learning Optimized Counterfactuals. Now that the perturbation generator is parameterized, the question arises: how can its parameters be learned? What type of self-supervised objective will cause the perturbation generator function to be context-specific and result in accurate flow vectors? Our main insight is that this problem can be "bootstrapped" in a robust fashion by generalizing the sparse asymmetric mask learning paradigm to encompass a coupled and mixed-mode RGB-Flow prediction problem without using labeled data (see Figure 3). Specifically, we jointly train the parameterized counterfactual motion prediction function, FLOW,, which estimates a set of flow vectors; together with a sparse now-conditioned predictor, f which takes a single frame along with sparse flow vectors to predict the next frame. We constrain FLOWa by passing its outputs as inputs to fl and training end-to-end using final RGB reconstruction loss on the predictions of g # As L # has no access to any RGB patches from the second frame 1z, it is only if the putative flows correct that it be possible for flow to use them to minimize the next-frame reconstruction loss.
Specifically, given an image pair 12), we estimate the motion for a set of pixels P- (p) (1)
Pi [n] using FLOW,, obtaining, a set of estimated forward flow vectors ((m)) J. Let flox (1, F) + I2 be a flow-conditioned next frame predictor with parameters 77 that takes the first frame RGB input /1 and predicts the next frame 12. conditioned on the flow input F. We jointly optimize @ and n, by minimizing ming.,, LMSE (Ig, I2). Figure 3 shows that optimizing end-to-end reconstruction does indeed couple tightly to upstream flow accuracy, as required for effective bootstrapping.
J F- (2t))
In this work, we investigate two nG base predictors, with 175M and IB learnable parameters. For optimizing the counterfactuals and ablations, we use the 175M model, and report benchmark results by applying the learned counterfactual probes to the IB model.
Inference- time Enhancements. A simple random masking strategy may inadvertently reveal the ground truth RGB at the next frame location we are trying to predict for particular point. In this event, the model will not carry over the counterfactual perturbation to the future frame, leading to an erroneous flow prediction. A simple yet effective inference-time -ime solution in which we apply multiple random masks and average across the resulting delta images to reduce the influence of sub-optimal masks. Following prior work [13 22 we also perform an iterative multiscale refinement of flow predictions by recursively applying FLOW, to smaller crops centered on the predicted point location, Po2 of the previous iteration. We observe that FLOW, is able to generate good initial flow predictions, and thus benefits from refinement (Table 2
multi-mast (MM),
4 Experiments
Evaluation Protocol. Our main datasets for evaluation are TiP-Vid DAVIS and TAP-Vid Kinet- ies (10), the DAVIS 34 and Kinetics (26) datasets with human flow and occlusion annotations, along with the synthetic Kubric [8] dataset where ground-truth truth flows and occlusions are known. All the methods we test output direct two-frame flow (point-tos to to-point correspondence) predictions. Some of them output occlusion predictions, which we use when available. For flow methods without an existing implementation of occlusion prediction, we use cycle consistency to compute occlusion estimates: occlusion is the event of inconsistency between forward and backward predictions greater than pixels. Models that can accept variable resolution inputs are run with the resolution closest to native that can be fit into memory, ensuring that each is run optimally. Metrics for both procedures

---

Opi-CNM (ouns)
Burs)
80
D0
Figure 4: Qualitative comparisons on real-world videas. These examples show the failure modes of methods reliant on visual similarity. We observe that the baseline models struggle against subtle but functionally important changes in largely homogeneous scenes depicting objects of similar color and texture ((a) (e)). Further, the use of photometric loss in self-supervised methods such as SMURF can also be susceptible to differences in light intensity across frame pairs ((f) (h)). In contrast, as a visual world model, Opt-CWM possesses a holistic understanding of scene transformations and object dynamies, and is able to find correspondence without arbitrary heuristics.
are always computed after rescaling predictions to 256 x 256 resolution, as [37.. Following the TAP-Vid First protocol proposed in [10]. for each point, we take the frame in which it is first visible and track its motion only forward time. We also use the first frame a point is visible as the reference frame, and track points in future time steps with reference to this frame. This is challenging setting as it involves tracking points across variable and often large frame gaps. It is also most comparable to many real-world use-case scenarios where the frame gap may be unknown or uncontrollable. We show additional results for the TAP- Vid Constant five-Frame Gap (CFG) protocol in the supplementary material, which is more favorable standard optical flow methods.
Metrics. We use the official metrics from the TAP-Vid evaluation protocol [T] 1) Average Jaccard (AJ), a precision metric measuring combination of point tracking and occlusion prediction; 2) Average Distance (AD) between the estimated pixel and ground truth locations; 3) < 6% which measures the average percentage of points predicted correctly within variety of pixel distance thresholds; and 4) Occlusion Accuracy (OA), the fraction of points correctly predicted as either occluded visible. Additionally, to account for the relative lack of occlusion events in the dataset, we also evaluate 5) Occlusion Fl (OF1), which computes the Fl score of the occlusion predictions.
Our evaluation protocol requires tracking points in videos through occlusion by finding temporal correspondence: given a frame pair, determine where the point went or whether was occluded. Therefore, the appropriate baselines are supervised and self-supervised optical flow methods, and self-supervised temporal correspondence methods. We run the following baselines:
Baselines.
CWM [3] [4] represents motion estimated through counterfactual extractions with a fixed hand- designed perturbation. This comparison shows how the innovations introduced by Opt-CWM lead to very substantial performance improvements.
GMRW 37 is self-supervised video correspondence approach that trains transformer using cycle consistency on contrastive random walks. GMRW is designed for temporal correspondence- -bssed long-range tracking and is the SOTA baseline for comparison on TAP-Vid Vid First.
SMURF 39 is an unsupervised method specifically designed for optical flow estimation. SMURF tailors the RAFT 440 architecture so it can be trained using combination of optical flow-specific
7

---

Table 1: Quantitative results on TAP-Vid dataset. In the TAP-Vid First protocol, point is tracked from when it is first visible to the end of the video, requiring motion estimation across large frame gaps. Opt-CWM outperforms both supervised and unsupervised baselines. "S" and "U" indicate supervised and unsupervised, respectively. Doduo is not strictly unsupervised as it uses segmentation labels. GMRW is trained the Kubric dataset, (marked with f), making a more favorable evaluation setting for that method because of the minimal domain gap. Best performing supervised models (shaded) are considered separately.
Method
DAVIS
Kinttici
Kulric
A3
An
<<
DA
OF1
A11
AD2
<.4
0A4
CIFI
A3.
ADD
(IA
OFF
8
19.85
19:04
57.95
76.87
47.1 1.133
16:11
25.00
53.41
77.29
59.45
76.41
5.10
81:3
90:45
71.06
RAFT [00
41.77
25:33
$4.37
56.40
56.12
41.92
23.00
53.49
74,67
70.62
71.93
5.60
82.15
88.54
68.34
SEA RAFT 50
43.41
20.18
39.69
06.34
56.23
33.40
30.72
46.11
64.55
04.39
75.05
6.34
84.63
89.50
70.92
DPHlow
33
49.30
16.36
62.31
00.15
47:33
17.05
58.74
81.
77.00
78.45
5.05
86.81
90.86
74.53
01
Dodoo
X1
25.34
13:41
48.30
47.1
09:43
31.51
15.05
66.87
66.71
06.01
54.98
5,31
72.20
75.56
52.67
GMRW
X3
35.E1
20:26
58,59
76.36
42.85
25.55
29.28
41.63
71.06
33.57
58, 58.36"
2.88
79.21
80.10
32.
SMURE
30.64
27.28
48.18
59.15
66.91
35.31
$2.56
64.37
66.00
60.00
05.11
6.81
361.2
37.11
29.42
OpeCCWM
47.03
79.98
heuristic losses like photometric loss and a variety of types of smoothness regularizations. SMURF trained on synthetic datasets often used for optical flow estimation learning.
Doduo [23] applies a SMURF-like combination of self-supervised photometric and smoothness losses, scaling them to larger model architectures and in-the-wild training videos [52]. It leverages the DINOv2 [ encoder to incorporate strong image priors. The Doduo model and training dataset are comparable in size to Opt-CWM, providing a control to ensure that the improved performance of Opt-CWM relative to SMURF not solely due model or dataset training size.
SEA-RAFT 500 is a supervised flow method building on RAFT 40 by adding additional pretraining novel mixture of Laplace loss, and improved initialization of the flow estimation.
on Airtan 49
DPFlow 33 is a very recent supervised method that leverages architectural advances to train against high-resolution flow data, leading to improvements relative to EEA-RAFT in many evaluations.
CoTracker-v3 2 supervised multi- model. We evaluate two-frame model, since the multi-frame evaluation protocol is not comparable to Opt-CWM or the other baselines.
Results. We present our main results in Table 1 Opt-CWM outperforms all other self- eelf-super baselines for all datasets, as well as the supervised methods in most cases. In particular, Opt-CWM especially improves upon AD, demonstrating robustness even in difficult (though more realistic) cases with long frame gaps or high motion. The gap is especially large on real-world datasets such as DAVIS, where the baselines struggle with videos violating the heuristic assumptions for which they were optimized. Our experiments on the synthetic Kubric dataset [18), which is more favorable to methods trained on synthetic data, demonstrate that Opt-CWM has the best performance this out-of-domain scenario.
Qualitatively, Opt-CWM makes strong use of its underlying world model, allowing it to accurately track a point's movement through long frame gaps and complex dynamics, including changes of lighting conditions. SEA-RAFT, Doduo, and SMURF, which lack an explicit dynamic world model, often lose track when the tracked object rotates, when lights turn on or off, or when shadows move (Figure Further qualitative examples, including videos, be found in the supplement.
Ablations and Hyperparameter Analysis. We perform several ablation studies of Opt-CWM. First (Table 2 left), we compare Opt-CWM with a spectrum of types of hard-coded perturbations, representing various forms of unoptimized CWM baseline, and find that learned interventions perform substantially better (see Table 2
left). The highly image-dependent nature of the optimized predicted perturbations is illustrated in the supplement. Increasing input resolution, the multi-mask inference (MM), and multiscale refinement (MS) procedures all improve performance.
We also study the effect of the core hyperparameters of our procedure both in training and inference (Table 3). We find that asymmetric masking during training is critical (which is likely why masked video models with standard masking procedures, such as VMAE 47 do not perform well at flow extraction), but that model is highly stable to parameter choices at inference time.
8

---

Table 2: (Left) Ablations. We evaluate multi-mask (MM) and multiscale (MS), in addition to compar- ing our optimized perturbations ("learned") with the fixed ones ("red square" square") S 44 MM and MS columns indicate the number of masking or zooming iterations. We observe clear improvement on all metrics, highlighting the need for bespoke, in in-distribution counterfactual pertur- bations, multi-mask inference and multi-scale refinement. (Right) Distillation into DPFlow. For fast inference, we distill Opt-CWM into the small and efficient DPFlow architecture by sparsely pseudo- Kinetícs with Opt-CWM. This approach outpeforms the self-supervised SMURF and is competitive with the supervised models, while requiring no labeled training data.
TAP-Vid CFG
AJ1
AD
L
< 6
OA 1
OF1 1
Type
MM
MS
Res.
AJ 1
AD
J
<
0gy 1
OA
1 OFI f
S RAFT A0
69.69
1,43
83.83
81.98
46.08
SEA-RAFT
60
69.89
1.44
84,82
82.00
47.52
learned
10
4
512
47.53
8,73
64.83
80.87
60.74
DPFlow
3
78.09
0.99
87.86
90.19
68.57
learned
4
512
42.85
9.82
59.72
78.55
60.20
learned
10
0
512
32.71
11.98
49.20
79.28
41.45
SMURF 39
65.75
2,40
79.45
82.26
42.65
"OpiCCMM 175M
09.53
1.19
83.75
88385
44.17
learned
2 512
40.51
9.72
58.57
80.34
50.06
CPL Distilled
74.77
1.46
85.03
88.74
55.39
red square
2 512
21.37
18.25
36.31
75.38
27.21
TAP- Vid First
Main Benchmuurk
green square
3
2. 512
30.44
12.72
47.37
76.89
19.10
learned
2 256
37,00
11.62
52.82
81.10
57.84
S
RAFT AM
41.77
25.33
54.37
66.40
56.12
SEA-RAFT 50
43.41
20.18
58.69
66,34
56.23
learned
0 256
21.85
20.55
34.34
78.03
53.10
DPFlow
49.20
16.86
62.51
71.51
60.15
red square
0 256
15.00
23.53
26.30
76.63
18.22
SMURF 39
30.64
27.28
44.18
59.15
46.91
green square
0 256
19.91
19.61
32.73
78.31
36.53
"Gp-CWM 175M
47.53
8.75
64.83
80.87
60.74
OP Distilled
45.55
15.68
59.36
69.27
58.41
Table 3: Mask hyperparmeter variants. (Left) At training time. We train RGB with non- temporally factored masking policies similar to Video-MAB 47). The notation of 55-55 indicates 55% of patches are masked in the first frame and 55% are masked in the second frame. Tube masking selects patches at the same spatial location over time, whereas random independently samples patches in each frame. MAE-style masking during training is strictly worse than the temporally-factore masking policy we use as the standard in this paper (shown for reference in the bottom row). All experiments here use 256x256 resolution, MM-3 and MS-2. (Right) At inference. We evaluate 512 resolution y RGB across various masking ratios for the second frame using the MM-3 and MS-2 setting. The standard masking ratio for all results in this work is included as 90% (Ref.) in this table.
Mask
Train
Test
AJ
f
AD
< 6m
1
OA
OF1 f
Masking Ratio AJ f AD 1 < 0gg f OA f OFI f
tube
55-55
0-90
23.94
15.61
36.90
72.19
52.36
50%
42.78
10.52
58.78
79.18
60.68
tube
75-75
0-90
22.55
15.86
39.63
58.20
52.27
60%
43.28
10.12
59.56
80.33
60,80
70%
43.25
9.72
59.95
81.24
59.68
tube
90-90
0-90
15.23
18.57
32.12
51.98
49.20
80%
42.68
9.44
59.76
81.64
57.53
random
75-75
0-90
29.09
14.64
42.57
73.51
57.06
85%
41.99
9.57
59.58
80.92
54.06
90% (Ref.)
40.51
9.72
58.57
80.34
50.06
random
75-75
0-75
34.06
12.79
47.54
76.07
60.81
95%
37.68
10.57
55.87
79.63
45.00
random 0-90 0-90 37.00
11.62
52.82
81.10
57.80
98%
32.85
13.15
50.48
78.19
41.42
Distillation into efficient architectures RAFT, SEA-RAFT, SMURF, and DPFlow use highly efficient but special-purpose flow architecture, rather than large ceneral-purpost ViTs [2] To isolate the effect of this specific architecture design, we train the DPFlow architecture with pseudo-labels generated by Opt-CWM. Specifically, we take a frame pair for each clip, pseudo-label the motion for I 1% of the pixels, and train DPFlow architecture on this pseudo-labeles dataset. We find that this distilled model outperforms the equivalent self-supervised baseline SMURF and is competitive with the supervised techniques (Table 2 right). This outcome pinpoints that the core reason for Opt- CWM's improved performance is our contribution of the novel optimized counterfactual extraction scheme, and the flexible ability to train on unrestricted data that this approach enables, rather than the ViT architecture as such. It is also a practically useful result, since it enables highly efficient inference using the lightweight DPFlow network.
Limitations and Negative Societal Impact The main limitation of our work is the high compute requirements and long training times of video vision transformers. These models are also costly at inference time, which can limit deployment in resource-constrained environments. We demonstrate a pathway to resolving this by distilling the trained Opt-CWM into small and efficient architectures.

---

This limitation also carries potential negative societal impact due to the significant environmental footprint associated with large-scale training on compute clusters.
Conclusion & Future Work
We have demonstrated the effectiveness Opt-CWM in learning motion concepts, achieving state-of- the-art performance real-world benchmarks. Our paper takes an essential first step in demonstrating the strong quantitative potential of optimized counterfactuals for probing pre-trained video predictors. An important extension of the current work within the domain of scene motion understanding will be to train a multi-frame version Opt-CWM to create the next generation of scalable self-supervised point trackers, as it has been shown that combining information form three more frames can help improve flow and occlusion estimation substantially [25]. We plan to scale Opt-CWM's training data, leveraging the wide availability of videos; and explore various alternatives to the masked autoencoder architecture, such as autoregressive or diffusion- based generative video models.
Equally importantly our twin idcas parameterizing an input-conditonned counterfactual probe generator and bootstrapping the learning of the probe-generator parameters with an end-to-end sparse prediction task are mesk-genrric rather than flow-specific. A key next step will thus be to explore the use of these Opt-CC Opt-CWM methods to create self-supervised estimators for a wide variety of visual quantities, including object segments, depth maps, and 3D shape [ 44].
Acknowledgments and Disclosure of Funding
This work was supported by the following awards: Simons Foundation grant SFI-AN-NC-GB- Culmination-0000098660000 National Science Foundation CAREER grant 1844724, National Science Foundation Grant NCS-FR 21 2123963, National Science Foundation Grant RI 221 1258, Office of Naval Research grant N00014-200 20-1-2889, ONR MURI N00014--1-1-2801,, ONR MURI N00014- 24-1-2748, and ONR MURI N00014-22-1-221
We also thank the Stanford HAI, Stanford Data Sciences, the Marlowe team, and the Google TPU Research Cloud team for computing support.
References
[1] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image trans- formers. arXiv preprint 1.Xiv:2106.08244 2021.
[2] Adrien Bardes, Jean Ponce, and Yann LeCun. Mc-jepa: A joint-embedding predictive architecture for self-supervised learning of motion and content features. arXiv preprint arXiv:2307.12699 2023.
[3] Daniel M Bear, Kevin Feigelis, Honglin Chen, Wanhee Lee, Rahul Venkatesh, Klemen Kotar, Alex Durango, and Daniel LK Yamins. Unifying (machine) vision via counterfactual world modeling. arXiv preprint arXiv:2306. 01828, 2023.
[4] Homanga Bharadhwaj. Debidatta Dwibedi, Abhinav Gupta, Shubham Tulsiani, Carl Doer- sch, Ted Xiao, Dhruv Shah, Fei Xia, Dorsa Sadigh, and Sean Kirmani. Gen2act: Human video generation in novel scenarios enables generalizable robot manipulation. arXiv preprint arXiv:2409. 16283, 2024.
[5] Homanga Bharadhwaj. Roozbeh Mottaghi, Abhinav Gupta, and Shubham Tulsiani. Track2act: Predicting point tracks from internet videos enables generalizable robot manipulation. In European Conference on Computer Vision, pages 306-324. Springer, 2024.
[6] Zhangxing Bian, Allan Jabri, Alexei A Efros, and Andrew Owens. Learning pixel trajectories with multiscale contrastive random walks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 6508 6519, 2022.
[7] D. Butler, J. Wulff, G. B. Stanley, and M. J. Black. A naturalistic open source movie for optical flow evaluation. In A. Fitzgibbon et (Eds.), editor, European Conf. on Computer Vision (ECCV), Part IV, LNCS 7577, pages 61 1-625. Springer-Verlag, October 2012.
10

---

[8] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[9] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In International conference on machine learning. pages 1691-1703. PMLR, 2020.
[10] Carl Doersch, Ankush Gupta, Larisa Markeeva, Adria Recasens, Lucas Smaira, Yusuf Aytar, Joao Carreira, Andrew Zisserman, and Yi Yang. Tap-vid: A benchmark for tracking any point in video. Advances in Neural Information Processing Systems, 35:1361 11/3666016200 2022.
[11] Carl Doersch, Pauline Luc, Yi Yang, Dilara Gokay, Skanda Koppula, Ankush Gupta, Joseph Heyward, Ignacio Rocco, Ross Goroshin, João Carreira, et al. Bootstap: Bootstrapped training for Baacking-any-pone In Proceedings of the Asian Conference on Computer Vision, pages 3257-3274, 2024.
[12] Alexey Dosovitskiy. An image is worth 16x 6 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1190 2020.
[13] Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg. Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 2758-2766 2015.
[14] Christoph Feichtenhofer, Haoqi Fan, Bo Xiong, Ross Girshick, and Kaiming He. A large-scale study on unsupervised spatiotemporal representation learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 3299-3309, 2021.
[15] Daniel Geng, Charles Herrmann, Junhwa Hur, Forrester Cole, Serena Zhang, Tobias Pfaff, Ta- tiana Lopez-Guevara, Carl Doersch, Yusuf Aytar, Michael Rubinstein, et al. Motion prompting: Controlling video generation with motion trajectories. arXiv preprint arXiv:2412.02700, 2024.
[16] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv: 1706.02077,77 2017.
[17] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz MuellerFreitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevie. The "Something Something" Video Database for Learning and Evaluating Visual Common Sense. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 5843-5851, 2017. URL Dataset: Nome-hhne
fittps://200 cww hotam//attttettsooottnnnnnnaasna Something V2.
[18] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, et al. Kubric: A scalable dataset generator. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3749-3761, 2022.
[19] Adam W Harley, Zhaoyuan Fang, and Katerina Fragkiadaki. Particle video revisited: Tracking through occlusions using point trajectories. In European Conference on Computer Vision, pages 59-75. Springer, 2022.
[20] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 16000-16009, 2022.
[21] Allan Jabri, Andrew Owens, and Alexei Efros. Space-time correspondence as a contrastive random walk. Advances in neural information processing systems, 33:19545-19560 2020.
[22] Wei Jiang, Eduard Trulls, Jan Hosang, Andrea Tagliasacchi, and Kwang Moo Yi. Cotr: Correspondence transformer for matching across images. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 6207-6217, 2021.
11

---

[23] Zhenyu Jiang, Hanwen Jiang, and Yuke Zhu. Doduo: Learning dense visual correspondence from unsupervised semantic-aware flow. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 12420-12427. IEEE, 2024.
[24] Rico Jonschkowski, Austin Stone, Jonathan T Barron, Ariel Gordon, Kurt Konolige, and Anelia Angelova. What matters in unsupervised optical flow. In Computer Vision-ECC Eitoon-CCVVAA 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 16, pages 557-572. Springer, 2020.
[25] Nikita Karaev, Iurii Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker3: Simpler and better point tracking by pseudo-labelling real vidcos. arXiv preprint arXiv:2410. 11831, 2024.
[26] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset, 2017.
[27] Liang Liu, Jiangning Zhang, Ruifei He, Yong Liu, Yabiao Wang. Ying Tai, Donghao Luo, Chengjie Wang, Jilin Li, and Feiyue Huang. Learning by analogy: Reliable supervision from transformations for unsupervised optical flow estimation. In Proceedings of the IEEE/CVF conference on computer vision and patiern recognition, pages 6489-6498, 2020.
[28] Shuaicheng Liu, Kunming Luo, Nianjin Ye, Chuan Wang, Jue Wang, and Bing Zeng. Oiflow: Occlusion-inpainting optical flow estimation by unsupervised learning. IEEE Transactions on Image Processing, 20:6420-64333 2021.
[29] Ily Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 03983,9983.. 2016.
[30] Uya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
[31] N. Mayer, E. 1lg, P. Häusser, Fischer, D. Cremers, A. Dosovitskiy, and T. Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow es- timation. In IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2016. http: httt:://lmminnfoonntii... inffrmatii.....mmm uni uni-freiburg, htt/Pullicationss//00 MIFDB16 1eXiv11512.022
[32] Lukas Mehl, Jenny Schmalfuss, Azin Jahedi, Yaroslava Nalivayko, and Andrés Bruhn. Spring: high-resolution high-detail dataset and benchmark for scene flow, optical flow and stereo. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4981-4991, 2023.
[33] Henrique Morimitsu, Xiaobin Zhu, Roberto M Cesar Jr. Xiangyang Ji, and Xu-Cheng Yin. Dpflow: Adaptive optical flow estimation with a dual-pyramid framework. arXiv preprint arXiv:2503. /4800, 2025.
[34] Jordi Pont-Tuset, Federico Perazzi, Sergi Caclles, Pablo Arbeláez, Alex Sorkine- Sorkine-Horunn,, and Luc Van Gool. The 2017 davis challenge on video object segmentation, 2018 URL
https://arxixs org/ aba/ /00./ 00675
[35] Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Yang, Huisheng Wang, Serge Belongie, and Yin Cui. Spatiotemporal contrastive video representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6964-6974, 2021.
[36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Pro- ceedings of Machine Learning Research, pages 8748-8766 8763. PMLR, 18-24 Jul 2021. URL. bttts://proceedingg mlr. gresss/v19/rrattttt
html
12

---

[37] Ayush Shrivastava and Andrew Owens. Self-supervised any tracking by contrastive random walks. In European Conference on Computer Vision (ECCV). 2024. URL https 7/arxiv.org www.oogg///bo.o. has/abbs/2400 2209. 16288
[38] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos the wild. arXiv preprint wvrXiv:/122.444 2012.
[39] Austin Stone, Daniel Maurer, Alper Ayvaci, Anelia Angelova, and Rico Jonschkowski. Smurf: Self-teaching multi-frame unsupervised raft with full-image warping. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pages 3887-3896, 2021.
[40] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pages 402-419. Springer, 2020.
[41] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for supervied video pre-training Advances in neural information processing systems, 35:10078-10093, 2022.
[42] Narek Tumanyan, Assaf Singer, Shai Bagon, and Tali Dekel. Dino-tracker: Taming dino for self-supervised point tracking in single video. arXiv preprint arXiv:2403.14556 2024.
[43] Mel Vecerik, Carl Doersch, Yi Yang, Todor Davchev, Yusuf Aytar, Guangyao Zhou, Raia Hadsell, Lourdes Agapito, and Jon Scholz. Robotap: Tracking arbitrary points for few-shot visual imitation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 5397-5403. IEEE, 2024.
[44] Rahul Venkatesh, Honglin Chen, Kevin Feigelis, Daniel M Bear, Khaled Jedoui, Klemen Kotar, Felix Binder, Wanhee Lee, Sherry Liu, Kevin A Smith, et al. Understanding physical dynamics with counterfactual world modeling. arXiv preprint arXiv:2312.0672 2023.
[45] Carl Vondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, and Kevin Murphy. Tracking emerges by colorizing videos. In Proceedings of the European conference on computer vision (ECCV), pages 391-408, 2018.
[46] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks for action recognition in videos. IEEE transactions on pattern analysis and machine intelligence, 41(1 (((1))274002755, 2018.
[47] Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 14549-14560, 2023.
[48] Qianqian Wang, Xiaowei Zhou, Bharath Hariharan, and Noah Snavely. Learning feature descriptors using camera pose supervision. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28. 2020, Proceedings, Part 16, pages 757-774. Springer, 2020.
[49] Wenshan Wang, Delong Zhu, Xiangwei Wang, Yaoyu Hu, Yuheng Qiu, Chen Wang, Yafei Hu, Ashish Kapoor, and Sebastian Scherer. Tartanair: A dataset push the limits of visual slam. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4909 4916. IEEE, 2020.
[50] Yihan Wang. Lahav Lipson, and Jia Deng. Sea-raft: Simple, efficient, accurate raft for optical flow, arXiv preprint arXiv:2405. 14793, 2024.
[51] Jiarui Xu and Xiaolong Wang. Rethinking supersupe correspondence learning: A video frame-level similarity perspective. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10075-10085, 2021.
[52] Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang. and Thomas Huang. Youtube-vos: A large-scale video object segmentation benchmark, arXiv preprint 10X.v3180.003327, 2018.
13

---

[53] Chengbo Yuan, Chuan Wen, Tong Zhang, and Yang Gao. General flow foundation affordance for scalable robot learning. 8th Annual Conference on Robot Learning, 2024.
[54] Miao Zhang, Jie Liu, Yifei Wang. Yongri Piao, Shunyu Yao, Wei Ji, Jingjing Li, Huchuan Lu, and Zhongxuan Luo. Dynamic context-sensitive filtering network for video salient object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1553-1563, 2021.
[55] Yi Zhou, Guillermo Gallego, Xiuyuan Lu, Siqi Liu, and Shaojie Shen. Event-based motion segmentation with spatio-temporal graph cuts. IEEE transactions on neural networks and learning systems, 34(8):4868-488 -880, 2021.
14

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the
paper's contributions and scope?
Answer: [Yes]
Justification: In the Abstract and Introduction we claim that we achieve "state-of- setate-of-the-ase performance for motion estimation real-world videos while requiring labeled data." This evident in Table our Experiments section (4).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have discussed the limitations in the paper, including section at the end of the Experiments section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide thorough implementation details in the method section and in the supplement. The accompanying code we will provide in the supplement can ensure reproducibility. We use standard publicly available data.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We standard datasets that are publicly available. We include code and instructions in the supplement, and will make this code publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
Soolic//guides/Codettbbmsss wtw.eeeuubisssiooooooooooooo for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wttps/cccppclcccuui.eeeeeeccc
wtteeee//ooSSSSSese..ee.e www.ssssiicggcoooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: These details are included the Method section, Experiment section, and the Training section of the Supplement document.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The computational cost of training self-supervised video models prohibits us from running multiple experiments and calculating error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The compute resources needed to reproduce the experiments are reported in
the Supplementary document.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The paper conforms with the Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss this in a section at the end of the Experiments section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We do not release any new data assets with this paper, and appropriately credit the data sources we use for training.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not include crowdsourced answers.
Guidelines:
The answer NA means that the paper doess not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
We did use any human subjects in our research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We do use LLMs this paper.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: //apps:///eurpp..... http://urrrrrrccoooo/eeeree////p cc//oomferences//222//LL. for what should or should be described.
21

---

A
Implementation Details
A.1
Architecture Details
A.1.1
RGB
The input video is first divided into non-overlapping spatiotemporal patches of size 8 x 8, with subset of patches masked. Unlike MAE, we train with both revealed input patches and mask tokens provided to the encoder. We train with the ViT-B architecture [20] with each transformer block consisting of multi-head self-attention block and an MLP block, both using LayerNorm (LN). The CWM decoder has the same architecture as the encoder. Fach spatiotemporal patch has a learnable positional embedding, which is added to both the encoder and decoder inputs. CWM does not use relative position or layer scaling [ 220 Please refer to [44 31 for more details on the architecture. The 175M CWM model is based on ViT-B [2] but has twice the number of total layers. The IB CWM model is similar to the 175M model, but has 48 layers with an embedding dimension of 2048 and 16 heads.
Default settings We show the default pre- pre-training settings in Table CWM does not use color jittering, drop path, gradient clip. Following ViT's official code, Xavier uniform is used initialize all transformer blocks. The learnable masked token is initialized as a zero tensor. Following MAE,
use the linear Ir scaling rule: Ir base_lr batch_size 256 20
Table 4: Default pre-training setting of CWM
config
value
optimizer base learning rate
AdamW [30]
1.5e-4
weight decay
0.05
optimizer momentum
31, .3
0.9, 0.95
accumulative batch size
4096
learning rate schedule
cosine decay 29
warmup epochs [16]
40
total epochs
800
flip augmentation augmentation
no
MultiScaleCrop [46
A.1.2
T flow
The architecture the flowccodditioeed predictor, fl is a vision transformer with 16 layers and 132M parameters. Input images are resized to 224x224, and the patch size is 8. Sinusoidal positional encodings are used. For the encoder, the embedding dimension 768, and 12 attention heads are used. For the decoder, the embedding dimension is 384, and 6 attention heads are used.
This model has two parallel "streams". the first of which takes RGB input and the second of which takes sparse flow, concatenated with RGB (which is masked to have the same sparsity as the flow), as input. All RGB inputs are from the first frame only; this requires the model depend solely on flow to modify the RGB and predict the next frame.
The transformer architecture then applies self-attention to each stream and cross-attention between streams. The encoder has 12 layers, split into three groups of 4. In each group, there is one layer with self-attention each stream and cross-attention from each stream to the other, followed by three layers with only self-attention on the first stream. The decoder has 4 layers; the first applies self-attention to each stream and cross-attention from each stream to the other; the second applies self-attention to the first stream and cross-attention from the second stream to the first; and the final two only apply self-attention the first stream.
A.2 Training Details
A.2.1
RGB
We train CWM 256 resolution for 800 epochs and finetune at 512 resolution for 100 epochs by interpolating the positional embeddings. It takes approximately 4 days to train 800 epochs TPU
22

---

v5-128 pod. We pre-train CWM on the Kinetics-400 dataset (26], without requiring any specialized temporal downsampling.
We train CWM 1B at 256 resolution for 200,000 iterations with a batch size of 512 on a custom video dataset called BVD (approximately equivalent 400 kinetics epochs). We use the AdamW optimizer, with norm clipping 1.0 and weight decay 0.1. We warmup the learning rate over 2,000 steps to a peak of 3e-4, then linearly decay over the next 198,000 steps. Training takes approximately day on 64 H100 GPUs.
A.2.2
L flow and FLOWa
We train fl and FLOW, jointly using an AdamW optimizer with weight decay of 0.05, betas of (0.9, 0.95), and a learning rate schedule with max learning rate 1.875 x 10 10-3, 40 warmup epochs (10% of total training epochs), and cosine decay. We used a batch size of 32, training on the Kinetics-400 dataset (26).
A.3 Training Data
We construct training dataset called BVD (Bid Video Dataset) which consists of publicly available datasets such as Kinetics400 [26] and Somethinggomething/2 [M along with other publicly avail- able videos. We filter the videos based on CLIP [6) categories to remove thumbnails and videos with a lot of text in the frame. We additionally filter the videos based on optical flow to remove videos with little motion such as slide shows, or mostly static videos.
A.4 Inference Techniques
A.4.1 Multi-Mask
In the process of computing flows in FLOWa, at inference time, we take an argmax over the difference between the predicted next frame with and without the counterfactual perturbation. This difference image, A, depends on the choice of the random mask as this mask is used by v for the next-frame reconstruction. As discussed in the main text, a random mask reveals patches too close to where the perturbation should be reconstructed, the predictor T R may not reconstruct the perturbation properly, and the difference image will be noisy and diffuse, preventing the model from accurately predicting the next-frame location. Additionally, the reconstructed pixels will not necessarily be the same across different random samplings of visible patches, which may add random noise to the difference image. Both of these issues ameliorated by our multi-mask technique, in which compute difference images for variety of sampled random masks (we found 10 to be a good number of masks for multi-masking), average over the difference images, and then take the argmax of this averaged Aay to compute next-frame location for determining flow.
A.4.2 Multiscale
Multiscale refinement of the original flow prediction improves Opt-CWM's performance, as observed in Figure 5] Given an input frame pair and an initial flow prediction, we perform iterative multiscaling through the following procedure. At each "zoom iteration", we take a 0.75H 0.75W crop of the input frames with original height H and width W. We center the second frame crop on the location predicted by the previous iteration.
The transformer-basee architecture of the next frame predictor y RGB imposes a limit to the input resolution, which may occasionally prevent small objects or minute features of the input frame from being accurately reconstructed in great detail. Multiscale refinement of the initial flow prediction can be greatly beneficial under these circumstances. However, Figure 5 suggests that the improvement is not monotonic; indeed, excessive cropping may lead to the loss of global context that is necessary to accurately reconstruct the scene. Opt-CWM is run on 4 zoom iterations, which we have empirically found to be optimal.
A.4.3 Occlusion Estimation
The difference image A can also be used to predict whether visible point becomes occluded in the next frame. Conceptually, described in Section 3 in the main text, when a point becomes occluded,
23

---

Total visible points within threshold
zo0m, @ters
200m,
200m_ _ters4
zoom, Iterss
iters8
0.6
0.4
0.0
16
Figure 5: < davg broken down across thresholds (x-axis). Fraction of points with error less than fixed threshold, as function of number of multiscale (MS) iterations, for pixel thresholds 1, 2, 4, and 16. We find that 4 zoom iterations tends perform the best, especially for robustness on difficult examples (evidenced by better performance on higher thresholds).
the counterfactual perturbation placed the object should not be reconstructed in the second frame. Thus, while we take argmax A to compute flow, we can instead use max A as signal for occlusion. In particular, we compare max A to some threshold face to predict occlusion (i.e., consider the model to have predicted that point becomes occluded if and only if max < too)).
In the multi-masking setting with 10 masking iterations, we have 10 difference images: 41, 42. Aiu. Instead of thresholding the average, Aavg. we can get an improved signal by considering max A; for each i I,.., 10. In this setting, we found that checking Iū provided a good signal for predicting occlusion, and this prediction criterion what was evaluated in the OA and OFI metrics of Table in the main paper.
- Ei max A; < 0.05
B Additional Quantitative Results
B.1 Constant Frame Gap Protocol
TAP-Vid Constant Frame Gap (CFG). For fair comparison with optical flow models, we also propose an additional protocol with fixed frame gaps that more advantageous for these baselines (see supplementary for the effect of frame gap on flow baselines). In particular, fixed 5-frame gap is used: metrics are computed on all trame-pairs that are frames apart (and the point is visible in the first).
We show results for this protocol in Table 5 to optical-flow based models, Opt-CWM largely outperforms all unsupervised methods and is competitive with state-of-the-art fully supervised methods.
We observe that in this setting, which is favorable
B.2 Precision Analysis
Figure 6 alttempss to explain the high performance of Opt-CWM on TAP-Vid First through a similar analysis done in Section A.4.2 Our best- pest-performin model (with optimal inference-tine configura- tions) is able to predict the next frame location within 16 pixels of the ground truth for over 85% of the total number of visible points. Unlike baseline models, Opt-CWM is able to predict most points within a reasonable boundary. Further, Opt-CWM predictions are precise; it predicts the majority of the query points within 2 pixels of the ground truth. While SEA-RAFT, which is supervised, is also precise for lower thresholds, the magnitude of the error for wrong predictions is evidently higher, as its performance quickly plateaus for higher thresholds.
As discussed in Section 4 in the main paper, we further evaluate on a custom constant-frame gap protocol (CFG) for fairer comparison with optical flow baselines. As shown here in Figure 7 all models improve significantly under this less challenging setup. In particular, optical flow baselines

---

Table 5: Quantitative results on ViP-Vid dataset (Constant five-Frame Gap (CFG). In the CFG protocol, point tracking is evaluated at fixed gaps of frames, making it an easier setting that is more favorable optical flow methods. "S" and "U" indicate supervised and unsupervised, respectively. Doduo is not strictly unsupervised as it uses segmentation labels. GMRW is trained on the Kubric dataset, (marked with f), making it more favorable evaluation setting for that method because of the minimal domain gap. Best performing supervised models (shaded) are considered separately.
OFL
OA
Collaaderwy
14.49
1.21
75.08
19.45
95.05
36.11
78:15
1.02
89.07
52.90
RAFT 40
59.60
1.43
$3.63
51 1.
46.08
79.01
0.36
67.59
92.75
49.49
73.58
1.34
83.73
91.00
63.17
SEA RAFT MT
19:89
1.44
$4.82
12.00
47.52
75.12
1,57
85.82
55.90
34.42
77.53
1,00
$7.00
92.50
06.05
DFFlcv
0.99
90.19
80.07
95.85
Dodao
25
25.61
1.61
72.56
$7.49
22.59
35.26
1.19
77.62
43.00
11.63
56.57
1.74
65.63
$7.26
55.0L
GMMOW
51.28
5.11
7301
40.31
75.04
1.23
88.89
411
1111
51991
2,40
82.36
78.76
97.16
28.13
CWM
25.56
3.65
38.35
68.00
3.41
38.00
3.88
45.30
65.17
5.05
30.35
4.05
22.33
$8.44
4227
01966
87.84
181
5482
28.15
0,800
M2.43
exhibit strong sub pixel precision. However, we see that in general, compared self-supervised baselines, Opt-CWM make reasonable predictions of a point's next frame location more often, at rate comparable to the fully supervised SEA-RAFT.
pet points
0.
65A
DNN
0.6
02
Figure 6: TAP-Vid First: comparing baseline models < dang broken down across thresholds (I-axis). Fraction of points with error less than a fixed threshold, as a function of baseline model. Compared to baseline models, Opt-CWM maintains high performance on all thresholds even when making predictions across large frame gaps, as is necessary for TAP-Vid First.
Total pct visible points wi within threshold
1.0 Smart
Dodio
REA- RAFT
0.0
ONM
02
0.0
Figure 7: TAP-Vid CFG: comparing baseline models on < Savg broken down across thresholds (z-axis). Fraction of points with error less than fixed threshold, function of baseline model. For fair comparison, we also evaluate on a constant frame gap setting that is more favorable to optical flow baselines. While baseline methods show strong performance for very low thresholds (<: pixels), we see that in general Opt-CWM outperforms self-supervised methods and is comparable with RAA- in predicting more points within a reasonable boundary.
25

---

B.3 Perturbation Across Epochs
The performance of FLOWg is greaty dependent on the quality of its learned Gaussian perturbations. In Figure [ we see that the appearance of the perturbation evolves alongside the training of Opt-CWM. As the perturbation converges into optimal patch bespoke for the input frame, the quality of the flow prediction improves in tandem.
predicted per perturaliion
frame pair
A
A
A
error: 14.38
arro::
Training g Epochs
15
165
Figure 8: Evolution of perturbations across training epochs: We observe how the predicted perturbations change as the model trains. The perturbation starts as a disjoint streak of colors and converges to localized peak. This in turn increasingly concentrates the difference image A and leads to better flow prediction. Green the ground truth flow obtained from the TAP-Vit Vid dataset, and blue is our model's prediction.
Amplitade
Dev.
Amplltdd
Shi
Figure 9: Perturbation maps reflect scene properties. For two example frame pairs, we show the amplitudes and standard deviations, at each spatial position and for each color channel, of the optimal Gaussian perturbations predicted by MLP, These "perturbation maps" emergently reflect scene properties, with perturbation parameters varying in size and magnitude depending on where they are located in the image, corresponding the presence of foreground objects and their parts.
B.4 Comparison with DINO- Tracker
DINO-Tracker
test-time training approach using pre-trained DINO features that shows promising results on real-world videos. For fair comparison with Opt- CWM and other baselines, we constrain DINO-Tracker, a multi-frame tracker, to run under the same two-frame constraint. Compared with Opt-CWM on TAP-Vid DAVIS, DINO- Tracker obtains an average distance (AD 1) 5.91 (vs. 7.70) and a score of 72.17 (vs. 68.63) on the 8 t metric. However, note that DINO-Tess Tracker is not directly comparable as a baseline for Opt-CW as it requires per-video test- time optimization and relies on flow predictions from RAFT [40], which is supervised. In contrast to this, Opt-CWM is feed-forward and completely self-supervised.
We compare our results with DINO-Tracker
42
26


Reviews:
[{"review_id": "XWEhjv0UVK", "reviewer": "Reviewer_ix1k", "summary": "This work explores a new idea for how to extract motion-primitives from large pretrained video models, specifically focusing on optical-flow and occlusion. The authors develop a method Opt-CWM which estimates optical flow and occlusion from pretrined video prediction models by leveraging counterfactual probes, which are input-conditioned perturbations to the appearance or position of visible patches. The authors develop a method for generating these probes in an optimal manner and show that the probe-generation parameters can be learned with a self-supervised method. Given, this Opt-CWM is shown to be SoTA in motion estimation.", "questions": "See above weaknesses if these are addressed I'm willing to raise my score.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 4, "strengths_and_weaknesses": "Strengths\n1. The paper is well-written and clear on the contributions.\n2. The method for learning optimiized counterfactuals via using the flow prediction as a bottleneck is interesting and novel to my knowledge and approaches this problem in a different way than many existing methods.\n3. The ablations are quite ocmprehensive and appreciated. \n\nWeaknesses\n1. The authors state that most recent works in point-tracking are semi-supervised (like BootsTAP, TAPNext etc.) and thus are out of scope for comparison. While I appreciate this, I feel it would be disingenuous not include some of these results in the paper for comparison to show where the SoTA is for these tasks. Methods like BootsTAP or TAPNext are extremely scalable because they use pseudo-labeling. Therefore, one might ask if the goal is to solve point-tracking through occlusions etc. generally then what is the inherent benefit of using something self-supervised if the performance is still drastically worse than those methods (see numbers in TAPNext paper Table 1 for example)? The main driver of self-supervised learning to me is it's generality and scalability to places where finding supervision is hard or not clean, but if semi-supervised learning can bridge this gap effectively should these tasks (e.g. TAP-Vid) be the main demonstration of the benefits of this approach?\n2. The authors miss some key citations of work that needs to be compared to. For example DINO-Tracker from Tumanyan, Singer et al (ECCV 2024) is also self-supervised and seems to present quantitative results that are far better than what is reported in this paper. Can the authors include this comparison or comment on why it is not necessary? \n3. There are some discrepancies in the numbers reported in this work vs. the literature. For example for DAVIS AJ and delta_avg, the authors report GMRW as 36.47 and 54.59. However in the GMRW paper they report: 41.8 and 60.9 . These kinds of discrepancies should be clarified in the work otherwise comparison across works becomes quite tricky. Is there something about the evaluation methodology used that differs from prior work for example?", "was_revised": true, "final_justification": "The authors sufficiently addressed my concerns in the paper and made key clarifications that now highlight the impact of their results as being relatively significant in context of prior work. Given similar addressing of the concerns of other reviewers I believe this work now meets the criteria for a solid acceptance (5) as it is novel and has the potential to lead to advancement in the field of SSL-based video representation learning.", "rebuttal": "**R1 Q1**: *The authors state that most recent works in point-tracking are semi-supervised (like BootsTAP, TAPNext etc.) and thus are out of scope for comparison. While I appreciate this, I feel it would be disingenuous not include some of these results in the paper for comparison to show where the SoTA is for these tasks. Methods like BootsTAP or TAPNext are extremely scalable because they use pseudo-labeling. Therefore, one might ask if the goal is to solve point-tracking through occlusions etc. generally then what is the inherent benefit of using something self-supervised if the performance is still drastically worse than those methods (see numbers in TAPNext paper Table 1 for example)? The main driver of self-supervised learning to me is it's generality and scalability to places where finding supervision is hard or not clean, but if semi-supervised learning can bridge this gap effectively should these tasks (e.g. TAP-Vid) be the main demonstration of the benefits of this approach?*\n\n**R1 A1**: We agree that this is important to achieve good results when compared to methods trained with more supervision. We believe we have achieved this, although it may be confusing since multiple protocols exist in the literature, e.g. Table 1 in the TAPNext paper reference uses a different (and somewhat easier) protocol than the one we report. On the standard but more relatively challenging benchmark that we report  (please refer to evaluation details below), our method is SoTA, even compared to the semi-supervised methods like CoTracker, and performance numbers for all methods are lower overall. \n\n*Why self-supervised?* Despite the recent strong results of semi-supervised learning, it is still fundamentally limited because it leads to a limited exploration of the possible label space of point tracks. Specifically, pseudo-labeling will further reinforce what the model can already predict confidently. Parts of the data distribution that are difficult to synthesize with graphics rendering engines (and therefore will not be in the initial training dataset before any pseudo-labeling is done) will not be learned with pseudo-labeling. Examples include non-rigid deformable motions (e.g., fabric, dough), or volumetric/particle motion (e.g., liquids, smoke, dust). Self-supervised learning through predictive modeling, like the Counterfactual World Modeling paradigm that we build upon in this paper does not have this limitation of pseudo-labeling and can learn such challenging scenarios from data.\n\n*Evaluation details*: We follow the more challenging “first” protocol for TAPVid evaluation, whereas the TAPNext paper follows the easier “strided” protocol, leading to higher reported tracking performance. Briefly, in the “first” protocol, points are tracked from the first time they appear through the end of the video. In the \"strided\" protocol, a point is queried every 5 frames if it is visible and tracked forward and backward in time. First is more challenging because on average points are tracked over longer temporal frame gaps. For more information refer to Appendix H of the original TAPVid paper. We also run all models on pairs of query and target frames to track a point through a video. This is (1) to remove the effect of additional algorithmic heuristics on how to chain frame-pair predictions into tracks and to reveal the capability of the underlying motion representation, and (2)  because this represents a fundamental unit of the problem of motion understanding–tracking directly emerges from estimating the motion/occlusion of a point across two frames\n\n---\n\n**R1 Q2**: *The authors miss some key citations of work that needs to be compared to. For example DINO-Tracker from Tumanyan, Singer et al (ECCV 2024) is also self-supervised and seems to present quantitative results that are far better than what is reported in this paper. Can the authors include this comparison or comment on why it is not necessary?*\n\n**R1 A2**:  We thank the reviewer for raising this important prior work, we will make sure to add this paper to the related work. Regarding a comparison, DINO-tracker is not a strictly self-supervised method because it assumes an off-the-shelf RAFT optical flow model to derive pseudo-labels, which has been trained with explicit optical flow supervision. In contrast, our approach is only supervised by a next frame pixel prediction loss. We will include a comparison with DINO tracker in the final version.\n\n---\n\n**R1 Q3**: *There are some discrepancies in the numbers reported in this work vs. the literature. For example for DAVIS AJ and delta_avg, the authors report GMRW as 36.47 and 54.59. However in the GMRW paper they report: 41.8 and 60.9 . These kinds of discrepancies should be clarified in the work otherwise comparison across works becomes quite tricky. Is there something about the evaluation methodology used that differs from prior work for example?*\n\n**R1 A3**:  We thank the reviewer for raising this point. As in the **R1 A1** response, we would like to clarify that in this paper we follow the more challenging “first” protocol for TAPVid evaluation, whereas the GMRW paper follows the simpler “strided” protocol, leading to higher reported tracking performance. As discussed, our more challenging evaluation protocol evaluates models' capabilities at the fundamental problem of estimating the motion of a point across a pair of frames."}, {"review_id": "SSNW62g7FD", "reviewer": "Reviewer_YyNn", "summary": "This paper proposed a new self-supervised learning based method, Opt-CWM, for learning motion concepts. It solves the limitations of an existing self-supervised learning based method method, CWM, in this field with two innovations. First, it propose to parameterize a counterfactual probe policy generator with a learnable neural network that can predict situation-specific probes that take into account the appearance context (both local and global) around target points to be tracked, and thus can be less out-of-distribution than hand-coded probes. Second,  it proposes to learn the probe generator in a principled fashion without relying on any supervision from labeled data or heuristics. Experiments shows the proposed method significantly improves performance of multiple motion concept prediction tasks.", "questions": "See the weakness above.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The proposed method is technically sound. \n- Experiments show significant improvement over existing methods. \n- The proposed approach is well motivated. \n\nWeakness\n- The proposed method uses two large base predictors, one with 175M and the other 1B learnable parameters, which makes the comparison not very fair over the existing methods. \n- Due to the introduce of the large models, the inference time of the proposed method should be much higher than the existing methods. It would be great to see the comparison on this axis. \n-  To address the limitation on high inference time, it is stated that \"We demonstrate a pathway to resolving this by distilling the trained Opt-CWM into small and efficient architectures\". However, I do not see such results.\n- What is the impact of the scale of the base predictors? Currently two models of 175M and the other 1B are employed. How about using smaller or even larger models? It would be great the see the trend of the performance with respect to the model size. \n- Many sentences are too long and complex, making the paper hard to read and follow.", "was_revised": false, "final_justification": "", "rebuttal": "**R2 Q1**: *The proposed method uses two large base predictors, one with 175M and the other 1B learnable parameters, which makes the comparison not very fair over the existing methods.*\n\n**R2 A1**:  We acknowledge that the capacity of our model differs from some baselines. Our base CWM is a general‑purpose video prediction foundation model (used here to extract motion, but can also be used to extract depth/segments), which justifies the larger parameter count. Regarding fairness at inference time, we follow a standard approach: distill Opt‑CWM into a small, task‑specific optical‑flow model. Concretely, Table 2 (right) shows that SEA‑RAFT distilled from Opt‑CWM (trained only on pseudo‑labels, no ground‑truth flow) outperforms SMURF (self‑supervised) and is competitive with supervised RAFT, while running at the small‑model speed. We summarize these results below, and present an additional experiment where we distill Opt-CWM into the newer DPFlow [A] model, which achieves better performance. It is also worth noting that this distillation procedure can also be further tuned and result in higher final performance. We will communicate this more directly in the paper and discuss compute cost details.\n\n| Model | AJ | AD | $<\\delta_\\text{avg}^x$ | OA | OF1 |\n|  :--- | :---: | :---: | :---: | :---: | :---: |\n| RAFT | 41.77 | 25.33 | 54.37 | 66.40 | 56.12 |\n| SEA-RAFT | 43.41 | 20.18 | 58.69 | 66.34 | 56.23 |\n| SMURF | 30.64 | 27.28 | 44.18 | 59.15 | 46.91|\n| Opt-CWM 175M | 47.53 | 8.73 | 64.83 | 80.87 | 60.74 |\n| SEA-RAFT (distilled) | 44.05 | 17.49 | 57.93 | 69.75 | 60.72 |\n| DPFlow (distilled) | 45.55 | 15.68 | 59.36 | 69.27 | 58.41 |\n\n[A] DPFlow: Adaptive Optical Flow Estimation with a Dual-Pyramid Framework - Henrique Morimitsu, Xiaobin Zhu, Roberto M. Cesar Jr., Xiangyang Ji, Xu-Cheng Yin\n\n------\n\n**R2 Q2**: *Due to the introduce of the large models, the inference time of the proposed method should be much higher than the existing methods. It would be great to see the comparison on this axis.*\n\n**R2 A2**: The size of the base predictor model does in fact lead to slower inference speed. We describe our distillation-based solution to this added inference time cost in **R2 A1** which will have the same inference-time speed as standard optical flow methods. We commit to adding inference time details in the final version.\n\n------\n\n**R2 Q3**: *To address the limitation on high inference time, it is stated that \"We demonstrate a pathway to resolving this by distilling the trained Opt-CWM into small and efficient architectures\". However, I do not see such results.* \n\n**R2 A3**: Thank you for raising this, those results are in Table 2 (right) of the main text, but we realize our text could emphasize them better. Please also refer to our discussion in **R2 A1**.\n\n------\n\n**R2 Q4**: *What is the impact of the scale of the base predictors? Currently two models of 175M and the other 1B are employed. How about using smaller or even larger models? It would be great to see the trend of the performance with respect to the model size.*\n\n**R2 A4**: We acknowledge that establishing scaling laws for base predictors is an important topic. Within our compute budget, we choose to train and evaluate at two orders of magnitude (175M, 1B) and show improvements of  47.53 $\\to$ 51.88  for AJ (higher is better) and 8.73 $\\to$ 7.70 for AD (lower is better).  A complete scaling study (smaller/larger CWMs) is a promising direction and we plan to include it in future work.\n\n------\n\n**R2 Q5**: *Many sentences are too long and complex, making the paper hard to read and follow.*\n\n**R2 A5**: Thank you for this helpful feedback, we will revise the draft for clarity and improve the sentence length and structure."}, {"review_id": "M0k43mfc0j", "reviewer": "Reviewer_5xm4", "summary": "This paper proposes Opt-CWM, a novel self-supervised approach for motion concept learning from videos. By leveraging large-scale pretrained video prediction models, the authors introduce optimized, learnable counterfactual probes that extract high-quality optical flow and occlusion information from real-world videos without any labeled data. The core innovations are: (1) parameterizing the counterfactual probe generator with a neural network, allowing probes to adapt to image content and context, and (2) a self-supervised learning objective that trains the probe generator via a sparse prediction information bottleneck, eliminating the need for heuristic losses or supervision.", "questions": "1. Could the authors provide more detailed analysis or ablation experiments showing how the model distinguishes between perturbations that correspond to moving/visible objects versus those that are static or occluded? Is there a theoretical or empirical criterion to ensure that significant model output changes are reliably associated with true object motion, and minimal changes indicate occlusion or background?\n2. The current layout places important figures (e.g., Figures 1 and 2) far from their detailed descriptions in the main text, which hinders readability. I recommend the authors to consider reorganizing the paper so that figures and their explanations appear together. \n3. Figure 2 does not fully reflect the described functional form in the main paper. In addition, it omits some input variables (such as $I_2$, which may cause confusion about the model’s actual input-output structure.\n4. Does Opt-CWM achieve true optimality in generating counterfactual probes, or are there cases where it may still be suboptimal? Is there any analysis or discussion of the theoretical limits of the proposed approach?\n5. What is the computational complexity of Opt-CWM compared to existing baselines? Is the method practical for large-scale or real-time applications?\n6. Can the authors provide a detailed explanation of all symbols and notations used in the tables, to enhance readability and reproducibility?", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: This paper introduces a novel approach to self-supervised motion and occlusion estimation by leveraging optimized counterfactual probes, which are parameterized by a neural network and learned entirely without supervision. Unlike traditional CWM that relies on fixed, hand-crafted perturbations, Opt-CWM framework utilizes learnable, adaptive perturbations that are dynamically generated based on image content and context. The method achieves state-of-the-art empirical performance on several challenging benchmarks, consistently outperforming both established supervised methods and several self-supervised baselines.\n\nWeaknesses\n1. The paper lacks a clear and principled explanation for how the model determines whether an added perturbation will be meaningfully carried forward with the moving object in the model’s prediction.\n2. The manuscript suffers from poor layout and figure placement. Important figures (e.g., Figure 1 and Figure 2) are physically distant from their detailed explanations, making it difficult for readers to follow the narrative and connect visuals with the text.\n3. While the paper points out that previous CWM methods are suboptimal due to their fixed, hand-crafted perturbations, it does not provide sufficient discussion or evidence on whether Opt-CWM achieves optimality or merely improves upon prior suboptimal solutions.\n4. The paper does not provide an analysis of the computational complexity for either training or inference.\n5. Symbols and formatting (e.g., underlines, arrows) in the result tables are not explained, which could cause confusion for readers unfamiliar with these conventions.", "was_revised": true, "final_justification": "Thank you for addressing my questions. I have increased my overall rating for the paper.", "rebuttal": "**R3 Q1**: *Could the authors provide more detailed analysis or ablation experiments showing how the model distinguishes between perturbations that correspond to moving/visible objects versus those that are static or occluded? Is there a theoretical or empirical criterion to ensure that significant model output changes are reliably associated with true object motion, and minimal changes indicate occlusion or background?*\n\n**R3 A1**:  To clarify, we briefly re-state the key “theory” behind our approach. Our counterfactual compares the predictor’s next‑frame output with vs. without a small, image‑conditioned perturbation localized at the query pixel in the first image. The resulting difference image $\\Delta$ reveals where and how strongly the model carries information from that pixel into the next frame.\n\nThis concept leads to different behavior for different points, depending on the condition:\n- **If the point is visible and moving...** then the perturbation appears at a new location in the next frame; $\\Delta$ exhibits a **sharp, high‑magnitude peak** with **some displacement** from original coordinates.\n- **If the point is visible and static...** then the perturbation appears in the same position in the next frame; $\\Delta$ has **sharp, high magnitude peak** but with **near‑zero displacement**.\n- **If the point becomes occluded/out‑of‑view...**  then the model cannot “carry” the perturbation forward; $\\Delta$ has **low magnitude** and a **diffuse response** indicating occlusion.\n\nThis behavior is agnostic to foreground/background: background points can move under camera motion (and we detect that as a displacement). We would like to emphasize that we *do not* claim that minimal change is equivalent to being on the background. Minimal change indicates occlusion/out‑of‑view rather than background.  But our motion method works both when the motion is due to camera motion (background) or object motion (foreground): the same calculation is accurate either way. \n\n**R3 Q1(b)**: *The reviewer asks about an **empirical criterion***\n\n**R3 A1(b)**: Regarding an empirical criterion, the evaluation of Occlusion Accuracy (OA) and Occlusion F1 (OF1) already establishes this. We determine whether a point is occluded based on the magnitude of the $\\Delta$ response, and observe competitive occlusion estimation performance on TAPVid First (DAVIS: OA 80.44, OF1 68.43; Kinetics: OA 80.74, OF1 76.21; Kubric: OA 90.11, OF1 72.56).\n\n**R3 Q1(c)**: *The reviewer also asks about ablation experiments.*\n \n**R3 A1(c)**: Our ablations also show that learned counterfactual probes help: we obtain better occlusion separation than fixed patches: OA 80.34 / OF1 50.06 (learned) vs. 75.38 / 27.21 (red square) and 76.89 / 19.10 (green square) on the 512px with MM‑3/MS‑2 setting.\n\n---\n\n**R3 Q2 & Q3**: *The current layout places important figures (e.g., Figures 1 and 2) far from their detailed descriptions in the main text, which hinders readability. I recommend the authors to consider reorganizing the paper so that figures and their explanations appear together. Figure 2 does not fully reflect the described functional form in the main paper. In addition, it omits some input variables (such as, which may cause confusion about the model’s actual input-output structure.*\n\n**R3 A2 & A3**: Thank you for this suggestion. We will revise the organization of the paper and update Figure 2 according to this recommendation to help with readability.\n\n---\n\n**R3 Q4**: *Does Opt-CWM achieve true optimality in generating counterfactual probes, or are there cases where it may still be suboptimal? Is there any analysis or discussion of the theoretical limits of the proposed approach?*\n\n**R3 A4**: This is an important point. We want to emphasize that we do not make any tight mathematical claims in our work. Our method has an intuitively clear idea for how to optimize the parameters of the function that predicts image-conditioned probes, based on the downstream flow-conditioned reconstruction objective of $\\Psi^{\\texttt{flow}}$.  This is a principled surrogate for learning counterfactual probes that result in accurate motion estimates. Intuitively it makes sense why it should work, even though we don’t have a math proof.\n\nWith our joint optimization approach, probe estimation and sparse flow-conditioned next-frame reconstruction are coupled. While we do not have a formal lower bound of the coupling strength, Figure 3B directly examines the coupling, and finds that it is sufficiently strong to enable optimization of the probes.  \n\nAnd ultimately this concept pays off:  while we cannot formally demonstrate global optimality, our learned image-conditioned probes consistently outperform hand-designed alternatives (Table 3 left). \n\n--- \n\n**R3 Q5**: *Can the authors provide a detailed explanation of all symbols and notations used in the tables, to enhance readability and reproducibility?*\n\n**R3 A5**: \nWe provide information on the symbols used in our tables: \n- S, U$^\\dagger$, U: Supervised, semi-supervised, self-supervised, respectively. U$^\\dagger$ is reserved for Doduo, which uses segmentation labels and thus is not strictly self-supervised. \n- $\\textbf{bold}$, $\\underline{\\text{underline}}$: 1st and 2nd in performance, respectively. \n- shaded: Best performing supervised models are considered separately. \n- $\\ddagger$: Trained on Kubric. This is to separate GMRW results on Kubric.\n- AJ, AD, $<\\delta^x_\\textrm{avg}$, OA, OF1: TAP-Vid metrics. Please refer to Section 4 for details on each metric.\n- MM, MS: Multi-mask and multi-scale, which are inference-time enhancements. Please refer to Section 3.2 for details."}, {"review_id": "QRCz66jkhV", "reviewer": "Reviewer_TsiR", "summary": "In this paper, the authors propose an unsupervised pipeline for training a motion estimator on unlabeled video data. First, learnable probes are inserted into an RGB frame, and a masked autoencoder is employed to predict the next frame based on the unmasked region and the previous frame, enabling the extraction of probe motion. Second, a flow-conditioned next-frame predictor is established to derive a supervision signal from pixel-level loss. Specifically, this predictor generates the subsequent RGB frame conditioned on the previous frame and the estimated flow. This design facilitates gradient backpropagation to both the probe generator and the flow predictor. Evaluated on multiple video datasets, the proposed method achieves consistent and significant improvements.", "questions": "1. The next-frame predictor handles temporal dynamics in a 2-frame context; however, there are high-frame-rate and low-frame-rate videos on the internet, and the motion across adjacent frames may have a lot of differences. Can the results of the proposed method be affected by this variance?\n2. The training pipeline fully relies on the next-frame prediction world model to generate the warped position of the probes in the masked region of the second frame. What if the unmasked region conflicts with the pasted probes? Does the mask strategy take into account this situation?", "limitations": "The training process utilizes a pre-trained world model to generate new positions of the inserted probs and train a flow model based on the updated positions of the probs. This includes two main limitations:\n- Firstly, the quality of the generated next frame depends on the pre-trained world model. Once its capacity can not match the scale of the training data, the flow model performance may be hindered by the sub-optimal synthesized frames.\n- Secondly, to calculate the position of the probs in the next frame, the authors apply a heuristic method that calculates the difference image and finds its peak. This means the inserted probs should be sparse and may be affected by occlusion of different probs; thus, the number of reference points can not be scaled up in an example. Although the proposed training pipeline neglects the number of available probs, it may still face the challenge of training efficiency.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n- The proposed Opt-CWM methods yield significant improvement over previous methods and even beat supervised methods across three different video datasets, which reveals their effectiveness and advancedness.\n- The distilled SEA-RAFT model from Opt-CWM using unsupervised data shows competitive results against the supervised learned SEA-RAFT variant, which shows a promising ability to learn robust motion representation from internet-scale videos.\n- All proposed modules (probe generator, flow function, flow-based next frame predictor) in the paper are trained unsupervised, without separate, direct supervision, yet they mutually reinforce each other and collectively enable robust motion extraction.\n\n**Weakness:**\n- Learning motion clues from unlabed video is similar to the previous idea [1], I recommend that the authors cite and discuss this literature in the related work section.\n- The proposed parameterized counterfactual probe generator is trained in an unsupervised manner without direct supervision, which means the generalization ability of the probe generator is still restricted by the heuristic used to generate the training data. Since the probe generation process is similar to the image editing task, the authors can try to use off-the-shelf image editing methods like [2] to generate probes.\n\n[1]. Masked Motion Encoding for Self-Supervised Video Representation Learning.\n[2]. LEDITS++: Limitless Image Editing using Text-to-Image Models.", "was_revised": true, "final_justification": "According to the summarized pros and cons above, and given that all my concerns about the proposed method are addressed, I recommend an acceptance decision.", "rebuttal": "**R1 Q1**: *Learning motion clues from unlabed video is similar to the previous idea [1], I recommend that the authors cite and discuss this literature in the related work section.*\n\n**R1 A1**:  Thank you for the suggestion. We will include a discussion of [1] in the related work section. That work focuses on learning video representations for video action recognition by reconstructing motion trajectories and object shapes from sparse video patches, using ground-truth motion trajectories extracted via traditional optical flow methods.\n\nBy contrast, our work serves a different purpose: we are not aiming to learn a video representation for video action recognition. Instead, we propose a method to extract optical flow signals from a world model that has already been trained to predict future RGB frames. Our approach is self-supervised in the sense that it does not require external flow annotations or supervision. In fact, the flows extracted via our method could be used as training data for models like [1], providing a viable alternative to using other available off-the-shelf flow algorithms.\n\n\n---\n\n**R1 Q2**: *The proposed parameterized counterfactual probe generator is trained in an unsupervised manner without direct supervision, which means the generalization ability of the probe generator is still restricted by the heuristic used to generate the training data. Since the probe generation process is similar to the image editing task, the authors can try to use off-the-shelf image editing methods like [2] to generate probes.*\n\n**R1 A2**: Thank you for the suggestion regarding the use of off-the-shelf image editing methods such as [2]. However, we would like to clarify that our probe generation process is fundamentally different from image editing, particularly from text-driven editing pipelines like [2]. Our method does not aim to generate semantically meaningful image edits. Instead, we apply a small, image-conditioned, localized perturbation (a learnable patch) to the input frame and analyze how this perturbation propagates across time through the model's predicted next frame. This allows us to estimate the local optical flow.\n\n---\n\n**R1 Q3**: *Firstly, the quality of the generated next frame depends on the pre-trained world model. Once its capacity can not match the scale of the training data, the flow model performance may be hindered by the sub-optimal synthesized frames.*\n\n**R1 A3**: Thank you for your comment. We agree that the quality of the generated next frame depends on the capacity of the underlying world model. As noted in prior work on CWM [2, 23] as well as results in our paper, increasing model capacity consistently improves performance on flow estimation. While it is possible that such gains may eventually saturate, current results suggest we have not yet reached that point.\n\nWe therefore believe it is premature to conclude that motion estimation will be fundamentally limited by the fact that the model's performance may saturate at larger scales. Rather, we view this as an open question for future work. For now, our approach is justified by the empirical improvements observed at the scales at which the current CWM model is trained.\n\n---\n\n**R1 Q4**: *Secondly, to calculate the position of the probs in the next frame, the authors apply a heuristic method that calculates the difference image and finds its peak. This means the inserted probs should be sparse and may be affected by occlusion of different probs; thus, the number of reference points can not be scaled up in an example. Although the proposed training pipeline neglects the number of available probs, it may still face the challenge of training efficiency.*\n\n**R1 A4**: We agree with the reviewer that inserting multiple probes into the same image can lead to interference effects such as occlusion and overlapping influence regions, which makes simultaneous probing challenging. However, our current implementation addresses this by applying a single perturbation per image and executing all probes in a batched and parallelized manner. This allows for efficient extraction of flow signals without compromising the integrity of individual probes.\n\nFurthermore, once we extract sparse flow supervision from the model using these probes, we can train a separate supervised optical flow estimator (e.g., SEA-RAFT) on this data. As shown in our paper (Table 2, right), these distilled models are not only significantly faster at inference time, but also achieve good performance, demonstrating that our approach is practical despite the sparsity of individual probe. We summarize these results below, and present an additional experiment where we distill Opt-CWM into the newer DPFlow [A] model, which achieves better performance.  \n\n| Model | AJ | AD | Pct | OA | OF1 |\n|  :--- | :---: | :---: | :---: | :---: | :---: |\n| RAFT | 41.77 | 25.33 | 54.37 | 66.40 | 56.12 |\n| SEA-RAFT | 43.41 | 20.18 | 58.69 | 66.34 | 56.23 |\n| SMURF | 30.64 | 27.28 | 44.18 | 59.15 | 46.91|\n| Opt-CWM 175M | 47.53 | 8.73 | 64.83 | 80.87 | 60.74 |\n| SEA-RAFT (distilled) | 44.05 | 17.49 | 57.93 | 69.75 | 60.72 |\n| DPFlow (distilled) | 45.55 | 15.68 | 59.36 | 69.27 | 58.41 |\n\n[A] DPFlow: Adaptive Optical Flow Estimation with a Dual-Pyramid Framework - Henrique Morimitsu, Xiaobin Zhu, Roberto M. Cesar Jr., Xiangyang Ji, Xu-Cheng Yin"}]

Author rebuttal:

Meta-review: This paper considers the task of extracting optical flow and occlusion from large pretrained video models using counterfactual probes: input-conditioned perturbations to the appearance of image regions. On the whole, the reviewers consider this a technically sound and well-motivated approach that elicits significant improvements in practice. The identified strengths include the design choices, such as the idea of using a probe generator that takes into account local and global appearance context so that the probes are more in-distribution than a hand-coded probe, the principled self-supervised learning objective, and the use of flow prediction as a bottleneck, as well as the performance that is competitive with supervised approaches. The identified weaknesses included concerns about evaluation settings (satisfactorily clarified in the author response), model capacity (likewise, via a distillation approach), the optimality claim (clarified to be meant in a loose, rather than mathematical sense), and presentation/readability/clarity.

The initial ratings skewed positive with some deviation between reviewers. Upon reading all reviews, author responses, and engaging in discussion with the authors, the reviewers formed a consensus to accept the paper. The reviewers concur that the authors addressed their primary concerns, making key clarifications. The AC sees no reason to override the consensus of the reviewers and is satisfied by the technical merit of the paper and its potential to make an impact on the field.

In addition to the revision items raised by the authors in their final response, I would request that they revise their claim that "the key problem we solve is optimal counterfactual probe generation". I concur with reviewer 5xm4 that a reasonable reader would expect this to be a mathematical claim of (global) optimality, which the authors clarify is not the case. Alternatives such as "high-quality" or similar would be more appropriate.
Decision: Accept (spotlight)