Title: Self-Supervised Learning of Motion Concepts by Optimizing Counterfactuals
Venue: NeurIPS 2025
Accepted: yes
Keywords: self-supervised learning

motion understanding

visual tracking

video foundation modeling
Abstract: Estimating motion primitives from video (e.g., optical flow and occlusion) is a critically important computer vision problem with many downstream applications, including controllable video generation and robotics. Current solutions are primarily supervised on synthetic data or require tuning of situation-specific heuristics, which inherently limits these models' capabilities in real-world contexts. A natural solution to transcend these limitations would be to deploy large-scale, self-supervised video models, which can be trained scalably on unrestricted real-world video datasets. However, despite recent progress, motion-primitive extraction from large pretrained video models remains relatively underexplored. In this work, we describe Opt-CWM, a self-supervised flow and occlusion estimation technique from a pretrained video prediction model. Opt-CWM uses ``counterfactual probes'' to extract motion information from a base video model in a zero-shot fashion. The key problem we solve is optimizing the quality of these probes, using a combination of an efficient parameterization of the space counterfactual probes, together with a novel generic sparse-prediction principle for learning the probe-generation parameters in a self-supervised fashion. Opt-CWM achieves state-of-the-art performance for motion estimation on real-world videos while requiring no labeled data.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose an unsupervised pipeline for training a motion estimator on unlabeled video data. First, learnable probes are inserted into an RGB frame, and a masked autoencoder is employed to predict the next frame based on the unmasked region and the previous frame, enabling the extraction of probe motion. Second, a flow-conditioned next-frame predictor is established to derive a supervision signal from pixel-level loss. Specifically, this predictor generates the subsequent RGB frame conditioned on the previous frame and the estimated flow. This design facilitates gradient backpropagation to both the probe generator and the flow predictor. Evaluated on multiple video datasets, the proposed method achieves consistent and significant improvements.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- The proposed Opt-CWM methods yield significant improvement over previous methods and even beat supervised methods across three different video datasets, which reveals their effectiveness and advancedness.\n- The distilled SEA-RAFT model from Opt-CWM using unsupervised data shows competitive results against the supervised learned SEA-RAFT variant, which shows a promising ability to learn robust motion representation from internet-scale videos.\n- All proposed modules (probe generator, flow function, flow-based next frame predictor) in the paper are trained unsupervised, without separate, direct supervision, yet they mutually reinforce each other and collectively enable robust motion extraction.\n\n**Weakness:**\n- Learning motion clues from unlabed video is similar to the previous idea [1], I recommend that the authors cite and discuss this literature in the related work section.\n- The proposed parameterized counterfactual probe generator is trained in an unsupervised manner without direct supervision, which means the generalization ability of the probe generator is still restricted by the heuristic used to generate the training data. Since the probe generation process is similar to the image editing task, the authors can try to use off-the-shelf image editing methods like [2] to generate probes.\n\n[1]. Masked Motion Encoding for Self-Supervised Video Representation Learning.\n[2]. LEDITS++: Limitless Image Editing using Text-to-Image Models.\n\nQUESTIONS:\n1. The next-frame predictor handles temporal dynamics in a 2-frame context; however, there are high-frame-rate and low-frame-rate videos on the internet, and the motion across adjacent frames may have a lot of differences. Can the results of the proposed method be affected by this variance?\n2. The training pipeline fully relies on the next-frame prediction world model to generate the warped position of the probes in the masked region of the second frame. What if the unmasked region conflicts with the pasted probes? Does the mask strategy take into account this situation?\n\nLIMITATIONS:\nThe training process utilizes a pre-trained world model to generate new positions of the inserted probs and train a flow model based on the updated positions of the probs. This includes two main limitations:\n- Firstly, the quality of the generated next frame depends on the pre-trained world model. Once its capacity can not match the scale of the training data, the flow model performance may be hindered by the sub-optimal synthesized frames.\n- Secondly, to calculate the position of the probs in the next frame, the authors apply a heuristic method that calculates the difference image and finds its peak. This means the inserted probs should be sparse and may be affected by occlusion of different probs; thus, the number of reference points can not be scaled up in an example. Although the proposed training pipeline neglects the number of available probs, it may still face the challenge of training efficiency.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Opt-CWM, a novel self-supervised approach for motion concept learning from videos. By leveraging large-scale pretrained video prediction models, the authors introduce optimized, learnable counterfactual probes that extract high-quality optical flow and occlusion information from real-world videos without any labeled data. The core innovations are: (1) parameterizing the counterfactual probe generator with a neural network, allowing probes to adapt to image content and context, and (2) a self-supervised learning objective that trains the probe generator via a sparse prediction information bottleneck, eliminating the need for heuristic losses or supervision.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: This paper introduces a novel approach to self-supervised motion and occlusion estimation by leveraging optimized counterfactual probes, which are parameterized by a neural network and learned entirely without supervision. Unlike traditional CWM that relies on fixed, hand-crafted perturbations, Opt-CWM framework utilizes learnable, adaptive perturbations that are dynamically generated based on image content and context. The method achieves state-of-the-art empirical performance on several challenging benchmarks, consistently outperforming both established supervised methods and several self-supervised baselines.\n\nWeaknesses\n1. The paper lacks a clear and principled explanation for how the model determines whether an added perturbation will be meaningfully carried forward with the moving object in the model’s prediction.\n2. The manuscript suffers from poor layout and figure placement. Important figures (e.g., Figure 1 and Figure 2) are physically distant from their detailed explanations, making it difficult for readers to follow the narrative and connect visuals with the text.\n3. While the paper points out that previous CWM methods are suboptimal due to their fixed, hand-crafted perturbations, it does not provide sufficient discussion or evidence on whether Opt-CWM achieves optimality or merely improves upon prior suboptimal solutions.\n4. The paper does not provide an analysis of the computational complexity for either training or inference.\n5. Symbols and formatting (e.g., underlines, arrows) in the result tables are not explained, which could cause confusion for readers unfamiliar with these conventions.\n\nQUESTIONS:\n1. Could the authors provide more detailed analysis or ablation experiments showing how the model distinguishes between perturbations that correspond to moving/visible objects versus those that are static or occluded? Is there a theoretical or empirical criterion to ensure that significant model output changes are reliably associated with true object motion, and minimal changes indicate occlusion or background?\n2. The current layout places important figures (e.g., Figures 1 and 2) far from their detailed descriptions in the main text, which hinders readability. I recommend the authors to consider reorganizing the paper so that figures and their explanations appear together. \n3. Figure 2 does not fully reflect the described functional form in the main paper. In addition, it omits some input variables (such as $I_2$, which may cause confusion about the model’s actual input-output structure.\n4. Does Opt-CWM achieve true optimality in generating counterfactual probes, or are there cases where it may still be suboptimal? Is there any analysis or discussion of the theoretical limits of the proposed approach?\n5. What is the computational complexity of Opt-CWM compared to existing baselines? Is the method practical for large-scale or real-time applications?\n6. Can the authors provide a detailed explanation of all symbols and notations used in the tables, to enhance readability and reproducibility?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a new self-supervised learning based method, Opt-CWM, for learning motion concepts. It solves the limitations of an existing self-supervised learning based method method, CWM, in this field with two innovations. First, it propose to parameterize a counterfactual probe policy generator with a learnable neural network that can predict situation-specific probes that take into account the appearance context (both local and global) around target points to be tracked, and thus can be less out-of-distribution than hand-coded probes. Second,  it proposes to learn the probe generator in a principled fashion without relying on any supervision from labeled data or heuristics. Experiments shows the proposed method significantly improves performance of multiple motion concept prediction tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The proposed method is technically sound. \n- Experiments show significant improvement over existing methods. \n- The proposed approach is well motivated. \n\nWeakness\n- The proposed method uses two large base predictors, one with 175M and the other 1B learnable parameters, which makes the comparison not very fair over the existing methods. \n- Due to the introduce of the large models, the inference time of the proposed method should be much higher than the existing methods. It would be great to see the comparison on this axis. \n-  To address the limitation on high inference time, it is stated that \"We demonstrate a pathway to resolving this by distilling the trained Opt-CWM into small and efficient architectures\". However, I do not see such results.\n- What is the impact of the scale of the base predictors? Currently two models of 175M and the other 1B are employed. How about using smaller or even larger models? It would be great the see the trend of the performance with respect to the model size. \n- Many sentences are too long and complex, making the paper hard to read and follow.\n\nQUESTIONS:\nSee the weakness above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work explores a new idea for how to extract motion-primitives from large pretrained video models, specifically focusing on optical-flow and occlusion. The authors develop a method Opt-CWM which estimates optical flow and occlusion from pretrined video prediction models by leveraging counterfactual probes, which are input-conditioned perturbations to the appearance or position of visible patches. The authors develop a method for generating these probes in an optimal manner and show that the probe-generation parameters can be learned with a self-supervised method. Given, this Opt-CWM is shown to be SoTA in motion estimation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper is well-written and clear on the contributions.\n2. The method for learning optimiized counterfactuals via using the flow prediction as a bottleneck is interesting and novel to my knowledge and approaches this problem in a different way than many existing methods.\n3. The ablations are quite ocmprehensive and appreciated. \n\nWeaknesses\n1. The authors state that most recent works in point-tracking are semi-supervised (like BootsTAP, TAPNext etc.) and thus are out of scope for comparison. While I appreciate this, I feel it would be disingenuous not include some of these results in the paper for comparison to show where the SoTA is for these tasks. Methods like BootsTAP or TAPNext are extremely scalable because they use pseudo-labeling. Therefore, one might ask if the goal is to solve point-tracking through occlusions etc. generally then what is the inherent benefit of using something self-supervised if the performance is still drastically worse than those methods (see numbers in TAPNext paper Table 1 for example)? The main driver of self-supervised learning to me is it's generality and scalability to places where finding supervision is hard or not clean, but if semi-supervised learning can bridge this gap effectively should these tasks (e.g. TAP-Vid) be the main demonstration of the benefits of this approach?\n2. The authors miss some key citations of work that needs to be compared to. For example DINO-Tracker from Tumanyan, Singer et al (ECCV 2024) is also self-supervised and seems to present quantitative results that are far better than what is reported in this paper. Can the authors include this comparison or comment on why it is not necessary? \n3. There are some discrepancies in the numbers reported in this work vs. the literature. For example for DAVIS AJ and delta_avg, the authors report GMRW as 36.47 and 54.59. However in the GMRW paper they report: 41.8 and 60.9 . These kinds of discrepancies should be clarified in the work otherwise comparison across works becomes quite tricky. Is there something about the evaluation methodology used that differs from prior work for example?\n\nQUESTIONS:\nSee above weaknesses if these are addressed I'm willing to raise my score.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-07-23"}