Title: DisMo: Disentangled Motion Representations for Open-World Motion Transfer
Venue: NeurIPS 2025
Accepted: yes
Keywords: Representation Learning

Motion Analysis

Video Generation

Self-Supervised Learning
Abstract: Recent advances in text-to-video (T2V) and image-to-video (I2V) models, have enabled the creation of visually compelling and dynamic videos from simple textual descriptions or initial frames. However, these models often fail to provide an explicit representation of motion separate from content, limiting their applicability for content creators. To address this gap, we propose DisMo, a novel paradigm for learning abstract motion representations directly from raw video data via an image-space reconstruction objective. Our representation is generic and independent of static information such as appearance, object identity, or pose. This enables open-world motion transfer, allowing motion to be transferred across semantically unrelated entities without requiring object correspondences, even between vastly different categories. Unlike prior methods, which trade off motion fidelity and prompt adherence, are overfitting to source structure or drifting from the described action, our approach disentangles motion semantics from appearance, enabling accurate transfer and faithful conditioning. Furthermore, our motion representation can be combined with any existing video generator via lightweight adapters, allowing us to effortlessly benefit from future advancements in video models. We demonstrate the effectiveness of our method through a diverse set of motion transfer tasks. Finally, we show that the learned representations are well-suited for downstream motion understanding tasks, consistently outperforming state-of-the-art video representation models such as V-JEPA in zero-shot action classification on benchmarks including Something-Something v2 and Jester. Project page: https://compvis.github.io/DisMo


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces DisMo, a novel approach for learning abstract motion representations directly from raw video data using an image-space reconstruction objective. Dismo disentangles motion from static content such as appearance, object identity, or pose. This allows motion to be transferred across semantically unrelated entities, even between vastly different categories, without requiring object correspondences. Dismo achieves state-of-the-art performance on several downstream tasks like like open-world motion transfer, zero-shot action classification.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. Dismo learns motion information directly from raw video data using only a standard image reconstruction loss by a  flow-matching-based reconstruction objective, which is interesting and concise.\n2. Experiments on motion transfer tasks and action classification tasks show the effectiveness of the motion embeddings.\n3. The motion representation can be combined with any existing video generator via lightweight adapters, allowing the approach to benefit from future advancements in video generate models.\n\n**Weaknesses:**\n\n1. Scaling. Will the model's performance be further enhanced if a more powerful motion extractor and frame generator are employed? Additionally, how will the training cost be affected?\n2. Is the experimental comparison in Table 1 fair? Since Dismo is trained using a pretrained diffusion transformer along with a light adapter, does this introduce any unfairness when compared to baselines such as VMC and DMT?\n3. In the action classification task, datasets like K600, THUMOS and AVA are more commonly used, their results should also be evaluated.\n\nQUESTIONS:\nPlease see the weaknesses. The method proposed in the paper is interesting and promising. My primary concerns focus on its generalizability and scaling capability. I will take into consideration the opinions of other reviewers when making my final decision.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed DisMo that learns abstract motion representations directly from video data via an image space reconstruction objective. As their model is independent of different static information, including appearance, pose, or object identity, it has the capability for open-world motion transfer that allows motion to be transferred across semantically unrelated entities without requiring object correspondences. Unlike prior methods that show a trade-off between motion fidelity and adherence to textual prompts, their approach disentangles the motion semantics from instance-specific appearance features. They demonstrated the effectiveness on diverse motion transfer tasks, outperforming existing state-of-the-art video representation models.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nS1. The idea of representation learning solely based on an image-space reconstruction objective is interesting and unique in terms of leading to an abstract motion representation space.\n\nS2. The authors performed a diverse set of experiments focusing on cross-category and cross-viewpoint settings. The downstream classification task also shows the efficiency of the learned motion embeddings.\n\nS3. One of the primary strengths of this paper is that the core idea of learning motion representations that are invariant to static content is well described and motivated.\n\nS4. The experiments are comprehensive with clear implementation details, ablation studies, and latent space analysis.\n\nS5. The paper is well-written, easy to read and follow. \n\n**Weaknesses**\n\nW1. I found that the current methodological development lacks detail about how the \"invariance\" is actually enforced or how it is being maintained with respect to the architectures that ensure this property.\n\nW2. While the authors show appearance disentanglement results in Tab. 2, they did not provide clear metrics to measure motion invariance specifically or whether motion patterns are preserved or not (or how much). Also, experiments for achieving invariance would be further strengthened by performing experiments in different scenarios such as with different lighting, backgrounds, and camera angles. Overall, I found that the current disentanglement evaluation is limited and does not directly measure motion invariance.\n\nW3. Computational costs and statistical significance tests are missing. Compared to the alternatives, what's the efficiency and complexity in terms of time and memory?\n\nI would strongly suggest the authors to justify their standings on the raised weaknesses and concerns in the rebuttal period.\n\nQUESTIONS:\nPlease see the \"Strengths and Weaknesses\" section.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper proposes DisMo, a method for learning abstract, disentangled motion representations from raw videos via image-space reconstruction. The goal is to enable open-world motion transfer without relying on object correspondences. DisMo can be plugged into existing video generators and shows strong results in fidelity, prompt alignment, and generalization.

STRENGTHS AND WEAKNESSES:
- The writing in this paper is clear and easy to follow.
- I appreciate the method’s flexibility and modular design — it is compatible with various pre-trained video generators. The ability to perform zero-shot transfer is particularly compelling.
- It’s also encouraging to see strong performance on both standard and out-of-distribution benchmarks.

Limitations:
- The idea is not very new to me. Abstraction and disentanglement exist in robotics and embodied AI; the contribution feels incremental.
- Evaluation scope limited: Would like to see more extreme or semantically challenging motion transfers to test generality.
- The method adds non-trivial motion modeling layers, but lacks analysis of training/inference cost. How is the stability of the method?

QUESTIONS:
Can the authors clarify the novelty of their approach, especially in relation to existing work in robotics and motion generation?

How do the authors address the visible artifacts in the generated videos? Is there any planned analysis or improvements for these issues?

I am not fully in this field, I would like to check other reviews opinion on this submission.

LIMITATIONS:
/
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents DisMo, a novel framework for learning abstract motion representations disentangled from appearance, pose, and category information. Unlike prior approaches that encode motion implicitly or rely on structure-specific representations, DisMo learns motion purely through an image reconstruction loss from augmented video clips using a flow-matching-based training objective. The resulting motion embeddings are invariant to static content and can be used to condition any video generator via lightweight adapters (e.g., LoRAdapter in LTX Video) for open-world motion transfer. DisMo is evaluated extensively across three main tasks, including Motion Transfer, Appearance Disentanglement and Zero-shot Action Classification.\n\nSTRENGTHS AND WEAKNESSES:\n## **Strengths**\n\n- **Clear and Novel Formulation**: DisMo introduces a simple and effective method for learning motion representations by leveraging standard image-space reconstruction losses, augmented with an information bottleneck and strong data augmentations to force motion extraction.\n- **Disentanglement and Generalization**: The model robustly disentangles motion from identity and visual content, enabling effective **cross-category** and **viewpoint-invariant** transfer. This is validated both qualitatively and quantitatively.\n- **Integration with Video Generators**: The framework is modular and scalable. The learned motion embeddings can be seamlessly integrated into pretrained video generation models (e.g., LTX Video) using lightweight LoRA adapters, eliminating the need to retrain the entire generative backbone.\n- **Strong Experimental Results**: DisMo outperforms state-of-the-art baselines in both generative and discriminative tasks—achieving best scores in motion fidelity and text adherence, while also outperforming strong backbones like V-JEPA on zero-shot action classification with only 17M training samples.\n- **Various Qualitative Analysis**: The paper provides a wide range of visual comparisons, illustrating robustness across scenarios such as camera motion transfer, same- and cross-category transfer, and structure-independent motion abstraction.\n\n## **Weaknesses**\n\n- **No Evaluation on High-Fidelity Generation Tasks Beyond LTX**: While motion quality is demonstrated using LTX Video, there is limited exploration of how DisMo integrates with other high-fidelity video generation models (e.g., Wan2.1, HunyuanVideo). Without such experiments, it is difficult to assess the general applicability of the proposed motion representations across different generation frameworks.\n- **Limited Analysis of Flow-Matching Design Choice**: The paper adopts flow matching as the core training objective for motion representation learning, but does not provide comparisons or ablations against alternatives like diffusion-based or contrastive objectives. It remains unclear whether the reported performance gains are due to the task formulation or specific to the flow-matching mechanism.\n- **Missing Comparisons with Other Motion Extractors**: Although DisMo’s own extractor is proposed as a key contribution, the paper does not compare its performance with existing video representation models (e.g., V-JEPA, VideoMAEv2) when used within the same motion transfer pipeline. This would clarify the advantage of the proposed extractor over well-established baselines.\n- **Absence of Human Study**: The evaluation relies entirely on quantitative metrics (e.g., CLIP-based similarity) and visual examples. However, given the inherently perceptual nature of motion realism and alignment, a user study would provide valuable insight into subjective quality and preference.\n- **No Experiments on Composite Motion Scenarios**: The paper does not explore how DisMo handles complex scenarios involving multiple motion types simultaneously—such as combining camera motion with object or human motion. This limits the understanding of its effectiveness in realistic, unconstrained settings.\n- **Lack of Analysis on the Role of Augmentations**: The paper claims that strong augmentations during training help enforce disentanglement by forcing the extractor to focus on temporal dynamics rather than appearance. However, there is no dedicated analysis or qualitative example demonstrating how this affects motion transfer performance. A targeted study could help isolate the impact of augmentations on representation quality and transfer fidelity.\n- **Ambiguous Reference in Line 252**: The sentence “Qualitative examples in Section 4.1 further illustrate this” refers to visual results, but it is unclear whether this points to Figure 4.\n\nQUESTIONS:\n- Have you tested DisMo with other high-quality video generators (e.g., Wan2.1, HunyuanVideo)? If not, do you anticipate any compatibility or performance limitations when applying your motion embeddings to those models?\n- Why did you choose flow matching over other generative training paradigms like diffusion, contrastive learning, or masked prediction?\n- How does the performance of your motion extractor compare to representations learned by models such as V-JEPA or VideoMAEv2, when used in the same transfer setup?\n- What specific impact do the augmentations during training have on the quality of motion transfer results?\n- Given the perceptual nature of motion transfer quality, have you considered running a user study or human evaluation to complement the CLIP-based metrics?\n- Can DisMo handle composite motion scenarios involving both object motion and camera motion simultaneously?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}