DisMo: Disentangled Motion Representations for Open-World Motion Transfer
Authors: Thomas Ressler-Antal
Frank Fundel
Malek Ben Alaya
Stefan Andreas Baumann
Felix Krause
Ming Gui
Björn Ommer
Venue: NeurIPS 2025 spotlight ()
Keywords: Representation Learning
Motion Analysis
Video Generation
Self-Supervised Learning
Primary area: deep_learning

Abstract: Recent advances in text-to-video (T2V) and image-to-video (I2V) models, have enabled the creation of visually compelling and dynamic videos from simple textual descriptions or initial frames. However, these models often fail to provide an explicit representation of motion separate from content, limiting their applicability for content creators. To address this gap, we propose DisMo, a novel paradigm for learning abstract motion representations directly from raw video data via an image-space reconstruction objective. Our representation is generic and independent of static information such as appearance, object identity, or pose. This enables open-world motion transfer, allowing motion to be transferred across semantically unrelated entities without requiring object correspondences, even between vastly different categories. Unlike prior methods, which trade off motion fidelity and prompt adherence, are overfitting to source structure or drifting from the described action, our approach disentangles motion semantics from appearance, enabling accurate transfer and faithful conditioning. Furthermore, our motion representation can be combined with any existing video generator via lightweight adapters, allowing us to effortlessly benefit from future advancements in video models. We demonstrate the effectiveness of our method through a diverse set of motion transfer tasks. Finally, we show that the learned representations are well-suited for downstream motion understanding tasks, consistently outperforming state-of-the-art video representation models such as V-JEPA in zero-shot action classification on benchmarks including Something-Something v2 and Jester. Project page: https://compvis.github.io/DisMo

DisMo: Disentangled Motion Representations for Open- Motion Transfer
Thomas Ressler-Antal
Frank Fundel" Malek Ben Alaya
Stefan Andreas Baumann
Felix Krause
Ming Gui Björn Ommer
CompVis @ LMU Munich, Munich Center for Machine Learning (MCML)
"A "akknng yellow monster"
Figure 1: Motion transfer examples enabled by our abstract motion representations. We extract abstract motion representations from driving videos and transfer them onto new content, represented either by source images (Left) and (Middle), or text prompts (Right).
Abstract
Recent advances in text-to-video (T2V) and image-to-vided (I2V) models, have enabled the creation of visually compelling and dynamic videos from simple tex- tual descriptions or initial frames. However, these models often fail provide explicit representation of motion separate from content, limiting their appli- cability for content creators. address this gap, we propose DisMo, a novel paradigm for learning abstract motion representations directly from raw video data via an image-space reconstruction objective. Our representation is generic and independent of static information such as appearance, object identity, pose. This enables open-world motion transfer, allowing motion to be transferred across semantically unrelated entities without requiring object correspondences, between vastly different categories. Unlike prior methods, which trade off mo- tion fidelity and prompt adherence, are overfitting source structure drifting from the described action, our approach disentangles motion semantics from ap- pearance, enabling accurate transfer and faithful conditioning. Furthermore, our motion representation can be combined with any existing video generator via lightweight adapters, allowing us effortlessly benefit from future advancements in video models. We demonstrate the effectiveness of our method through diverse set of motion transfer tasks. Finally, we show that the learned representations are well-suited for downstream motion understanding tasks, consistently outper- forming state-of-the. video representation models such as V-JEPA zero-shot action classification on benchmarks including Project page: attps://compviee titthb.b.ooii...
v2 and Jester.
Equal contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
A striking feature of human visual intelligence is innate ability to understand motion and to reason about it independently from static attributes such as appearance, position, and viewpoint, generalizing seamlessly across different instances within object category and even beyond to entirely different objects and scenarios [Johansson] 1973]. Already at young age, we are able to generalize motion despite stark deviations in environment and content different scene allowing already little children to imagine the plausible movement of fictitious characters. This capacity highlights a fundamental cognitive ability represent and reason about motion as abstract, invariant phenomenon.
Recent breakthroughs in generative video synthesis have enabled models to produce visually com- pelling and dynamically realistic videos from minimal inputs such textual descriptions (text-to- video, T2V) or single initial frames (image-to-vides) I2V) [Blattmann et al. 2023 Brooks et al.) 2024 HaCohen et al. 2024 Kong al., 2024 Veo-Team et al.) 2024). Despite the remarkable ren- dering quality achieved by these models, they lack explicit representations of underlying kinematics, inherently limiting their ability to precisely control or manipulate the motion of generated scenes.
Existing T2V and 12V approaches implicitly represent motion by modeling videos as sequences of static frames, which aligns poorly with human perception that inherently treats motion as distinct and explicitly perceived entity rather than merely a tomography across time (Kandel et al.) 1991]. Furthermore, precise textual descriptions of complex or abstract motion remain challenging, underscoring the need for novel representation approaches that go beyond verbal or purely frame- based encodings.
Explicit methods for representing dynamic scene content have traditionally relied on dense, pixel- level motion descriptors such as optical flow [Zhao et al., 2024 Chefer et al. 2025 Liang et al.) 2024). While optical flow effectively captures detailed per-pixel changes between frames, remains strongly coupled to specific instance structures, locations, and viewpoints, thus lacking the invariance needed for robust motion generalization and transfer across diverse scenarios. Similarly, sparse representations such as trackers that follow key points through time are somewhat less dependent on appearance but still fundamentally require structural consistency across frames [Karaev et al., 2024)). Consequently, neither dense nor sparse methods readily support motion transfer between significantly different instances, object categories, and viewpoints.
Inspired by recent successes in text-to-image (T2I) synthesis. where explicit structural control mechanisms have greatly enhanced user-driven generation (Rombach et al.) al.] [222]] Zhang et al.) 2202a Stracke et al. [224]] we argue that generative video models similarly require novel mechanisms for explicitly representing and controlling motion separate of appearance and spatial configurations. Such explicit and abstract representations would significantly enhance user control over dynamically
generated content.
In this work, we introduce method for learning abstract motion representations that are invariant to static appearance and structure-basd information, directly from raw video data using only a standard image reconstruction loss. Remarkably, our representation emerges entirely through end-to-end training regime without the need for specialized regularization, contrastive losses, or complex training setups. We leverage simple yet effective flow-matchin matching-based based reconstruction objective, commonly used in visual generation methods, to obtain a representation that is largely invariant to factors typically encoded in the reconstruction loss, such as position, identity, category, and pose. The result a motion representation robust enough to perform challenging and creative open-world motion transfers, enabling scenarios such as transferring the movement of a human to that of an ape, an imaginary cartoon character, or even completely unconventional transformations. This capability is achieved through information bottleneck further by strong augmentations that makes it impossible to reconstruct the target using visual and appearance- information alone, forcing the model to analyze changes between frames and encode them into an abstract embedding.
We evaluate these embeddings via zero-shot probing across multiple action classification datasets and ensure their representative performance. Additionally, we design these motion representations to be invariant to the choice of renderer, allowing seamless integration with state-of-the-art video diffusion models through lightweight adapters [Stracke et al.) 2024), which reduces compute as expensive video pipelines do not need to be retrained. This orthogonal approach transforms existing generative video models from merely powerful renderers into highly controllable generative systems.

---

Our main contributions are as follows:
We construct a powerful representation learning paradigm powered only by an image-space reconstruction objective and show how this leads to abstract motion representation space that is trained for invariance to static content information such as appearance or structure.
We leverage this invariance and condition a pre-trained generative video model on our learned motion representations, which, despite training in a matched setting capable of zero-shot motion transfer.
We show how this setup achieves state-of-the-an performance open-world motion transfer with high degree of transferability, especially cross-category and cross-viewpoint settings.
We show how our motion embeddings can be utilized downstream action classification settings.
2
Related Work
Generative image and video models have experienced rapid advances in recent years. largely driven by diffusion-basse training objectives [Ho et al. 2020 Song et al.) [020 Lipman al. 2022 Originally developed for the image domain, these powerful frameworks have been extended to the video domain, demonstrating impressive generative capabilities and realism. [Brooks al. (0224 demonstrated that large-scale video synthesis enables the realistic generation of complex
Video Synthesis
spatiotemporal dynamics from simple text and image prompts. Subsequently, several state-of-the-an
text-to-video (T2V) and image-to-vided (12V) diffusion models have further significantly advanced video synthesis capabilities Blattmann et al. [023 Yang et al. 2024 HaCohen et al.) 2024 Kong
al. 2024 Veo-Team al. 2024] Polyak et al. 2025] and made such models widely accessible.
Despite their success, current T2V/IVV methods primarily focus on synthesizing visually appealing videos from textual and image prompts but do not explicitly model the underlying motion separately from appearance. As result, they inherently lack the capacity for precise or intuitive control over the generated motion, limiting their applicability in scenarios requiring detailed, user-specified dynamics.
Motion Control Video Generation Enabling detailed motion control in generative video models remains challenging and active research area. Recent methods typically achieve motion control
by providing explicit low-level spatial or temporal constraints, such as bounding boxes (Jain et al.
2024
Wang al. 2024a Wu et al. [2055] trajectories [Wang et al. 2024b Wu et al. [024]] Yin et al.
2023
or structural maps like optical flow and depth sequences Zhang et al.) 2023b] Esser et al.
2023
Although effective within certain scenarios, these conditioning signals inherently tie motion
control to precise spatial locations and/or structures in the source video, limiting their invariance and transferability. Consequently, these approaches struggle with scenarios involving substantial variations in viewpoint, appearance, or semantic category.
Parametric approaches offer another perspective by learning comain-specifie motion representations, frequently used in specialized areas such as facial reenactment and talking -ead generation (Siarohin
et al. 2019 2021 Wang et al. 2021 1 Zhao and Zhang 2022, Wang et al. 2022 Hong et al. 2022
Yang et al.] 2022 Gao al. 2023]. These methods typically rely on pre-detined or learned keypoint- based wwulegooy-speciiee representations (either 2D or 3D) to encode motion explicitly. For instance, First-Order Motion Models (FOMM) SSirohhi et al. 2019), Motion Representations for Articulated Animation (MRAA) (Siarohin et al. 2021]. and Latent Image Animator (LIA) [Wang et al. 2022 have demonstrated impressive performance. However, due to their specialized nature and dependence on predefined structural cues, parametric models have to be specifically developed for the target domain/category and typically generalize poorly beyond their narrowly defined application, not allowing for advanced applications such cross-category motion transfer.
In contrast to both explicit low-level conditioning and domain-specific parametric models, our approach aims to learn a highly abstract, invariant motion representation. By explicitly modeling motion independently of appearance, pose, viewpoint, and semantic category, we substantially enhance the flexibility and transfcrability required for diverse motion control scenarios.
3

---

Training
b) Autorggresine Sampling
X1
X;
mi
Xy
mig
Fe
Fq
F9
Augmnen
48
4T
4) Video Model Adaptation
10g
Motion Extractor M₁
Cim
M
mt
A
X +
Perraleed
X
VedroGGnmmtice Morid
ml
Frame Generator Fq
Delta Time Sampling
Legend
~
p(A,)
Daila caw
Merie Unbeoding
XL1A
Figure 2: Method Overview. (a) During training, our motion extractor Mo receives augmented frames from a video X, along with additional motion query tokens Q. These are then individually passed to the frame generator Fv. alongside the corresponding source frame Xr, from which it learns to reconstruct a frame at a future timestep + A, (b) To transfer motion sequence M onto another target image, we can directly utilize the trained Frame Generator Fg autoregressively as low-cost option. (c) For high-quality motion transfer, we adapt pre- pre-trained off-the-shell video generation models. A motion sequence is first embedded using mapping network, before being introduced to the frozen video model. The processed motion sequence is arranged such that each token at timestep t in the pre-trained backbone receives conditioning only from the temporally corresponding motion embedding m..
Method
Recent approaches in video modeling commonly utilize generative diffusion or flow-based objectives applied either directly on vidco frames [Ho et al. 2022 or on latent representations ex extracted from pretrained autoencoders [Blattmann al.) 2023 Brooks al., 2024 HaCohen et al. 2024 Kong et al.] 2024 Veo- Team et al. 2024]. While these have been successful for high-fidelig synthesis tasks, our goal difters fundamentally. instead of directly generating videos, we aim to learn abstract, motion representations that are invariant the appearance of the scene. Specifically, we define motion as the abstract temporal dynamics that describe how scenes and the objects therein evolve, independent from their actual appearance, style, spatial configuration, or even semantic category. The representations can subsequently serve as a powerful, generalizable conditioning signal for controlling motion in downstream video generation discriminative tasks.
3.1 Preliminaries
Flow Matching Flow matching (Lipman et al. 2022 is generative modeling framework that learss distributions via continuous-time flow field that smoothly transforms simple prior distribution- typically a Gaussian-into a complex target data distribution. Given data samples z Paata (z) they learn vector field v (2,, 7) with learnable parameters deseribing the flow of the distribution intermediate times 5 [0, 1]. This vector field is learned by minimizing the MSE loss
((v) E. 20.22.20 Vor, T) u(z,, T)12.
(I)

---

where Z, = = ~ Pdata, u(z ž1 is the target flow field, and Zo is drawn from the prior distribution, which is typically chosen to Gaussian Zp -N(,,)) Samples from this learned distribution can be generated by drawing from the prior and solving the learned ODE.
3.2 Learning Transferable Motion Representations
To learn abstract motion representations, we introduce DisMo, a method consisting of two primary components learned jointly: a motion extractor Ma and motion-conditioned frame generator Fø- The motion extractor computes meaningful motion embeddings from raw input videos, while the frame generator leverages these embeddings alongside source frames to reconstruct target frames. The overall framework is trained end-to-end using simple yet powerful generative image-space reconstruction objective, modeling the distribution of frames conditioned on source frame and learned motion embedding. The learned motion embedding inherently encodes temporal dynamics disentangled from frame contents and appearance. An overview of the architecture and training paradigm depicted in Fig.
Motion Extraction. Consider an input video X = (xx] consisting of T frames X,- From this, our goal is to extract sequence of motion embeddings M (mm))==- We formally define the motion extractor as Ma : X - M with parameters 0. This motion extractor jointly processes the given frames and extracts sequence of motion embeddings for all timesteps (1,..., 7). Each motion embedding encodes local temporal dynamics while ignoring static visual information.
Dual-Stream Frame Generation. The frame generator F parametrized by th, reconstructs target frames conditioned on a motion embedding and a source frame, and is responsible for providing the learning signal for the motion extractor. Formally, the generator learns the conditional distribution
Fø X p(X++A, Xx, m.; Ae)
describing the future destination frame X timestep l + Ar. As conditioning, the source frame Xt, the motion embedding at the source time m, extracted using our motion extractor Ma and the distance A, to the target frame are given. A, can either be constant or can be sampled independently for each prediction. Effectively, this provides the frame generator with two streams of available information: content information captured by the source frame X;, and information that is embedded into the motion representation m,, Due to the information bottleneck induced on the motion embedding via the limited embedding size, this representation is encouraged store only high-level, invariant temporal information. Essentially, this represents the residual knowledge not encoded by x, that supports the frame generator predicting the target frame X(+
We use this frame generator Fg to jointly learn both it and the motion extractor Mg end-to-end by (implicitly) maximizing the likelihood [Kingma and Gao (0223 of the reconstructed destination frames A conditioned on respective source frames and their corresponding motion emheddings:
Ex-pas. [log(p Xx, Me t))].
We practically implement this via a flow matching Lipman et al. 2022) objective (cf. Eq.
(() Ex Ex-paata77 (0000a
[2,, Xx, Ma (X, - u(z,, 7) 1
(4)
where z, TX (1 T)zo, and vo is the vector field learned by the frame generator Fe..
Augmentation Pipeline. To ensure that motion embeddings remain invariant to appearance, we additionally employ an augmentation pipeline inspired by self- supervupe learning methods (He et al.) 2022. Tong et al. 2022 [Cen et al. 2020). Appearance augmentations include random photometric transformations (e.g., hue, contrast, brightness adjustments) and geometric transformations (e.g., cropping, rotation, translation, shearing, aspect ratio changes), applied uniformly across all frames to avoid confusing these changes with actual motion. By applying these augmentations during training, we further encourage the learned motion embeddings to capture high-level, transferable dynamics rather than video-specific details.
3.3 Motion-Guided Video Generation
Once the motion extractor and frame generator are trained, they can be directly used for autoregressive motion transfer. Given a source video, we extract sequence of motion embeddings and, starting
5

---

from a target initial frame, we condition the frame generator on these embeddings at each time step to predict the next frame in an autoregressive manner (see Fig. 2p). While effective, this setup limited by the temporal scope captured in single motion embedding and the capacity of the frame generator. To address these limitations and enable longer, more coherent video generation, we further adapt off-the-shelf video generation model by conditioning them on sequences of motion embeddings. This approach allows us to leverage the strong generative priors of large video models while guiding their synthesis with our learned motion representation.
To adapt a pre-trained video backbone, we utilize LoRAdapters [Stracke et al., 2024], adaptive low-rank adapters [LoRAs, Hu et al. [022) that allow introducing additional conditioning into frozen backbone. We extend pre-trained spatio-temporal video model with conditional LoRAs that receive sequence of motion embeddings during the generation process as input. Importantly, the conditioning is designed such that each spatiotemporal token gets influenced only by the motion embedding that aligns with its temporal position. This makes the approach efficient, yet effective. This fine-tuning regime illustrated in Fig. 2 Our generated motion embeddings are generally applicable wide range of off-the-shelf video models, We demonstrates that DisMo can be adapted to various video models in the Appendix (see Table
while benefitting from their ad advantages.
4 Experiments
We evaluate our learning paradigm and show the effectiveness of the learned motion representations for motion transfer tasks. We further show that the motion embeddings are disentangled from appearance and structure-specifie information, preventing unwanted information leakage during transfer tasks. Finally, we demonstrate that our motion representations encode higher-level temporal dynamics that is useful in zero-shot action classification settings.
Implementation Details. We implement our motion extractor as 3D vision transformer Dosovit
skiy al. [202T] 86M params] with initial frame embedding stage consisting of DINOv2- [Oquab et al.; 2023] model, and adapt our frame generator from a pretrained diffusion transformer [Peebles and Xiel
components using AdamW [Loshchilov and Hutter 2019 for 530k steps at batch size of 32 on open-world videos
2023 675M params]. We train the
from K-710 al. [022]. SSv2 Goyal al. 2017], Moments in Time [Monfort al. 201 and
Vpen 1m Naa et al. [2225 to learn general purpose motion representations. Unless otherwise specified, we use LTX- Video-2B [HaCohen al. [024] as our video generation backbone that we adapted following the procedure described in Section 3.2 Evaluations with other video model back- bones, additional implementation details and ablations of our main design choices and components are provided in the Appendix.
4.1 Open- Open-World World Motion Transfer
a) Anto-Regressive Frame Generator
Driving Video
Tarpet Image
b) Adapted Video Model
Figure 3: Qualitative motion transfer comparison between (a) our auto-regressive frame gener- ator and (b) an adapted video model. We transfer motion extracted from driving video onto new target image using each model. While both approaches manage to transfer high-level motion semantics view- and appearance-invariant manner, the adapted video model achieves higher generation fidelity.

---

Setto-Distiibttoon Transfer
Camera Trajectories
Angle-invariann Multpart Motion Understanding
"A cortoon nubber duck is talking and luming is hood
"Four colorfu mo rolors are standing on- green Hil
"A mon is sitting at the oor gesturing with hiss zime
Text + Image Guidlance
MC
Text Guidenns
OPA
Figure 4: Motion transfer examples in different settings. We compare different motion transfer methods on three settings: (Left) Inter-category motion transfer. (Middle) An example showcasing camera motion. (Right) Composed motion transfer.
We qualitatively compare our auto-regressive frame generator and the adapted high-fidelity video model in motion transfer scenario in Fig. 3 While both successfully transfer higher-level motion from the source video onto the target image, the adapted video model yields a substantial improvement in generation quality. We therefore rely solely on the adapted model in the following experiments.
Specifically, we evaluate on general and out-of-distribution motion transfer scenarios and compare it against four state-of-the of-tte--tttttt art approaches: VMC [Jeong et al., 2024], DMT (Yatim et al. 2024]] MotionClone [Ling et al. 2024] and MotionDirector [Zhao et al. 2024 VMC and MotionDirector fine-tur fine-tuning- based methods that require extensive retraining per motion concept, whereas DMT and MotionClone are training- free approaches that leverage aggregated priors from a pre-trained video diffusion model for test-time optimization objective.
Unlike many motion transfer methods, such as VMC and DMT, which rely solely on text-based appearance guidance, method optionally incorporate start frame conditioning. This provides more nuanced control over the content, appearance, and structure the target entities. However, our approach also performs effectively using only textual prompts.
To comprehensively assess method performance, we build upon prior evaluation strategies (Zhao et al.] [2224] Jeong et al.) 2024 Yatim et al.) [2224] Park et al., [2024]. assembling a diverse set of videos and text prompts primarily from the DAVIS dataset [Pntt- Tuset et al., 2017]. Additionally, we include challenging examples from Open Vid-IM IM Naan et 2025 and other sources. Our evaluation set features varied scenarios such as stationary and moving cameras, in-domain and out-of-domain motion transfers, and creatively challenging out-of-distribution cases.
We quantitatively compare methods using three established metries: Motion Fidelity: Measured using a tracking-basse based metric inspired by Chamfer distance, evaluating alignment between generated and driving video tracklets [Yatim al. 2024]. Prompt Adherence: Evaluated by computing the cosine similarity between CLIP embeddings of generated frames and corresponding textual prompts. Temporal Consistency: Assessed by measuring the alignment of CLIP embeddings between consecutive frames. Driving Video Similarity: assessed by measuring alignment of CLIP embeddings of source and generated video frames. Our evaluation (see Table shows that DisMo consistently matches or outperforms other state-of-the-art methods across all four metrics. Notably, the evaluated models exhibit inherent trade-offs: DMT and MotionDrrector achieve high motion fidelity but frequently overfit on the exact structure of source videos, compromising textual adherence, while
7

---

Table 1: Quantitative Motion Transfer Evaluation. We compare with four other motion transfer methods with respect to motion fidelity, text adherence, temporal consistency, and similarity to the source video during our generative transfer experiments. DisMo performs best on all four metries, while other approaches struggle with the trade-off between motion fidelity and condition adherence.
Motion Fidelity
Prompt Adherence 1
Temporal
Driving Video
Method
Consistency 1
Similarity I
VMC*
Jeong
2024
0.57
0.26
0.94
0.59
DMTf
Yatım et al.
2024
0.70
0.24
0.93
0.66
Motion
lonet [ing ci al.
2024
0.63
0.27
0,91
0.59
MiriooDirecttor* Zhao et al.
2024
0.70
0.16
0.92
0.82
DisMo (Ours)
0.75
0.27
0.95
0.55
Per-sample tinetuning
LInference-time optimizationn
[Ceneral pro-trained
VMC and MotionClone maintain strong textual adherence the expense of accurate motion transfer. Meanwhile, our approach performs best on both metrics, not suffering from this trade-off. Qualitative examples in Fig. 4 further illustrate this. Other methods either fail transfer the object/camera motion faithfully or fail to adhere to the appearance of the prompt/start frame. DisMo's ability represent motion abstractly allows greater flexibility and generalization compared to existing approaches.
Furthermore, perceptual aspects such motion transfer quality are best assessed through human evaluation in addition to automated metrics. To this end, we conducted user study comparing several state-of-thee tert--rtt models, including own (DisMo), to evaluate their performance in terms of realism, prompt adherence, and motion transfer quality. As shown in Table 2 while MotionDirector achieves high motion transfer quality (25.96%), it exhibits notable decline in prompt adherence (7.47%), likely due to visual inconsistencies and content bleeding. In contrast, our method, DisMo, attains comparable motion transfer quality (24.71%) while substantially outperforming MotionDirector in both realism and prompt adherence. Moreover, DisMo achieves these results with markedly higher inference efficiency than existing methods. We refer the reader to Section in the supplementary for further details.
Table 2: Human Evaluation of Motion Transfer. We compare with four other motion transfer methods with respect to realism, prompt matching, and motion transfer quality. DisMo achieves the highest scores in realism and prompt matching by a wide margin, while remaining competitive in motion transfer quality.
Method
Realism (%)
Prompt Matching (%)
Motion Transfer Quality (%)
DMT Yatim et al.
2024
10.93
9.60
17.73
MotionDirector for Zhao el al.
2024
10.98
7.47
25.96
VMC Jeong et al
2024
20.04
26.13
16.98
MotionCione Ling cl al
2024
19.91
19.42
14.62
DisMo (Ours)
38.13
37.38
24.71
4.2 Appearance and Structural Disentanglement
Identity Reconstruction from Motion Representations. To assess the extent to which motion representations disentangle person-specific appearance from action dynamics, we designed a con- trolled identity classification experiment. Ideally, motion representations should capture dynamic patterns while minimizing appearance information. To evaluate this property, we utilize the Invariant Action Recognition Dataset (IARD) [acchettii al., 2019), which contains controlled recordings of five actors performing five different actions from multiple viewpoints. We extract features from videos of different actors performing a subset of actions and evaluate identity classification on disjoint action excluded from the training set, enabling analysis of identity-invarian motion rep- resentations. A k-Nearest Neighbors classifier (k = 20) was used to predict the identity of the actor based solely on the extracted representations. Higher classification accuracy indicates greater retention of appearance- to identity. We compare our method against VideoMAE, VideoMAEv2, and V-JEPA in Table 3
information, while lower accuracy suggests stronger invariance

---

The results reveal clear differences in the amount of identity information each model retains: Our method achieves 23.82% accuracy, close to the 20% random baseline for five-way classification, indicating minimal encoding of identity features. In contrast, VideoMAEv2 reaches 58. 58.94%, while V-JEPA and VideoMAF achieve 96. 26.23% and 99,14%, respectively. These findings suggest that our approach disentangles motion from appearance more effectively, making it particularly well-suited for identity-invariant motion understanding and transfer.
We further ablate action classification accuracy on IARD in similar setting. Specifically, we set up a &NN-based action classification experiment and adopt an identity- split: videos from one individual are held out for testing, while videos from the remaining four individuals are used for training. This evaluation protocol offers more realistic and challenging assessment by requiring generalization across identities, thereby measuring model's ' ability to capture action dynamics independently of person-specific cues. As shown Table 3 our representations achieve the best performance on this task, demonstrating that they are both structured and semantically meaningful.
Table 3: Identity Disentanglement. We evaluate disentanglement from appearance by performing identity classification on IARD, which contains annotations for both actions and identities. Low identity classification performance indicates high disentanglement, Our motion representations encode significantly less appearance information, while performing best on action classification.
Model
Action Accuracy 1
Identity Accuracy
Random (20%)
VideoMAE Toong et al. 2022
73.44
99.14
+79.14
VideoMAFV2 Wang et al. 2023
80.25
58.94
+38.94
V-JEPA [Bardes 2024
82.03
96.23
+76.23
DisMo (Ours)
90.74
23.82
+3.82
Appearance and Structure Invariance during Motion Transfer. We further evaluate the in- variance of motion representations under appearance and structural changes motion transfer setting. Starting with an original driving video, we construct two augmented versions: one with photometric transformations (e.g., changes brightness, contrast, and color) and one with geometric transformations (e.g., rotations, cropping, and spatial shifts). We then perform motion transfer using all three driving videos, the original and the two augmented versions, while keeping the same source prompt and seed. Ideally, invariant motion representation should produce minimal differences between these outputs. Hence, we compare the generated videos driven by the two augmented views against the generated using the unaltered video. We compute the CLIP distance [Radford et al.) 202T] as a semantic, SSIM [WangZhou et al., 2004) as a structural, LPIPS [hhang et al.) 2018 a patch-wiss Table 3, our method demonstrates significantly higher invariance towards these transformations compared to other motion transfer methods all metrics, indicating high degree of appearance and structural disentanglement. We also notice the positive effect that the augmentations provide during training. While model trained without augmentations nevertheless demonstrates substantially higher invariance compared to other baselines, a model trained with augmentations performs notably better, especially with regard to geometric transformttions.
well the L1 distance
pixel-wise similarity measurement. As shown in
Table 4: Generation Similarity under Driving Video Augmentations. We measure invariance between generated videos driven by unaltered and augmented input videos using both photometric and geometric transformations. DisMo showcases a significantly higher degree of invariance towards these alternations compared to the baselines. This effect is reinforced when applying augmentations during training.
Method
Photometric Augmentations
Geometric Augmentations
CLIP
SSIM
L1
LPIPS
CLIP 1
SSIM
L1
LPIPS
DMT Yatim et al. 2024
0.66
0.51
0.17
0.59
0.66
0.48
0.18
0.62
MotionDDrectta 7 hao el al. 2024
0.62
0.46
0.15
0.55
0.60
0.44
0.16
0.60
VMC [oong ct a. 2024
0.71
0.44
0.17
0.58
0.71
0.40
0.20
0.62
MotionCione [Lang et al. 2024
0.76
0.52
0.13
0.46
0.72
0.43
0.18
0.58
DisMo (wo augmentations)
0.70
0.09
0.26
0.88
0.65
0.11
0.33
DisMo (Ours)
0.90
0.72
0.08
0.25
0.90
0.71
0.08
0.26

---

4.3 Action Classification
DisMo's motion embeddings, while particularly useful for generative motion transfer tasks, demon- strate significant generalization potential for discriminative downstream applications due to their abstract representation of dynamic properties. To investigate their generalization and effectiveness, we evaluate DisMo embeddings on a discriminative task, namely action classification.
The strengths of DisMo's motion embeddings are particularly apparent in zero-shot action classifi- cation tasks on datasets where motion is primary discriminative cue. We benchmark our method against state-of-the-art video representation learning approaches on datasets that emphasize temporal dynamics and motion patterns, making them ideal for evaluating motion-centric representations. Specifically, we evaluate on ARID (Xu al., 202T], which tests robustness under low-light con- ditions: IARD [Tacchetti et al. 2019]. designed to assess adentity-invariann motion understanding; Jester [Materzynska al. 2019,, which comprises short hand gesture clips for evaluating fine-graiie ane-grained local motion sensitivity; and Something Soathihing V2 [Goyal et al. 2017]. large-scale dataset focused on nuanced temporal interactions.
Table 5: Zero-shot Action Classification. We compare DisMo with state-of-the-ar baselines in frozen evaluation with &NN probe on multiple downstream action classification datasets. Extracted features are mean-poole before being passed to the kNN classifier. All models are evaluated at resolution 224 using a single clip from the video. Compared to other video baselines, DisMo exhibits a consistent improvement across all downstream datasets, showcasing the superiority of abstract motion representations on zero-shot frozen evaluation over competing video representations.
Method
Arch.
Params.
Samples seen
ARID
Jester
SSv2
IARD
VideoMAE Tonn al. 2022
ViT-L/16
343M
410M
17.29
20.11
7.06
73.44
VideoMAE Wang aL 2023
VIT-L/16
304M
108B
32.61
43.83
16.56
80.25
V-JEPA [Bardes et al.
2024
VIT-1/16
200M
270M
25.16
30.84
21.11
82.03
DisMo (Owrs)
DisMo-B
172M
17M
57.29
56.66
22.19
90.74
The number samples reported here refers only to video clips and does include images used pre train the DINOv2 and DiT backhones.
Our model demonstrates strong performance across variety of zero-shot action classification benchmarks, with a notable lead in the k-NN setting (see Table On datasets like ARID, Jester, and SSv2, which all require understanding temporal motion patterns and generalize across diverse visual conditions, DisMo consistently outperforms state-of-the-art baselines. For instance, DisMo improves k-NN accuracy by large margin ARID (+24.68 points over VideoMAEv2) and achieves the highest scores on Jester and SSv2, indicating robust motion-sensitive representations. In the identity- conditioned IARD setting, DisMo achieves the highest k-NN accuracy (90.745 74%), outperforming all baselines, reflecting cleaner, less entangled representation, without any supervision or adaptation.
Our identity reconstruction analysis (Section 4.2 further supports this interpretation. We observed that DisMo encodes substantially less identity information than other models, which often reach near- perfect identity reconstruction. This property of disentangling motion from appearance enables better generalization across individuals and tasks, as the representations focus on action dynamics rather than performer-specific features. These results collectively underscore the promise of motion-centric representation learning for building generalizable and robust video models.
5 Conclusion
In this work, we introduce DisMo, a novel paradigm for learning abstract motion representations from video data using a purely reconstruction-hant hased objective. We demonstrate that these representations are inherently decoupled from appearance, making them well-suited for motion transfer. DisMo outperforms existing open-world transfer methods on qualitative and quantitative evaluations. Fur- thermore, we demonstrate that the learned embeddings are also effective for downstream tasks such as action classification, outperforming state-of-the-e the-aat video representation models in zero-shot setting. While our analysis focuses on zero-shot probing, exploring more advanced techniques such as linear or attentive probing could offer deeper insights into the structure and capabilities of the learned motion space, which we leave for future work.
10

---

Acknowledgements
This project has been supported by the project "GeniusRobot" (011S24083) funded by the Federal Ministry of Research, Technology and Space (BMFTR), the bidt project KLIMA-MEMES, the German Federal Ministry for Economic Affairs and Energy (BMWE) within the project "NXT GEN AI METHODS Generative Methoden für Perzeption, Prädiktion und Planung", and the Horizon Europe project ELLIOT (GA No. 101214398). The authors gratefully acknowledge the Gauss Center for Supercomputing for providing compute through the NIC on JUWELS/UPPITER at JSC and the HPC resources supplied by the NHR FAU Erlangen.
References
Gunnar Johansson. Visual perception of biological motion and a model for its analysis. Perception & psychophysics, 14:201-211, 1973.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models world simulators. 2024. URL https://openai com/research/ video- gereeoggnerttiinn models-aaeeeo as- world- simulators
Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Fitan Richardson, Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon, Poriya Panet, Sapir Weissbuch, Victor Kulikov, Yaki Bitterman, Zeev Melumian, and Ofir Bibi. Ltx-video: Realtime video latent diffusion. arXiv preprint arXiv:2501. 001003, 2024.
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03600 2024.
Veo-Team, :, Agrim Gupta, Ali Razavi, Andeep Toor, Ankush Gupta, Dumitru Erhan, Eleni Shaw, Eric Lau, Frank Belletti, Gabe Barth-Maron, Gregory Shaw, Hakan Erdogan, Hakim Sidahmed, Henna Nandwani, Hernan Moraldo, Hyunjik Kim, Irina Blok, Jeff Donahue, José Lezama, Kory Mathewson, Kurtis David, Matthieu Kim Lorrain, Mare van Zee, Medhini Narasimhan, Miaosen Wang, Mohammad Babaeizadeh, Nelly Papalampidi, Nick Pezzotti, Nilpa Jha, Parker Pieter-Jan Kindermans, Rachel Hornung, Ruben Villegas, Ryan Poplin, Salah Zaiem, Sander Dieleman, Sayna Ebrahimi, Scott Wisdom, Serena Zhang, Shlomi Fruchter, Signe Nørly, Weizhe Hua, Xinchen Yan, Yuqing Du, and Yutian Chen. Veo 2. 2024. URL https:tt// //deepmidd soogle//eechnooogies//eeeeee www.giiiiiiieeeeee2//2
Barnes,
Eric R. Kandel, James H. Schwartz, and Thomas M. Jessell, editors. Principles of Neural Science. Elsevier, New York, third edition, 1991.
Rui Zhao, Yuchao Gu, Jay Zhangjie Wu, David Junhao Zhang, Jia-Wei Liu, Weijia Wu, Jussi Keppo, and Mike Zheng Shou. Motiondirector: Motion customization of text-to-video diffusion models. In European Conference on Computer Vision, pages 273-290. Springer, 2024.
Hila Chefer, Uriel Singer, Amit Zohar, Yuval Kirstain, Adam Polyak, Yaniv Taigman, Lior Wolf, and Shelly Sheynin. Videojam: Joint appearance-motion representations for enhanced motion generation in video models. arXiv preprint arXivv2502.02499, 2025.
Jingyun Liang, Yuchen Fan, Kai Zhang, Radu Timofte, Luc Van Gool, and Rakesh Ranjan. Movideo: Motion-aware video generation with diffusion model. In European Conference Computer Vision, pages 56-74. Springer, 2024.
Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalía Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: better to track together. European Conference on Computer Vision, pages 18-35. Springer, 2024.

---

Robin Rombach, Andreas Blattmann, Dominik Lorenz. Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, June 2022.
Lvmin Zhang, Anyi Rao, and Mancesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3836-3847, 2023a.
Nick Stracke, Stefan Andreas Baumann, Joshua Susskind, Miguel Angel Bautista, and Bjöm Ommer. Ctrloralter: Conditional loradapter for efficient O-shot control and altering of t2i models. In ECCV, pages 87-103. Springer, 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-668 6851, 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint 0rXiv:2011.13453 2020.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arX1v:2408.06072, 2024.
Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yao Ma, Ching- Yao Chuang. David Yan, Dhruv Choudhary, Dingkang Wang, Geet Sethi, Guan Pang. Haoyu Ma, Ishan Misra, Ji Hou, Jialiang Wang, Kiran Jagadeesh, Kunpeng Li, Luxin Zhang, Mannat Singh, Mary Williamson, Matt Le, Matthew Yu, Mitesh Kumar Singh, Peizhao Zhang, Peter Vajda, Quentin Duval, Rohit Girdhar, Roshan Sumbaly, Sai Saketh Ramb- hatla, Sam Tsai, Samanch Azadi, Samyak Datta, Sanyuan Chen, Sean Bell, Sharadh Ramaswamy, Shelly Sheynin, Siddharth Bhattacharya, Simran Motwani, Tao Xu, Tianhe Li, Tingbo Hou, Wei- Ning Hsu, Xi Yin, Xiaoliang Dai, Yaniv Taigman, Yaqiao Luo, Yen-Cheng Liu, Yi-Chiao Wu, Yue Zhao, Yuval Kirstain, Zecheng He, Zijian He, Albert Pumarola, Ali Thabet, Artsiom Sanakoyeu, Arun Mallya, Baishan Guo, Boris Araya, Breena KerT, Carleigh Wood, Ce Liu, Cen Peng, Dimitry Vengertsev, Edgar Schonfeld, Elliot Blanchard, Felix Juefei-Xu, Fraylie Nord, Jeff Liang. John Hoffman, Jonas Kohler, Kaolin Fire, Karthik Sivakumar, Lawrence Chen, Licheng Yu, Luya Gao, Markos Georgopoulos, Rashel Moritz, Sara K. Sampson, Shikai Li, Simone Parmeggiani, Steve Fine, Tara Fowler, Vladan Petrovic, and Yuming Du. Movie gen: A cast of media foundation models, 2025. URL httpt:t//tx:xiii/ wttp//rg/aabbb//oaaae 2410. 13720
Yash Jain, Anshul Nasery, Vibhav Vineet, and Harkirat Behl. Peekaboo: Interactive video generation via maakedddiiffusion In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1079-8088, 2024.
Jiawei Wang, Yuchen Zhang, Jiaxin Zou, Yan Zeng, Guoqiang Wei, Liping Yuan, and Hang Li. Boximator: Generating rich and controllable motions for video synthesis. arXiv preprint arXiv:2402.01560 2024a.
Jinzznn Wu, Xiangtai Li, Yanhong Zeng, Jiangning Zhang, Qianyu Zhou, Yining Li, Yunhai Tong, and Kai Chen. Motionbooth: Motion- customized text-to-video generation. Advances in Neural Information Processing Systems, 37:34322-34348, 2025.
Zhouxia Wang, Ziyang Yuan, Xintao Wang. Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. In ACM SIGGRAPH 2024 Conference Papers, pages 1-11, 2024b.
Weijia WU, Zhuang Li, Yuchao Gu, Rui Zhao, Yefei He, David Junhao Zhang, Mike Zheng Shou, Yan Li, Tingting Gao, and Di Zhang. Draganything: Motion control for anything using entity representation. In European Conference on Computer Vision, pages 331-348. Springer, 2024.
12

---

Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory. arXiv preprint arXiv:2308. 2023.
Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. Con- trolvideo: Training-free controllable text-to-video generation. arXiv preprint arXiv:2305. 13077, 2023b.
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. Proceedings of the IEEE/CVF international conference computer vision, pages 7346-7356, 2023.
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. Advances in neural information processing systems, 32, 2019.
Aliaksandr Siarohin, Oliver Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov. Motion representations for articulated animation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 13653-13662, 2021.
Ting- Chun Wang. Arun Mallya, and Ming- Yu Liu. One-shot fireevieww neural talking- head synthesis for video conferencing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10039-10049 2021.
Jian Zhao and Hui Zhang. Thin-plate spline motion model for image animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3657-3666 2022.
Yaohui Wang, Di Yang, Francois Bremond, and Antitza Dantcheva. Latent image animator: Learning to animate images via latent space navigation. arXiv preprint arXiv:2203.0904: 2022.
Fa- Ting Hong, Longhao Zhang, Li Shen, and Dan Xu. Depth-aware generative adversarial network for talking head video generation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 2397-3406, 2022.
Kewei Yang, Kang Chen, Daoliang Guo, Song-Hai Zhang, Yuan-Chen Guo, and Weidong Zhang. Face2face p: Real-time high-resolution one-shot face reenactment. In European conference on computer vision, pages 55-71. Springer, 2022.
Yue Gao, Yuan Zhou, Jinglu Wang, Xiao Li, Xiang Ming, and Yan Lu. High-fidelity and freely controllable talking head video generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5609-5619, 2023.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans. Imagen video: High definition video generation with diffusion models, 2022. URL http:: http://xxivv..rr/ 222/ 22/0 002003.
Diederik Kingma and Ruiqi Gao. Understanding diffusion objectives as the elbo with simple data augmentation. Advances in Neural Information Processing Systems, 06665484-65516 2023.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data- efficient learners for self-supervised video pre-training. Advances in neural information processing systems, 35:10078-10093, 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICMI. pages 1597-1607. PMIR, 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
13

---

Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL het//forum2dd-iicbaail
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo. Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning Robust Visual Features without Supervision, April 2023. URL http://arxiv wrreiiii.rrg//ee org/ 230////////....////// 071933 arXiv:2304.07193 [cs].
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195 4205, 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https
//arxiv. wrg.iiiwwrg/oooooo abaaab///aa /etp//711.0000 05101
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Limin Wang, and Yu Qiao. Uniformerv2: Spatiotemporal learning by arming image vits with video uniformer, 2022. URL https://arxiv org//bs/ 2211 2211. 09552.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, al. The" something something" video database for learning and evaluating visual common sense. In CVPR, pages 5842-5850, 2017.
Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfruend, Carl Vondrick, and Aude Oliva, Moments in time dataset: one million videos for event understanding, 2019a. URL
1801.03150
Kepan Nan, Rui Xie, Penghao Zhou, Tiehan Fan, Zhenheng Yang, Zhijie Chen, Xiang Li, Jian Yang, and Ying Tai. Openvid-Im: A large-scale high-quality dataset for text-to-video generation. In ICLR, 2025. URL httttttt//p/otrsreetet
net net/forumiid-j7kkkkee
Hyeonho Jeong, Geon Yeong Park, and Jong Chul Ye. Vme: Video motion customization using temporal attention adaption for text-to-video diffusion models. In CVPR, pages 9212-9221, 2024.
Danah Yatim, Rafail Fridman, Omer Bar-Tal, Yoni Kasten, and Tali Dekel. Space-time diffusion features for sero text-driven motion transfer. In CVPR, pages 8466 8476, 2024.
Pengyang Ling, Jiazi Bu, Pan Zhang, Xiaoyi Dong, Yuhang Zang, Tong Wu, Huaian Chen, Jiaqi Wang, and Yi Jin. Motionclone: Training-fir free motion cloning for controllable vidco generation. arXiv preprint 24Xiv:2406.0533. 2024.
Geon Yeong Park, Hyeonho Jeong, Sang Wan Lee, and Jong Chul Ye. Spectral motion alignment for video motion transfer using diffusion models. arXiv preprint arXiv:2403. 15.449,,.224,, 2024.
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Dorkine-Hornune and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint (00:::70000007. 2017.
Andrea Tacchetti, Leyla Isik, and Tomaso Poggio. Invariant Action Recognition Dataset, 2019. URL https://doi.org /rgg///o..ooo00///// 10. /910//VN/DNNTTT DVN//MTON
Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14549-14555 2023.
Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video, 2024. URL https: /ttpvviivvv///////////aa///a.avv org/ www.///g//////e...o....o...eeeee 2204 2444/404040000
14

---

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-874 8863. PmL.R, 2021.
BovikAC WangZhou, Hamid R Sheikh, et al. Image qualityassessment: From errorvisibilityoostuee tural similarity. IEEE Transon ImageProcessing. 13(4):600, 2004.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and patlern recognition, pages 586-595, 2018.
Yuecong Xu, Jianfei Yang, Haozhi Cao, Kezhi Mao, Jianxiong Yin, and Simon See. Arid: A new dataset for recognizing action in the dark. In Deep Learning for Human Activity Recognition: Second International Workshop. DL-HAR 2020, Held in Conjunction with IJCAI-PRICAI 2020, Kyoto, Japan. January 8, 2021, Proceedings 2, pages 70-8 84. Springer, 2021.
Joanna Materzynska, Guillaume Berger, Ingo Bax, and Roland Memisevic. The jester dataset: A large-scale video dataset of human gestures. In Proceedings of the IEEE/CVF international conference on computer vision workshops, pages 0-0, 2019.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv: /00.:::7700065555 2017.
Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan. Lisa Brown, Quanfu Fan, Dan Gutfruend, Carl Vondrick, et al. Moments in time dataset: one million videos for event understanding. IEEE TPAMI, pages 1-8, 2019b. ISSN 0162-8828. doi: 10.1 109/TP..M.2200 2019.2901966 2901464..
H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre. HMDB: large video database for human motion recognition. Proceedings of the International Conference on Computer Vision (ICCV), 2011.
Noam Shazeer. Glu variants improve transformer, 2020. URL httpp::/aaxxiv.orraabbss 05202
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical Review E E-Statistical, Nonlinear, and Soft Matter Physicx, 69(6))066138, 38. 2004.
Brian C. Ross. Mutual information between discrete and continuous data sets. PLOS ONE, 9(2): 1-5, 02 2014. doi:
https://////ai org wtt......//.............. 1371/journal
pone 0087357
Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse controls to text-to-video diffusion models, 2023. URL https://axxii..orrt ort.abs////aa1 16933
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong. Xiaohan Zhang, Guanyu Feng, Da Yin, Yuxuan Zhang, Weihan Wang. Yean Cheng, Bin Xu, Xiaotao Gu, Yuxiao Dong, and Jie Tang. Cogvideox: Text-to-video diffusion models with an expert transformer, 2025. URLhttps: http:/tttts//aii..ataas /tt///aaxvvo//aaaaa 20tpbb//4080007777
Qiuheng Wang. Yukai Shi, Jiarong Ou, Rui Chen, Ke Lin, Jiahao Wang, Boyuan Jiang, Haotian Yang. Mingwu Zheng, Xin Tao, Fei Yang, Pengfei Wan, and Di Zhang. Koala-36m: A large-scale video dataset improving consistency between fine-grained conditions and video content, 2025. URL https://arxxv..rrg wbs./or.//aa/./..ooooo........s/ 24000 08266
Christoph Feichtenhofer, Axel Pinz, and Richard P Wildes. Temporal residual networks for dynamic scene recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 4728-4737, 2017.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The claims made in the abstract and introduction are accurate and aligned with the contributions and scope of the paper. They clearly state the novelty of our proposed approach and the empirical findings that support our conclusions. Limitations are also appro- priately acknowledged ensure the claims do not overstate the generality or applicability of our results.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: the paper acknowledges key limitations, including generalization challenges and potential biases.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
16

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not contain any theoretical results; it is entirely empirical. Therefore, this question not applicable.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The paper introduces new modeling paradigm and includes all necessary details to reproduce the main experimental results. We provide a clear and complete description of the model architecture, training procedure, datasets used, and evaluation metrics. Furthermore, we outline the key hyperparameters and implementation choices critical to the validity of our claims. This ensures that the results and conclusions can be independently verified.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
17

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We include anonymized code and detailed reproduction instructions in the supplementary material to ensure faithful replication of our main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
publicc/guides/CodeSuuubue wwtpaaumisssiiooooooooooooo onPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips. ctajps.cc/puulic/uuiiae gwwp///ieee//eee/ Codeeubmii ssi ssionnolicy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The main paper includes a clear description the experimental setup necessary to understand and contextualize the results, including dataset splits, model architectures, and key hyperparameters. Full training and testing details-sure such as exact hyperparameter values, optimizer settings, and selection criteria-ae are provided in the supplementary material to ensure reproducibility without detracting from the clarity of the main text.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
18

---

Justification: Due to the high computational cost of training and evaluating the proposed models, was not feasible run multiple independent trials to compute error bars or statistical significance metrics. While we acknowledge the importance of such measures for assessing the robustness of experimental results, the scale of the experiments presented in the paper made repeated runs impractical given our available resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report compute details including GPU type, memory, and training time in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics stttps://neurips httpppblicc/fthhccccccieeeeeeeee
Answer: [Yes]
The research conducted fully complies with the NeurlPS Code of Ethics. does not involve human subjects, crowdsoureing, sensitive data. The paper promotes transparency and reproducibility, and all experimental procedures adhere to ethical research standards.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
19

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses both the positive potential of the method as well as the possible negative misuse of it in the supplementary.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymennt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We aim to release our models under a license that requires responsible use, and we mention that in the supplementary material.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
20

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes)
Justification: The licenses, terms use, and proper credits for all external assets (e.g.. datasets, code) used in this work are explicitly provided in the supplementary material.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide anonymized code and documentation in the supplementary mate- rial, and will release our models publicly upon acceptance.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve any crowdsoureing research with human subjects. All experiments are conducted using publicly available datasets or synthetic data, and no human participants were involved in data collection evaluation. Therefore, this question is not applicable.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
21

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve any research with human subjects the of crowdsourced data. Therefore, Institutional Review Board (IRB) approval or equivalent review applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Large language models (LLMs) were not used as an important, original, or non-standard component of the core methodology in this research. The work is centered around motion modeling approach developed independently of L.L.-baseed methods.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy httt::::/eerrppsscc////ffferreee for what should or should be described.
22

---

A Broader Societal Impact
While this work aims to advance realistic motion transfer with potential applications digital storytelling, entertainment, and educational purposes, such progress also carries risks of misuse to generate harmful content without consent. We emphasize transparency datasets and evaluation, and encourage ongoing efforts to address biases and promote responsible deployment.
B Limitations
Our work is limited by the capabilities of the flow-matching- based decoder, which, while efficient, may struggle to generate -ege-fidliity or semantically rich images in complex scenarios. Additionally, the performance is constrained the distribution and inherent biases of the training dataset, which can affect generalization, fairness, and robustness to out-of-distribution samples.
C Implementation Details
C.1 Pretraining details
Architectures We use DINOv2-B frame embedder. As sequence embedder 3D ViT-B/16 with RoPe positional embeddings. As our frame generator, we use a DiT-XL The frame embedder takes input a short video clip of frames, sampled 6 fps.
Table A: Pretraining aaper-parameerrs for our model.
Category
Hyper-parameter
Value
datasets
K- X=7100SS22MIT/OQeeeiilllll
resolution
256 256
data
num frames
fps
6
scale
(0.75, 1.0)
translate
(-0.3, 0.3)
angle
(-30, 30)
geometric augmentations
shear
(-15, 15)
horizontal flip
padding mode
border
brightness
(0.5, 1.5)
contrast
(0.5, 1.5)
photometric augmentattioss
huc saturation
(-0.3, 0.3)
(0.5, 1.5)
batch_size
32
total number of iterations warmup iterations scheduler Ir
530k
5000
optimization
constant w/ warmup
10-4
AdamW /
(0.9, 0.95)
frame_ seine_emhed__ee depth
12
Trame_ember dim
768
sequence_embed embed depth
12
architecture
sequence_embeer dim cateeeeeeeertt__deege frame_generator_diig
768
28
1152
dtype accelerator
bfloat
hurdware
GH200 96G
Training Data To cover wide variety of real-world videos, we train a mixture of datasets
covering over videos. This mixture is composed of Kinetics-710 [Kay et al. (017]. which combines Kinetics-400, 600 and 700, Samethingg-Somehhing.. (SSv2) [Goyal et al. 2017], Moments in Time
23

---

(MiT) [Monfort et al., 2019b], and Open VidIM Nan et al. 2025]. We show an overview of the size and domain of the various datasets in Table B Overall, this dataset composition is similar to that used by V-JEPA [Bardes et al. 2024], which among others combines Kinetics-710 with Jomething-Something---
We further extend this mixture with the high-quality Open Vid-IN [Nan]
et al. 2025 dataset, which is more suitable for generative tasks compared to the other datasets.
Dataset
Domain # Clips
Duration [h]
Kinetics-777 710 Kay et al. 2017 Goyal et al. 2017
human-centered
0.65M
1.8k
open
108k
121
Moments Time [Monfort et al. 20196
open
1M
833
OpenVid- IM (Nan et al. 2025
open
IM
2.1k
Combined
open
2.8M
4.9k
Table B: Training Data Composition.
Kinetics-7I0 Overlaps There are significant overlaps between train and evaluation splits of the various subsets (K-400, K-600, K-700) that comprise the Kinetics-710 dataset. To enable fair comparisons, we remove any clips present in any of the three validation or test sets from our training dataset. We also de-duplicate the remaining training dataset by their respective YouTube video ids.
C.2 Evaluation details
Datasets Something-Something v2 (SSv2) Goyal et al. 2017 is a large dataset of humen-object interactions designed to test fine-grained motion understanding while minimizing reliance on back- ground or object identity. Kinetics-400 (Kay et al. [017) contains 240,000 YouTube clips labeled with 400 action classes and serves understanding. ARID (Xu et al. 2 (Action Recognition in the Dark) includes 5,500 RGB-I) videos captured in low-light settings to evaluate robustness to visibility degradation. Jester [Materzyn ska et al., 2019) contains over 148,000 short clips of hand gestures, providing a benchmark for evaluating fine-grained, local temporal motion sensitivity. IARD (Tacchetti et al. [019] (Invariant Action Recognition Dataset) includes controlled recordings of five actors performing five actions from multiple viewpoints, enabling analysis of identity- and viewpoint-invariant motion recognition. HMDB51 (uuhhne al.) [011] is a widely used benchmark for human action recognition, consisting of ~7000 video clips distributed across 51 action categories, collected from various sources including movies and YouTube, with diverse scene complexity and motion dynamics.
general purposs benehmark for aypearance-driveen video
LTX Finetuning details We adapt both the attention and feedforward layers of L.TX using conditional LoRA modules with a rank of 64. Each LoRA module receives motion sequence as input. We use batch size of 64 for training. For optimization, we use the AdamW optimizer with 81 0.9 and B/- 0.99, learning rate of 5 x 10-5, weight decay of 0.0001, and training is performed with 32-bit precision.
LTX operates on high frame rate videos and temporally downsamples them by a factor of 8 using its VAE. contrast, our motion extractor processes videos lower frame rate, taking only 8 frames as input. To ensure alignment between the motion sequence and the LTX latent representations, we apply the following strategy:
We sample video at approximately 24 fps with total of 29 frames, where the first frame serves as the source frame.
These 29 frames are chosen such that we can extract 8 frames spaced 4 frames apart: the source frame (frame 0), followed by frames 4, 8, 12, 16, 20, 24, and 28. This satisfies the equation:
1 x 4 = 29
LTX downsamples this video by encoding the source frame and every 8th frame thereafter (frames 0, 8, 16. 24) into latent representation.
To align motion embeddings with these latents, we extract every 4th frame (frames 0, 4, 12, 16, 20, 24, 28) and pass them through our motion extractor.
24

---

For each pair of consecutive motion embeddings (except for the first one, which remains singular embedding), we concatenate them to form a combined embedding. This results in motion sequence of 4 embeddings aligned with the 4 L.TX latents.
This sequence is then transformed by a mapping network and used to condition the LoRA modules. The mapping network consists of two feedforward layers. each applying RMS normalization at the beginning and end, and using a Linear-GEGl.. [Shazeer, 2020] activa- tion. Each feedforward layer expands the dimensionality by factor of and subsequently reduces it back, operating at a width of 768.
Identity Reconstruction from Motion Representations We provide additional details on the identity classification experiment on the IARD dataset. We selected videos of 5 individuals and used only four actions (['eat', 'run', 'walk', , 'jump']) for training, holding out the 'drink' action for testing. Motion representations were extracted using VideoMAE, VideoMAEv2, and Video JEPA. Videos were processed 6 FPS with 16-frame windows, resized to 224 224, and uniformly preprocessed across models. Temporal features were averaged to produce single embedding per video. k-NN classifier (k: 20) was trained to predict identity based on the training actions and evaluated on the held-out 'drink' action to assess the extent of identity information encoded in the representations.
LARD bdset split for action classification In this experiment on the Invariant Action Recognition Dataset (IARD), we used an identity-baseh split better test the generalization of motion representation models across individuals. The dataset includes 5 people performing 5 actions (walk, run, jump, drink, eat). We trained models on videos from four individuals (Andrea, Leyla, Gu, Georgios) and tested on videos from the fifth (Steve), ensuring no overlap in identity between train and test. This split contrasts with standard random splits and provides a more realistic measure of how well models capture action dynamics independently of personal appearance cues.
C.3 Ablation Study
During training, introduce several key strategies to enhance disentanglement between motion and appearance in our model. Additionally, we apply dropout to motion and frame conditioning, which serves a dual purpose: it stabilizes the training process and allows for subsequent evaluation of the frame-unconditional reconstructor. This evaluation is conducted by reconstructing frames solely from motion embeddings, without previous frame conditioning.
To comprehensively measure performance during ablation, we employ three complementary metries: LPIPS [Zhang et al. 2018). CLIP [Radford et al. [022]] score, and Mutual Information Ratio (MIR). Ideally, if no appearance information leaks into the motion embeddings, we expect poorer reconstructions, indicated by higher LPIPS scores and lower CLIP similarities. We use subset of the HMDB-51 dataset for LPIPS and CLIP, and a subset of the IARD dataset for MIR.
With LPIPS we quantify the perceptual similarity with respect to visual realism, whereas with CLIP we measure the overall semantic similarity between the reconstructed frame and the original one. MIR provides a formal quantification of disentanglement by comparing the mutual information between the motion embeddings and two distinct factors actions and identities. Formally, we define MIR as:
[(2; Caction)
MIR
T(2; Cadentty)
where Caction and Cidentity are the respective labels for embeddings a and the mutual information ((::.) measures the shared information between variables, defined as:
T(2: v) H(y) H(y =),
(2)
with entropy (-) quantifying uncertainty, We estimate entropy using K-NN (k 5) distances, as described in [Kraskov et al. 22044) and [Ross] 2014].
Table C summarizes the outcomes of our ablation study. Starting from a baseline model without conditioning or augmentations, adding previous frame conditioning significantly prevents appearance leaking and improves disentanglement. Integrating augmentations further boosts these improvements, especially regarding disentanglement. Collectively, these results underscore the efficacy and necessity of each component in achieving robust disentanglement between motion and appearance.
25

---

Table C: Ablation Study, Comparison of model variants with and without previous frame condi- tioning and augmentations.
Table Disentanglement. Comparison of ap- pearance disentanglement performance between our model and V-JEPA.
Model
LPIPS
CLIP
MIR
1
Model
LPIPS 1
CLIP
MIR 1
Baseline
0.47
0.81
0.47
V-JEPA
0.70
0.70
3.72
Previous frame
0.72
0.65
3.07
Ours
0.76
0.65
5.56
Augmentations (ours)
0.76
0.65
5.56
To enable direct performance comparison with V-JEPA, we trained an identical reconstructor using the same setup as our model, with the motion extractor replaced by frozen V-JEPA. As shown in Table our model achieves superior disentanglement.
We additionally ablate DisMo with other high-quality video generation backbones, namely SparseC- trl [Guo et al., 2023] and Cog VideoX-5B (Yang al. 2025]. As shown in Table B using a more capable generator like CogVideoX-577 leads to improved motion fidelity and temporal consistency, without any modification to the motion embeddings produced by DisMo. This demonstrates that DisMo is generally applicable to off-the-shelf video models without being constrained to a specific one. We further assessed the quality and realism of the generated videos using FID and FVD scores, observing consistent improvements with stronger video backbones. This demonstrates that DisMo benefits from more powerful architectures and remains compatible with future models. For this evaluation, we generate 2,000 videos per model, each 25 frames long, yielding 50,000 frames in total for FID computation. Motion is transferred from randomly sampled videos and prompts from the Koala-36M [Wang et al. 2025 dataset.
Table E: Ablating Video Generation Backbones for Motion Transfer. Effect of different video generation backbones on temporal consistency, motion fidelity, and visual quality metries (FID, FVD) of generated motion transfers. Using stronger generator like Cog VideoX improves motion fidelity and temporal consistency without altering DisMo's motion embeddings.
Model
Temporal Consistency 1
Motion Fidelity
FID
FVD
LTX [HCoohn et al. 2024
0.95
0.75
88.54
298.4
SparseCir! Guo et al. 2023
0.95
0.73
116.8
327.9
Cog VideoX-5B Yang al. 2025
0.97
0.78
62.99
206.3
D Latent Space Analysis
D.1 Cluster Visualization
We visualize embeddings from both DisMo and V-JEPA on the IARD dataset, which provides action and identity labels for each video sample. Using UMAP for dimensionality reduction and coloring the resulting sample points by either action or identity reveals notable differences between the two models, see Fig. A When colored by action, embeddings from our model form well-defined clusters, while coloring by identity results in no clear grouping indicating strong sensitivity to motion and minimal association with appearance. In contrast, V-JEPA's embeddings show little to no clustering by either action or identity. Instead, its clusters primarily reflect viewpoint differences, consistent with the five distinet camera angles present in the dataset.
groupig--indcating
D.2 Cycle Identification
Fig. Bisoows PCA visualizations of individual videos from three different classes in the IARD dataset: eat, walk, and run. The repetitive nature of these actions produces distinct cyclic patterns in the latent space. Slower movements, such as eating, yield smoother and more continuous trajectories, while faster actions like walking result in less smooth, more abrupt trajectories. None-repeating motions, such as someone diving into pool, do not result in cycles. This highlights how differences motion intensity and cycling movements captured in our model's 's latent representations.
26

---

Actions
Identities
V-JEPA
V-JEPA Identities
ponglts
UMAP 1
UMAP
UMAP
UMAP
Figure A: UMAP visualization of the IARD dataset. Compared V-JEPA, our model shows better grouping by action and almost no grouping by identity.
Eat
Walk
Dive
UNSAP 3
JMAP
UNAP
Figure B: PCA cycle visualization of three different classes from the IARD dataset. Three videos per class, indicated by different colors, showing clear cycles in our model's latent representations.
D.3 Reversibility
A task that is solely based motion, where appearance provides no useful cues, is determining whether a video is played forward or backward. In such cases, temporal directionality must be inferred from the dynamics of the scene alone. To test this, conducted experiment where each video sample is paired with a temporally reversed version. We then visualize the latent representations using UMAP. As shown in Fig. C our model separates forward and reversed sequences in the latent space, but only when the action is inherently irreversible, such as diving into a pool. In contrast, when the motion is reversible e.g. someone doing push-ups. no meaningful separation emerges. V-JEPA, on the other hand, fails to distinguish forward and reversed sequences in both reversible and irreversible cases. Its latent space shows no significant separation, suggesting its representations are largely appearance biased.
V.JEPA
V-JEPA
Dive
floor
Ours
VJJEAA
Ours
V-JEPA
Coll
Pustopp treveraal
LMMP2
Sut Ccl (revaton)
Redragg
UMAP 1
UMAPP
UMAP
UMAP1
Figure C: UMAP visualizations of reversible motion and irreversible motion on the HMDB-51 dataset. Compared to V-JEPA, our model shows a better separation for irreversible motion.
27

---

E Composite Motion Analysis
We designed a targeted experiment to evaluate the composability of DisMo's motion embedding space. To assess which aspects of motion are encoded in the embeddings, we use the YUP++ dataset [Feichtenhofer et al.] 2017), which contains 600 static videos without camera motion. We then construct three types of video clips:
1. Object Motion Only. A randomly sampled clip from the dataset, containing natural object motion but no camera movement.
2. Camera Motion Only. A different clip, augmented with temporally smooth geometric transformations simulating camera motion, applied only to the initial frame (thus eliminating object motion).
3. Combined Motion.
The sampled clip with both object motion and the same simulated camera motion applied across all frames.
This setup allows us to isolate the two motion components, camera and object, and test whether embeddings extracted from the combined motion videos contain information from both individual mo- tion types. We sample 3200 such triplets, each containing the three types of clips mentioned before, and encode each of them using DisMo's motion encoder. We then estimate the mutual information (MI) between the embeddings of the combined- mottin clip (Maoth) and those of the camera-only Mcam) and object-only Moby) clips, respectively. High mutual information would suggest that Mooth shares information with Meam and Mobj. which a key indication of composability.
To quantify this, we use the Traskov-Stögbauer Srruoo---tgebbuueeerrrrrggrrr (KSG) estimator [Kraskov et al. 2004] with k = 5. To compute statistical significance, we further estimate MI under 100 random permutations (i.e., computing MI between unpaired clips), yielding a null distribution with means of around 0.2 nats and standard deviations of around 0.4. The observed MI values between both (Mooth Moam) and (Mooth. Mobj) are depicted in Table F They are significantly higher than the null baseline, indicating a strong statistical dependency and thus supporting the hypothesis that DisMo's s latent space is compositional with regard to simultancous camera and object motion.
Table F: Motion composability. We compute the mutual information (MI) between motion embed- dings extracted from videos with both camera and object motion, and embeddings extracted from videos consisting of only camera and object motion, respectively. We observe high significance compared to a random baseline for both types of motion. Object motion MI is significant even with strong camera motion, albeit decreasing the heavier the added camera motion gets.
MI (nats)
Light Camera Motion
Medium Camera Motion
Heavy Camera Motion
Camera Motion Object Motion
1.37
2.25
4.40
3.59
2.97
1.57
F
Motion Transfer Efficiency
We report inference time and number of parameters for DisMo and other baselines during motion transfer in Table DMT, VMC, and MotionDirector rely on per-sample fine-tuning or optimization, which significantly decreases inference efficiency. In contrast, forward generator. Across all configurations, we report DisMo to be the most efficient approach except when paired with Cog VideoX-BB as the video generation backbone, which increases inference time due to sheer model size. However, it is important to note that using Cog VogeiooX-558 is not required achieve strong motion transfer results: evidenced by Ta DisMo combined with lighter backbones such as LTX-Video-2B already achieves better or comparable motion transfer fidelity.
method is an off- the-shelf feed-
G Computational Resources for Experiments
This section gives an overview of the compute resources required to run the various individual experiments presented in this paper. We primarily use Nvidia GH200 96GB modules in an internal
28

---

Table G: Inference Efficiency Comparison. Inference time and parameter count of each model on an A100 GPU. Optimization-base based baselines (VMC, DMT, MotionDirector) require per-sample fine-tuning, leading to slow inference, while our feed-forward DisMo variants enable substantially faster generation across all configurations.
Model
Inference Time (A100)
# Params
VMC
10 min video
6B
DMT
7.5 min / video
1.7B
MotionClone
45 see / video
0,983B
MotionDirector
~5 min / video
0.983B
DisMo-CogVides
2 min / video
6.072B
DisMo-LTX
30 sec video
2.172B
DisMo-S DisMo-SparseCCn
10 sec / video
1.115B
The reported number of puramntters is approximate and refers the largest component of each model, the best of knowledge. Exact values may vary depending on implementation details and auxiliary components.
cluster for pre-training and Nvidia H200 141GB in an internal cluster for evaluations. The project used approximately 600 H200-hrs across all experiments, including initial and other explorations that did not directly make it into the paper. In the following, we detail the required resources for each experiment.
Pre-training: 192 H200 hours.
Ablations: 2 x 96 H200 hours.
Action classification: 8 H200 hours
Identity classification: 0,66 H200 hours
Motion transfers: 17.2 H200 hours
LTX fine-tuning: 200 H200 hours
H Licenses of Used Assets
H.1 Licences video datasets
TARD
License: CCO 1.0.
Terms: Community Norms in https://dataverse. org/best-pratticcs dataverse- wttteo-oooonnttttooeeee
as well as good scientific practices expect that
proper credit given via citation.
htt.r//ee....arrrrdeeeetttttttte http://sttsttt.caaapppppeereeeee Source: attps://dataveree
10. 70.77990/////CCE
Something-Someething v2 (SSv2)
License: Research Use License (Qualcomm).
Terms:
Available for research purposes only. Users must agree to Qualcomm's license terms
before accessing the dataset.
Source:
https://wwi :/tp:////www.// http://ww.aaalllooocc////eeererr
something wwwsssieeetttntne 2-dataet
Jester (Gesture Recognition)
License: Research Use License (Qualcomm).
Terms:
Available for research purposes only. Users must agree to Qualcomm's license terms
before accessing the dataset.
Source: httpp:://///.. //ww::/wwwwww httt:s///www...aaacccccco//o//// http://www.......ooooooo/oooo/oa
ARID (Action Recognition the Dark)
License: Creative Commons Attribution 4.0 International (CC BY 4.0). Terms: Free use, modify, and redistribute with proper attribution.
Source: https: atttps//xuyy00111 watpyyyuo01ogggthtt. aarid.
29

---

OpenVid-1Ny
License: Creative Commons Attribution 4.0 International (CC BY 4.0).
Terms:
The video samples are collected from publicly available datasets. Users must follow the related licenses Panda, ChronoMagic, Open-Sora-plan, CelebvHQ(Unknow)) to use these video samples.
Source:
Kinetics-710
License: Not explicitly specified.
Terms: Kinetics-710 video benchmark built upon Kinetics-400, -600, and -700. Its usage is subject to the licensing terms of these underlying datasets.
Source:
btops:://gthule com/ haaammmmlllbbbee http://////////////////////////a
S0metics710/REAMMG md
Moments in Time
License:
Not explicitly specified.
Terms: Intended for academic research.
Source: httpp:/momen:: csail www. edu
HMDB51
License: Creative Commons Attribution 4.0 International (CC BY 4.0). Free use, modify, and redistribute with proper attribution.
Terms:
Source:
https://serre- lab. clps brown. cowooneed/reeouueee
hmdb a- aagaaaaaarn-uuuuaooooooooattaaaa
I Qualitative Comparisons on Text-based Motion Transfer
We present additional qualitative results for motion transfer and compare them with the previously introduced baseline methods. As most existing approaches do not support motion transfer from source image, we restrict our comparison to examples generated solely from text prompts. All experiments were conducted using our adapted LTX model on videos that were not seen during training.
These results further demonstrate the effectiveness of abstract motion representation in enabling motion transfer across diverse scenarios. Our approach consistently preserves both the high-level motion semantics and the content specified in the text prompt without relying on the structure the driving video, a trade-off that competing methods often struggle to achieve.
In Fig. D Fig. Fig. and Fig. G show that approach outperforms existing methods in cateorrt motion transfer, especially when structural differences exist between the reference and generated videos. Additionally, our abstract motion representations enable effective cross- coteegyy transfer, allowing motion to be applied across semantically different domains. We demonstrate this capability in Fig. H Fig. [ and Fig. J where motion successfully transferred across categories. In contrast, existing methods often struggle in such scenarios, as they typically rely heavily on the structure of the input video and face trade-off between accurately transferring motion and remaining faithful to the content described in the text prompt. Our method is decoupled from the reference structure and instead describes motion as the change between states. This becomes apparent in Fig. [ where DisMo applies the change of the pose, rather than aligning the structure. We showcase that our approach performs extraordinarily well for camera motion transfer while being completely independent of the reference structure. This capability is demonstrated in Fig. Fig. M and Fig. N which highlight examples of successful camera motion transfer.
30

---

Reference
"A man with a beard and glasses is standing against solid blue background. He is wearing black jacket, blue beanie, and black gloves. The man appears to be speaking or gesturing [...]'
Ours
DMT
VMC
MD
MC
Figure D: Same-category Transfer. The reference video displays man leaning towards the camera, rotating his torso and raising his arm. Our method is able to transfer these movements to new identity with differing appearance, pose, and structure. Other methods either fail to capture the motion completely (VMC, MD) rely heavily the source structure (DMT, MC).
31

---

Reference
"A standing in front a display of motorcycle helmets, holding and examining racing helmet with vibrant red, white, and black design. wearing dark purple t-shirt [...]'
Ours
100
DMT
VMC
MD
MC
Figure E: Same-category Transfer: The reference video shows the upper torso of a person, moving their head and upper body. Our method successfully captures this motion and transfers it to the generated scene, closely following the instructions of the text prompt. Other methods fail to interpret the motion and the input prompt in a meaningful manner, the only exception being VMC.
32

---

Reference
"A man stands beside vibrant red and white classic muscle car, smiling and engaging with the camera. The car, vintage Pontiac GTO, is parked indoors, showcasing its sleek design [...]'
Ours
DMT
VMC
MD
MC
Figure F: Same-category Transfer. The reference video depicts talking head in dark setting, slowly leaning to the left-hand side. Our approach effectively captures the details of the text prompt and accurately attaches the reference head movement to the semantically appropriate part of the target video, despite significant structural differences. In contrast, other methods fail to transfer the reference motion correctly.
33

---

Reference
"A man and woman are sitting at table in a cozy, well-lit room, engaging in conversation while eating snacks. The woman, wearing glasses and an orange sweatshirt with Halloween theme [...]'
Ours
UVT
WO
DMT
VMC
MD
MC
The reference video depicts close-up up shot of a person looking at the camera while performing head and arm movements. Our approach successfully transfers the nuances of this motion to the two target characters and even reinterprets to represent the action described in the text prompt (eating). In contrast, DMT overly relies on the reference structure, limiting its ability to capture the intended semantics. Furthermore, VMC, MD, and MC fail to capture key nuances of the reference motion, such as the arm movement.
Figure G: Same-category Transfer:
34

---

Reference
PGS MRGPLLLOS B/D GUT CLURIO
PRG US PLOO LE
VIG CAL
MESPPPPLLLLLL AOFICUUP
mature buck with impressive antlers walking through grassy field dry, tall grasses sparse the background. [...]'
Ours
DMT
VMC
MD
MC
Figure H: Cross-category Transfer. The reference video shows a person rotating their head. Among all methods, our method and MC are the only approaches capable of performing this cross-category transfer effectively. Other methods either leak the appearance from the source video or fail to transfer the motion accurately.
35

---

Reference
"an animated character, a cartoonish raccoon, who is lying in bed and appears to be waking up. The raccoon's facial expressions change from sleepy to wide-eyed and surprised [...]'
Ours
DMT
VMC
MD
MC
Figure I: Cruss-category Transfer. The reference video shows the upper body of person performing rotating head movements. Our method successfully identifies the relative head pose changes as motion and applies them the target character, starting from different initial head pose. In contrast, most other methods fail to capture this motion. The only exception MC, which, however, remains tied to the reference pose.
36

---

Reference
BAUAAS
BAUARK
BAKAR
SAGAR
LANAS
"a group of guinea pigs cozy, grass filleed enclosure. The guinea pigs actively foraging and interacting with each other [...]'
Ours
DMT
VMC
MD
MC
Figure J: Cross-category Transfer. The source video depicts woman holding cup and moving with both hands to the left side. Our method is the only approach that effectively captures this motion transfer across categories. It interprets the rotation and translation of the reference objects (cups and arms) as the movements of the guinea pigs, that serve as the general directional guidance. MD the only other method that performs reasonably well, while most others fail significantly.
37

---

Reference
2274
woman is kitchen preparing a cake batter using red stand mixer. She is wearing a floral dress and glasses, and her hair is curly. She is seen mixing the ingredients a large metal bowl [...]'
Ours
DMT
VMC
MD
MC
Figure K: Camera Motion Transfer. The reference video depicts rotating camera movement around a car. Our method successfully transfers this motion while remaining faithful to the text prompt. In contrast, other methods either overly rely the input structure (e.g.. DMT) leak visual appearance (e.g.. MD), failing to reflect the prompt accurately. While VMC and MC capture aspects of the prompt, they largely disregard the reference motion.
38

---

Reference
'A anchor delivering report television The anchor with short, dark
hair. [...]'
Ours
DMT
VMC
MD
MC
Figure L: Camera Motion Transfer. The reference video solely shows zoom-in to the top left of the frame. Our method captures this motion even while transfering to a substantially different target category, while the others fail to capture it or even fail to produce realistic videos due to the strong change category.
39

---

Reference
"a close-up view of freshly baked loaf of bread adorned with sliced strawberries. The bread is placed white plate [...]'
Ours
DMT
VMC
MD
MC
Figure M: Camera Motion Transfer. The reference video depicts a camera zoom-out combined with rotation. Our method the only approach capable of capturing both the intended motion while maintaining a coherent and realistic generated video.
40

---

Reference
scene of two white vans parked side by side on a wet, snowy surface, likely in parking lot or similar outdoor area. The vans are parked close to a building with warm [...]'
Ours
DMT
VMC
MD
MC
Figure N: Camera Motion Transfer. The reference video depicts soccer match with camera moving backward and then to the right. Our approach is the only method that faithfully captures this motion, while adhering the text prompt, notably while filtering unnecessary noise from the individual soccer players. Other methods either fail to replicate the motion accurately or do not align with the prompt.
41


Reviews:
[{"review_id": "Fi3aBgdAky", "reviewer": "Reviewer_hxyP", "summary": "This paper presents DisMo, a novel framework for learning abstract motion representations disentangled from appearance, pose, and category information. Unlike prior approaches that encode motion implicitly or rely on structure-specific representations, DisMo learns motion purely through an image reconstruction loss from augmented video clips using a flow-matching-based training objective. The resulting motion embeddings are invariant to static content and can be used to condition any video generator via lightweight adapters (e.g., LoRAdapter in LTX Video) for open-world motion transfer. DisMo is evaluated extensively across three main tasks, including Motion Transfer, Appearance Disentanglement and Zero-shot Action Classification.", "questions": "- Have you tested DisMo with other high-quality video generators (e.g., Wan2.1, HunyuanVideo)? If not, do you anticipate any compatibility or performance limitations when applying your motion embeddings to those models?\n- Why did you choose flow matching over other generative training paradigms like diffusion, contrastive learning, or masked prediction?\n- How does the performance of your motion extractor compare to representations learned by models such as V-JEPA or VideoMAEv2, when used in the same transfer setup?\n- What specific impact do the augmentations during training have on the quality of motion transfer results?\n- Given the perceptual nature of motion transfer quality, have you considered running a user study or human evaluation to complement the CLIP-based metrics?\n- Can DisMo handle composite motion scenarios involving both object motion and camera motion simultaneously?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## **Strengths**\n\n- **Clear and Novel Formulation**: DisMo introduces a simple and effective method for learning motion representations by leveraging standard image-space reconstruction losses, augmented with an information bottleneck and strong data augmentations to force motion extraction.\n- **Disentanglement and Generalization**: The model robustly disentangles motion from identity and visual content, enabling effective **cross-category** and **viewpoint-invariant** transfer. This is validated both qualitatively and quantitatively.\n- **Integration with Video Generators**: The framework is modular and scalable. The learned motion embeddings can be seamlessly integrated into pretrained video generation models (e.g., LTX Video) using lightweight LoRA adapters, eliminating the need to retrain the entire generative backbone.\n- **Strong Experimental Results**: DisMo outperforms state-of-the-art baselines in both generative and discriminative tasks—achieving best scores in motion fidelity and text adherence, while also outperforming strong backbones like V-JEPA on zero-shot action classification with only 17M training samples.\n- **Various Qualitative Analysis**: The paper provides a wide range of visual comparisons, illustrating robustness across scenarios such as camera motion transfer, same- and cross-category transfer, and structure-independent motion abstraction.\n\n## **Weaknesses**\n\n- **No Evaluation on High-Fidelity Generation Tasks Beyond LTX**: While motion quality is demonstrated using LTX Video, there is limited exploration of how DisMo integrates with other high-fidelity video generation models (e.g., Wan2.1, HunyuanVideo). Without such experiments, it is difficult to assess the general applicability of the proposed motion representations across different generation frameworks.\n- **Limited Analysis of Flow-Matching Design Choice**: The paper adopts flow matching as the core training objective for motion representation learning, but does not provide comparisons or ablations against alternatives like diffusion-based or contrastive objectives. It remains unclear whether the reported performance gains are due to the task formulation or specific to the flow-matching mechanism.\n- **Missing Comparisons with Other Motion Extractors**: Although DisMo’s own extractor is proposed as a key contribution, the paper does not compare its performance with existing video representation models (e.g., V-JEPA, VideoMAEv2) when used within the same motion transfer pipeline. This would clarify the advantage of the proposed extractor over well-established baselines.\n- **Absence of Human Study**: The evaluation relies entirely on quantitative metrics (e.g., CLIP-based similarity) and visual examples. However, given the inherently perceptual nature of motion realism and alignment, a user study would provide valuable insight into subjective quality and preference.\n- **No Experiments on Composite Motion Scenarios**: The paper does not explore how DisMo handles complex scenarios involving multiple motion types simultaneously—such as combining camera motion with object or human motion. This limits the understanding of its effectiveness in realistic, unconstrained settings.\n- **Lack of Analysis on the Role of Augmentations**: The paper claims that strong augmentations during training help enforce disentanglement by forcing the extractor to focus on temporal dynamics rather than appearance. However, there is no dedicated analysis or qualitative example demonstrating how this affects motion transfer performance. A targeted study could help isolate the impact of augmentations on representation quality and transfer fidelity.\n- **Ambiguous Reference in Line 252**: The sentence “Qualitative examples in Section 4.1 further illustrate this” refers to visual results, but it is unclear whether this points to Figure 4.", "was_revised": true, "final_justification": "The rebuttal adequately addresses most of my concerns. I will therefore maintain my positive score.", "rebuttal": "Thank you for your detailed and insightful review. We appreciate your recognition of the novelty, formulation, and experimental results of our work, as well as your constructive suggestions for improvement.\n\n### Other Video Generation Backbones\nTo the reviewer's request, we additionally test DisMo with other high-quality video generation backbones, namely **SparseCtrl** and **CogVideoX**. As shown in the table below, using a more capable generator like CogVideoX leads to improved motion fidelity and temporal consistency, without any modification to the motion embeddings produced by DisMo. This demonstrates that DisMo is generally applicable to off-the-shelf video models withouth being constrained to a specific one. We further assessed the quality and realism of the generated videos using FID and FVD scores, observing consistent improvements with stronger video backbones. This demonstrates that DisMo benefits from more powerful architectures and remains compatible with future models. For this evaluation, we generate 2,000 videos per model, each 25 frames long, yielding 50,000 frames in total for FID computation. Motion is transferred from randomly sampled videos and prompts from the Koala-36M [4] dataset.\n\n| Model      | Temporal Consistency↑ | Motion Fidelity↑ | FID↓     | FVD↓     |\n| ---------- | --------------------- | ---------------- | --------- | --------- |\n| SparseCtrl [5] | 0.95                  | 0.73             | 116.8     | 327.9     |\n| LTX [6]        | 0.95                  | 0.75             | 88.54     | 298.4     |\n| **CogVideoX [7]**  | **0.97**              | **0.78**         | **62.99** | **206.3** |\n\n\n### Other Training Paradigms\nWe chose flow matching because it has been shown to be as effective a generative paradigm as diffusion for images [1] and offers a significantly simpler training process, as it does not require careful tuning of a noise schedule for example.\nWe also performed an ablation study in which we replaced flow matching with a discriminative masked prediction objective, followed by CogVideoX fine-tuning. As shown in the table below, flow matching leads to a notable gain in motion fidelity, while keeping other metrics comparable:\n\n\n| Model       | Prompt Adherence↑ | Temporal Consistency↑ | Driving Video Similarity↓ | Motion Fidelity↑ |\n| ----------- | ---------------- | -------------------- | ------------------------ | --------------- |\n| DisMo (MAE) | 0.26             | 0.97                 | 0.55                     | 0.65            |\n| V-JEPA      | 0.26             | 0.97                 | 0.59                     | 0.67            |\n| DisMo       | 0.27         | 0.97             | 0.56                 | 0.78        |\n\n### Motion Transfer with other Representations\n\nIn response to the comment suggesting comparisons with other motion extractors, we train a large I2V model (CogVideoX) using a frozen V-JEPA backbone, consistent with the fine-tuning procedure described in **Sec 3.3**. However, for $M_t$, we use spatially averaged embeddings from V-JEPA. The results in the table above demonstrate, how this increases the similarity to the driving video, indicating content bleeding.\n\n### Augmentations and Motion Transfer\n\nAs shown in **Table C**, while previous frame conditioning significantly prevents appearance leaking and improves disentanglement, integrating augmentations further boost these improvements. Furthermore, we evaluate how our motion representations handle changes in appearance and geometry by applying photometric and geometric augmentations to a driving video. Using the same source prompt, we perform motion transfer with the original and both augmented videos and then compare the outputs. The tables below show that DisMo with augmentations exhibits notably greater invariance to driving video changes, particularly in response to geometric transformations.\n\n**Photometric Augmentations**\nApplying photometric augmentations yields a 2.4% relative improvement in L1 and a 1.4% relative improvement in SSIM:\n| Model           | L1↓ | SSIM↑  |\n| --------------- | ----- | ---- |\n| DisMo (w/o augmentations) | 0.082  | 0.71 |\n| **DisMo**       | **0.080**  | **0.72** |\n\n**Geometric Augmentations**\nGeometric augmentations lead to a 2.3% relative improvement in L1 and a 4.4% relative improvement in SSIM:\n| Model           | L1↓ | SSIM↑  |\n| --------------- | ----- | ---- |\n| DisMo (w/o augmentations) | 0.087  | 0.68 |\n| **DisMo**       | **0.085**  | **0.71** |\n\n\n### Human Evaluation\nWe agree that perceptual aspects such as motion transfer quality benefit from human evaluation in addition to automated metrics. To this end, we conducted a user study comparing several state-of-the-art models, including our own (DisMo), to better understand how they perform across realism, prompt adherence, and motion transfer quality. The results of the study are shown below:\n\n| Method         | Realism (%) | Prompt Matching (%) | Motion Transfer Quality (%) |\n| -------------- | ------- | ----------------| ------------------------ |\n| DMT            | 10.93   | 9.60             | 17.73                    |\n| MotionDirector | 10.98   | 7.47             | **25.96**                    |\n| VMC            | 20.04 | 26.13   | 16.98                    |\n| MotionClone    | 19.91   | 19.42            | 14.62                    |\n| *DisMo*      | **38.13** | **37.38**       | 24.71                 |\n\nWhile **MotionDirector** achieves high motion transfer quality (25.96%), users reported a notable drop in prompt adherence (7.47%), likely due to visual inconsistencies and content bleeding. Our method, **DisMo**, matches MotionDirector in motion transfer quality (24.71%) while significantly outperforming it in both realism and prompt matching.\n\n\n### Composite Motion\nWe designed a targeted experiment to evaluate the composability of DisMo’s motion embedding space.\n\nTo assess which aspects of motion are encoded in the embeddings, we use the YUP++[2] dataset, which contains 600 static videos with no camera motion. Using these, we can sample three types of video clips:\n\n- **Object Motion Only**: A randomly sampled clip from the dataset, containing natural object motion but no camera movement.\n- **Camera Motion Only**: A different clip, augmented with temporally smooth geometric transformations simulating camera motion, applied only to the initial frame (thus eliminating object motion).\n- **Combined Motion**: The sampled clip with both object motion and the same simulated camera motion applied across all frames.\n\nThis setup allows us to isolate the two motion components, camera and object, and test whether DisMo’s embeddings for the combined motion contain information from both individual motion types.\n\nWe sample 3200 of such triplets, each containing the three types of clips mentioned before, and encode each of them using DisMo’s motion encoder. We then estimate the mutual information (MI) between the embeddings of the combined-motion clip ($M_{both}$) and those of the camera-only ($M_{cam}$) and object-only ($M_{obj}$) clips, respectively. High mutual information would suggest that $M_{both}$ captures shared information from $M_{cam}$ and $M_{obj}$, which is a key indication of composability.\n\nTo quantify this, we use the Kraskov–Stögbauer–Grassberger (KSG) estimator [3] with $k$ = 5, as also used in **Sec C.3**. To compute statistical significance, we further estimate MI under 100 random permutations (i.e., computing MI between unpaired clips), yielding a null distribution with means of around 0.2 nats and standard deviations of around 0.4. The observed MI values for both ($M_{both}$, $M_{cam}$) and ($M_{both}$, $M_{obj}$) are significantly higher than this baseline, indicating a strong statistical dependency and thus supporting the hypothesis that DisMo’s latent space is compositional with regard to simultaneous camera and object motion.\n\nThe table below summarizes the mutual information results.\n\n|      MI (nats)          | Low Camera Motion  | Medium Camera Motion | High Camera Motion   |\n| ----------------- | ---- | ------ | --- |\n| Camera Motion |   1.37   |    2.25    |  4.40   |\n| Object Motion | 3.59 |  2.97  |   1.51  |\n\n\n### References\n[1] Ma, N., Goldstein, M., Albergo, M. S., Boffi, N. M., Vanden-Eijnden, E., & Xie, S. (2024, September). Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In European Conference on Computer Vision (pp. 23-40). Cham: Springer Nature Switzerland.\n[2] Feichtenhofer, C., Pinz, A., & Wildes, R. P. (2017). Temporal residual networks for dynamic scene recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (pp. 4728-4737).\n[3] Kraskov, A., Stögbauer, H., & Grassberger, P. (2004). Estimating mutual information. Physical Review E—Statistical, Nonlinear, and Soft Matter Physics, 69(6), 066138.\n[4] Qiuheng Wang et al. (2024) Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Content\n[5] Guo, Y., Yang, C., Rao, A., Agrawala, M., Lin, D., & Dai, B. (2024, September). Sparsectrl: Adding sparse controls to text-to-video diffusion models. In European Conference on Computer Vision (pp. 330-348). Cham: Springer Nature Switzerland.\n[6] Yang, Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu, J., ... & Tang, J. (2024). Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072.\n[7] HaCohen, Y., Chiprut, N., Brazowski, B., Shalem, D., Moshe, D., Richardson, E., ... & Bibi, O. (2024). Ltx-video: Realtime video latent diffusion. arXiv preprint arXiv:2501.00103."}, {"review_id": "y2vLtexnOE", "reviewer": "Reviewer_Ph2Z", "summary": "This paper proposes DisMo, a method for learning abstract, disentangled motion representations from raw videos via image-space reconstruction. The goal is to enable open-world motion transfer without relying on object correspondences. DisMo can be plugged into existing video generators and shows strong results in fidelity, prompt alignment, and generalization.", "questions": "Can the authors clarify the novelty of their approach, especially in relation to existing work in robotics and motion generation?\n\nHow do the authors address the visible artifacts in the generated videos? Is there any planned analysis or improvements for these issues?\n\nI am not fully in this field, I would like to check other reviews opinion on this submission.", "limitations": "/", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "- The writing in this paper is clear and easy to follow.\n- I appreciate the method’s flexibility and modular design — it is compatible with various pre-trained video generators. The ability to perform zero-shot transfer is particularly compelling.\n- It’s also encouraging to see strong performance on both standard and out-of-distribution benchmarks.\n\nLimitations:\n- The idea is not very new to me. Abstraction and disentanglement exist in robotics and embodied AI; the contribution feels incremental.\n- Evaluation scope limited: Would like to see more extreme or semantically challenging motion transfers to test generality.\n- The method adds non-trivial motion modeling layers, but lacks analysis of training/inference cost. How is the stability of the method?", "was_revised": true, "final_justification": "Thank you for the response. I appreciate the authors’ clarifications regarding the novelty of the approach compared to robotics pipelines, and I find the distinction around unconstrained, general-purpose motion representations to be well-articulated. The additional results with stronger video generators (e.g., CogVideoX) also address my concerns about generation artifacts. Overall, I am satisfied with the rebuttal and the points raised were adequately addressed. I will maintain my original score.", "rebuttal": "Thank you for your review and for highlighting the flexibility and clarity of our method. We appreciate your feedback on the evaluation scope, computational analysis, and novelty. We understand the concern and would like to clarify the specific contributions that distinguish our work from prior methods.\n\n### Novelty\n\nWhile there is prior work on learning inverse dynamics in robotics, our method is novel in several key aspects:\n* **Unconstrained motion space and training**: Latent dynamics spaces that the robotics community attempts to learn, typically specialize to specific domains and use-cases, such as robotic movements or video game controls. This is usually enforced by highly reduced and/or regularized embeddings with the goal of capturing individual, atomic and often discrete actions. Meanwhile, DisMo leverages an unconstrained and larger embedding space to represent generic and composable motion. This design choice is crucial for open-set video training where complex dynamics with multiple modes (e.g., multiple moving objects, objects parts, and/or camera movement) must be captured.\n* **General-purpose applicability**: While some recent robotics methods have begun exploring open-world data, many still focus on structured and constrained settings (e.g., repetitive arm movements or indoor navigation). In contrast, our approach is designed to operate on unstructured, in-the-wild video data, enabling broader generalization across diverse and unconstrained scenarios. This introduces significantly more variability in appearance, scene dynamics, and motion patterns, requiring more general and robust representations. This gives rise to general-purpose motion representations, that can be used for a variety downstream tasks, such as action classification (**Section 4.3**) or general latent space analysis (**Section D**). Most importantly, though, we demonstrate how our motion representations are well-suited for open-world motion transfer across arbitrary scenes and subjects, rather than task-specific behavior cloning or control, which is usually the primary focus in robotics.\n* **Second-stage video model finetuning**: Our method is designed to be plug-and-play, allowing motion embeddings to be combined with any video generator. We further demonstrate performance gains through second-stage finetuning of the video model, which is uncommon in robotics pipelines.\n\n### Addressing Artifacts in Generated Videos\nWe address the visible artifacts by employing a more powerful video generation model, CogVideoX, instead of LTX. This substitution significantly reduces artifacts, as evidenced by the improved FID and FVD scores reported in the table below. While we are unable to provide additional qualitative results during the rebuttal period due to NeurIPS regulations, the quantitative improvement in FID and FVD offers strong support for the effectiveness of this change. To this end, we generate 2,000 videos per model, each consisting of 25 frames, resulting in a total of 50,000 frames for FID evaluation. The videos are generated by transferring motion from randomly sampled videos and prompts of the Koala-36M [1] dataset.\n\n| Model      | FID↓   | FVD↓   | Temporal Consistency↑ | Motion Fidelity↑ |\n| ---------- | ----- | ----- |--------------------  | --------------- |\n| SparseCtrl [2] | 116.8 | 327.9 | 0.95                 | 0.73            |\n| LTX [3]        | 88.54 | 298.4 | 0.95                 | 0.75            |\n| **CogVideoX [4]**  | **62.99** | **206.3** | **0.97**                 | **0.78**            |\n\n\n[1] Qiuheng Wang et al. (2024) Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Content\n[2] Guo, Y., Yang, C., Rao, A., Agrawala, M., Lin, D., & Dai, B. (2024, September). Sparsectrl: Adding sparse controls to text-to-video diffusion models. In European Conference on Computer Vision (pp. 330-348). Cham: Springer Nature Switzerland.\n[3] Yang, Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu, J., ... & Tang, J. (2024). Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072.\n[4] HaCohen, Y., Chiprut, N., Brazowski, B., Shalem, D., Moshe, D., Richardson, E., ... & Bibi, O. (2024). Ltx-video: Realtime video latent diffusion. arXiv preprint arXiv:2501.00103."}, {"review_id": "UlKNKTR5mC", "reviewer": "Reviewer_XVLY", "summary": "The authors proposed DisMo that learns abstract motion representations directly from video data via an image space reconstruction objective. As their model is independent of different static information, including appearance, pose, or object identity, it has the capability for open-world motion transfer that allows motion to be transferred across semantically unrelated entities without requiring object correspondences. Unlike prior methods that show a trade-off between motion fidelity and adherence to textual prompts, their approach disentangles the motion semantics from instance-specific appearance features. They demonstrated the effectiveness on diverse motion transfer tasks, outperforming existing state-of-the-art video representation models.", "questions": "Please see the \"Strengths and Weaknesses\" section.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nS1. The idea of representation learning solely based on an image-space reconstruction objective is interesting and unique in terms of leading to an abstract motion representation space.\n\nS2. The authors performed a diverse set of experiments focusing on cross-category and cross-viewpoint settings. The downstream classification task also shows the efficiency of the learned motion embeddings.\n\nS3. One of the primary strengths of this paper is that the core idea of learning motion representations that are invariant to static content is well described and motivated.\n\nS4. The experiments are comprehensive with clear implementation details, ablation studies, and latent space analysis.\n\nS5. The paper is well-written, easy to read and follow. \n\n**Weaknesses**\n\nW1. I found that the current methodological development lacks detail about how the \"invariance\" is actually enforced or how it is being maintained with respect to the architectures that ensure this property.\n\nW2. While the authors show appearance disentanglement results in Tab. 2, they did not provide clear metrics to measure motion invariance specifically or whether motion patterns are preserved or not (or how much). Also, experiments for achieving invariance would be further strengthened by performing experiments in different scenarios such as with different lighting, backgrounds, and camera angles. Overall, I found that the current disentanglement evaluation is limited and does not directly measure motion invariance.\n\nW3. Computational costs and statistical significance tests are missing. Compared to the alternatives, what's the efficiency and complexity in terms of time and memory?\n\nI would strongly suggest the authors to justify their standings on the raised weaknesses and concerns in the rebuttal period.", "was_revised": true, "final_justification": "My questions and concerns are well addressed. I will keep my original score.", "rebuttal": "Thank you for your encouraging review and for acknowledging the strengths of our method and comprehensive analysis. We value your detailed suggestions about the invariance,  computational costs and statistical significance of our model.\n\n### Enforcing Invariance\nHere is a more detailed explanation of why augmentations help to enforce invariance in the learned motion embeddings. During training, the motion extractor is fed augmented frames from the video, while the source frame provided to the frame generator (for reconstruction) is not augmented, and the target frame that the generator is tasked to reconstruct is also not augmented. This setup encourages the frame generator to use the motion embedding $m_t$, which was derived from an augmented input frame $\\mathbf{x_t}$, to reconstruct the unaltered future frame $\\mathbf{x}_{t+\\Delta t}$. In doing so, the model is incentivized to learn motion embeddings that are robust to visual perturbations and focus solely on the underlying motion, thereby promoting invariance to augmentations.\nWe evaluate this by applying photometric and geometric augmentations to a driving video. Using the same source prompt, we perform motion transfer with the original and both augmented videos and then compare the outputs. The tables below show that DisMo with augmentations exhibits notably greater invariance to driving video changes, particularly in response to geometric transformations.\n\n**Photometric Augmentations**\n| Model           | L1↓ | SSIM↑  |\n| --------------- | ----- | ---- |\n| DisMo (w/o augmentations) | 0.082  | 0.71 |\n| **DisMo**       | **0.080**  | **0.72** |\n\n**Geometric Augmentations**\n| Model           | L1↓ | SSIM↑  |\n| --------------- | ----- | ---- |\n| DisMo (w/o augmentations) | 0.087  | 0.68 |\n| **DisMo**       | **0.085**  | **0.71** |\n\n### Invariance of the Motion Representation\n\nWe evaluate the invariance of motion representations under appearance and geometric changes. Starting with an original driving video, we generate two augmented versions: one with photometric augmentations (e.g., changes in brightness, contrast, and color) and one with geometric augmentations (e.g., rotations, cropping, and spatial shifts).\nWe then perform motion transfer using all three driving videos, the original and the two augmented versions, while keeping the same source prompt and seed. This results in three generated videos. To assess invariance, we compare the generated videos from the two augmented driving inputs against the one generated using the original driving video. Ideally, an invariant motion representation should produce minimal differences between these outputs.\nThe tables below show that our method demonstrates significantly higher consistency across these conditions compared to prior work, indicating stronger invariance to both photometric and geometric changes.\n\n**Photometric Augmentations**\n| Model          | CLIP ↑ | SSIM ↑ | L1 ↓ | LPIPS ↓ |\n| -------------- | ------ | ------ | ---- | ------- |\n| DMT            | 0.66   | 0.51   | 0.17 | 0.59    |\n| VMC            | 0.71   | 0.44   | 0.17 | 0.58    |\n| MotionDirector | 0.62   | 0.46   | 0.15 | 0.55    |\n| MotionClone    | 0.76   | 0.52   | 0.13 | 0.46    |\n| **DisMo**          | **0.90**   | **0.72**   | **0.08** | **0.25**    |\n\n**Geometric Augmentations**\n| Model          | CLIP ↑ | SSIM ↑ | L1 ↓ | LPIPS ↓ |\n| -------------- | ------ | ------ | ---- | ------- |\n| DMT            | 0.66   | 0.48   | 0.18 | 0.62    |\n| VMC            | 0.71   | 0.40   | 0.20 | 0.62    |\n| MotionDirector | 0.60   | 0.44   | 0.16 | 0.60    |\n| MotionClone    | 0.72   | 0.43   | 0.18 | 0.58    |\n| **DisMo**          | **0.90**   | **0.71**   | **0.09** | **0.26**    |\n\n\n### Computational Costs\nWe report the inference time and the number of parameters for each model during motion transfer generation in the table below. It's important to note that DMT, VMC, and MotionDirector are optimization-based methods or rely on per-sample fine-tuning, which significantly increases inference time, as they perform iterative adaptation for each video. In contrast, our method (DisMo) is feed-forward at inference time, enabling significantly faster generation. Across all configurations we report, DisMo is faster and more efficient—except when paired with CogVideoX as the video generation backbone, which increases inference time due to the size and complexity of the underlying model. However, it is important to note that using CogVideoX is not required to achieve strong motion transfer results: as evidenced by Table 1 in the main paper, DisMo combined with lighter backbones such as LTX already achieves better or comparable motion fidelity and transfer quality, while being substantially faster and more efficient as seen in table below.\n\n| Model            | Inference Time (A100) | # Params |\n| ---------------- | --------------------- | -------- |\n| VMC              | 10 min / video        | 6B       |\n| DMT              | 7.5 min / video       | 1.7B     |\n| MotionClone      | 45 sec / video        | 0.983B   |\n| MotionDirector   | ~5 min / video        | 0.983B   |\n| DisMo-CogVideo   | 2 min / video         | 6.072B   |\n| DisMo-LTX        | 30 sec / video        | 2.172B   |\n| DisMo-SparseCtrl | 10 sec / video        | 1.115B   |\n\n\nNote: The reported number of parameters is approximate and refers to the largest component of each model, to the best of our knowledge. Exact values may vary depending on implementation details and auxiliary components."}, {"review_id": "zARKcafGZ7", "reviewer": "Reviewer_Ss33", "summary": "This paper introduces DisMo, a novel approach for learning abstract motion representations directly from raw video data using an image-space reconstruction objective. Dismo disentangles motion from static content such as appearance, object identity, or pose. This allows motion to be transferred across semantically unrelated entities, even between vastly different categories, without requiring object correspondences. Dismo achieves state-of-the-art performance on several downstream tasks like like open-world motion transfer, zero-shot action classification.", "questions": "Please see the weaknesses. The method proposed in the paper is interesting and promising. My primary concerns focus on its generalizability and scaling capability. I will take into consideration the opinions of other reviewers when making my final decision.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n\n1. Dismo learns motion information directly from raw video data using only a standard image reconstruction loss by a  flow-matching-based reconstruction objective, which is interesting and concise.\n2. Experiments on motion transfer tasks and action classification tasks show the effectiveness of the motion embeddings.\n3. The motion representation can be combined with any existing video generator via lightweight adapters, allowing the approach to benefit from future advancements in video generate models.\n\n**Weaknesses:**\n\n1. Scaling. Will the model's performance be further enhanced if a more powerful motion extractor and frame generator are employed? Additionally, how will the training cost be affected?\n2. Is the experimental comparison in Table 1 fair? Since Dismo is trained using a pretrained diffusion transformer along with a light adapter, does this introduce any unfairness when compared to baselines such as VMC and DMT?\n3. In the action classification task, datasets like K600, THUMOS and AVA are more commonly used, their results should also be evaluated.", "was_revised": true, "final_justification": "The work is solid and the rebuttal addresses most of my concerns. I will maintain my positive score.", "rebuttal": "Thank you for your review and for recognizing the effectiveness and modularity of our approach. We appreciate your comments on scalability, fairness of comparisons, and further evaluations.\n\n### Scaling\nWe agree that using a more powerful motion extractor and frame generator has the potential to further improve performance. While we did not conduct a full scaling experiment due to computational and time constraints, we instead focused our computational resources on evaluating other and more powerful video models to scale the motion transfer capabilities. For this evaluation, we generate 2,000 videos per model, each 25 frames long, yielding 50,000 frames in total for FID computation. Motion is transferred from randomly sampled videos and prompts from the Koala-36M [4] dataset. Our results show, how scaling the video generation model greatly enhances the motion transfer capabilities of our model.\n\n| Model      | Temporal Consistency↑ | Motion Fidelity↑ | FID↓     | FVD↓     |\n| ---------- | --------------------- | ---------------- | --------- | --------- |\n| SparseCtrl [6] | 0.95                  | 0.73             | 116.8     | 327.9     |\n| LTX [7]        | 0.95                  | 0.75             | 88.54     | 298.4     |\n| **CogVideoX [8]**  | **0.97**              | **0.78**         | **62.99** | **206.3** |\n\nWe consider scaling the motion extractor and frame generator a valuable direction for future work. Incorporating larger architectures would likely lead to increased training cost, particularly in terms of memory and training time, due to their higher parameter count and complexity. Nevertheless, our method is modular by design and can readily benefit from such improvements.\n\n\n### Fairness of Comparison\n\n\nWe appreciate the reviewer’s concern regarding the fairness of the comparison in Table 1. In fact, we believe the comparison is conservative with respect to DisMo. While DisMo combines a pretrained diffusion transformer with a lightweight motion adapter, its overall model size and the amount of training data used are substantially smaller than those of several baselines.\nFor example, VMC utilizes a 6B parameter text-to-video model trained on WebVid-10M [1], which contains 10 million video-text pairs—an order of magnitude larger than the 2.1M video samples used to train DisMo-LTX, which itself has only 2.17B parameters. Moreover, VMC and DMT both rely on image encoders pretrained on LAION [2], gaining significant advantages from large-scale image-text pretraining. DMT also builds upon Zeroscope [3], a model trained on both WebVid-10M and LAION. Similarly, MotionClone and MotionDirector are based on Stable Diffusion, pretrained on hundreds of millions of image-text pairs.\nAll compared methods rely on heavy pretraining and modular adaptation. Yet, DisMo achieves competitive—and often superior—results across benchmarks with significantly fewer parameters and less pretraining data, showcasing the efficiency and robustness of our approach.\n\n\n### Action Classification on other Datasets\n\nWhile AVA and Kinetics-600 are widely used benchmarks for action classification, we were not able include it due to time and/or computational constraints (Kinetics-600 has over 500k videos). Moreover, as demonstrated in the V-JEPA [5] paper, state-of-the-art performance on Kinetics-400 (the preceding version of Kinetics-600) can be achieved using static visual features (e.g. DINOv2) alone, suggesting that the dataset can be largely solved without modeling motion. For this reason, we focused on datasets where motion understanding plays a more central role. \n\nTo the reviewer's request, in the table below we include action classification results on THUMOS13 [4], a benchmark consisting of 24 classes and 3,207 videos. However, we found that this benchmark is mostly solvable by static appearance-based cues alone, as is evident, when probing the dataset using a solely frame-based model (DINOv2). While our method (DisMo) does not achieve the best performance on this dataset, it remains competitive despite its limited training data. Combined with our strong results on other benchmarks presented in the main paper, this supports the effectiveness of our approach for learning meaningful *motion* representations.\n\n| Model     | THUMOS13 |\n| ---------- | -------- |\n| VideoMAE   | 0.57    |\n| VideoMAEv2 | 0.97    |\n| V-JEPA     | 0.88    |\n| DINOv2-B     | 0.92    |\n| DisMo      | 0.80    |\n\n\n### References\n[1] Max Bain, Arsha Nagrani, G¨ ul Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1728–1738, 2021. 5, 8.\n[2] Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., ... & Jitsev, J. (2022). Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural information processing systems, 35, 25278-25294.\n[3] Wang, J., Yuan, H., Chen, D., Zhang, Y., Wang, X., & Zhang, S. (2023). Modelscope text-to-video technical report. arXiv preprint arXiv:2308.06571.\n[4] Y.-G. Jiang, J. Liu, A. R. Zamir, I. Laptev, M. Piccardi, M. Shah, R. Sukthankar, THUMOS’13: ICCV Workshop on Action Recognition with a Large Number of Classes, http://crcv.ucf.edu/ICCV13-Action-Workshop/, 2013.\n[5] Adrien Bardes et al. (2024), Revisiting Feature Prediction for Learning Visual Representations from Video\n[6] Guo, Y., Yang, C., Rao, A., Agrawala, M., Lin, D., & Dai, B. (2024, September). Sparsectrl: Adding sparse controls to text-to-video diffusion models. In European Conference on Computer Vision (pp. 330-348). Cham: Springer Nature Switzerland.\n[7] Yang, Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu, J., ... & Tang, J. (2024). Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072.\n[8] HaCohen, Y., Chiprut, N., Brazowski, B., Shalem, D., Moshe, D., Richardson, E., ... & Bibi, O. (2024). Ltx-video: Realtime video latent diffusion. arXiv preprint arXiv:2501.00103."}]

Author rebuttal:

Meta-review: # Summary

This paper proposes DisMo, a novel approach for learning abstract motion representations from raw video data using an image-space reconstruction objective. The main idea is to disentangle motion from static content such as appearance, object identity, or pose. DisMo achieves state-of-the-art performance on several downstream tasks such as open-world motion transfer, zero-shot action classification.

# Strength
* Novel approach of disentangling motion from appearance for abstract motion representation learning.
* Strong experimental results on various data

# Weakness
* Some reviewer(s) concern about the level of novelty since disentangling of motion and appearance had been used in other field such as robotics.
* Some ablation / additional analysis needed.

# Rebuttal / Discussion
During rebuttal, the author(s) were able to address most of the concerns from the reviewers. At the end all 4 reviewers recommend to accept the paper. AC agrees with the recommendation, thus recommends to accept the paper.
Decision: Accept (spotlight)