EchoMotion: Unified Human Video and Motion Generation via Dual-Modality Diffusion Transformer
Authors: Yuxiao Yang
Hualian Sheng
Sijia Cai
Jing Lin
Jiahao Wang
Bing Deng
Junzhe Lu
Haoqian Wang
Jieping Ye
Venue: ICLR 2026 Poster (2026)
Keywords: Video Generation
Human Motion Generation;
Primary area: generative models

Abstract: Video generation models have advanced significantly, yet they still struggle to synthesize complex human movements due to the high degrees of freedom in human articulation. This limitation stems from the intrinsic constraints of pixel-only training objectives, which inherently bias models toward appearance fidelity at the expense of learning underlying kinematic principles. To address this, we introduce EchoMotion, a framework designed to model the joint distribution of appearance and human motion, thereby improving the quality of complex human action video generation. EchoMotion extends the DiT (Diffusion Transformer) framework with a dual-branch architecture that jointly processes tokens concatenated from different modalities.
Furthermore, we propose MVS-RoPE (Motion-Video Syncronized RoPE), which offers unified 3D positional encoding for both video and motion tokens. By providing a synchronized coordinate system for the dual-modal latent sequence, MVS-RoPE establishes an inductive bias that fosters temporal alignment between the two modalities. We also propose a Motion-Video Two-Stage Training Strategy. This strategy enables the model to perform both the joint generation of complex human action videos and their corresponding motion sequences, as well as versatile cross-modal conditional generation tasks. 
To facilitate the training of a model with these capabilities, we construct \textit{HuMoVe}, a large-scale dataset of approximately 80,000 high-quality, human-centric video-motion pairs.
Our findings reveal that explicitly representing human motion is complementary to appearance, significantly boosting the coherence and plausibility of human-centric video generation. Project page at: https://yuxiaoyang23.github.io/EchoMotion-webpage/.

Published as a conference paper at ICLR 2026
ECHOMOTION: UNIFIED HUMAN VIDEO AND MOTION GENERATION VIA DUAL-MODALITY
DIFFUSION TRANSFORMER
Yuxiao Yang 1.2 Hualian Sheng² Sijia Cai Jing Lin
Jiahao Wang'
Bing Deng"
Junzhe Lu- Haoqian Wang Jieping Ye2.
'Tsinghua University
'Alibaba Group
NNayyagg Technology University 'Xi'an an Jiaotong University
ABSTRACT
Video generation models have advanced significantly, yet they still struggle to synthesize complex human movements due to the high degrees freedom hu- man articulation. This limitation stems from the intrinsic constraints of pixel- only training objectives, which inherently bias models toward appearance fidelity at the expense of learning underlying kinematic principles. To address this, we introduce EchoMotion, a framework designed to model the joint distribution appearance and human motion, thereby improving the quality of complex hu- man action video generation. EchoMotion extends the DiT (Diffusion Trans- former) framework with dual-branch architecture that jointly processes tokens concatenated from different modalities. Furthermore, we propose MVS-RoPE (Motion Video Syncronized RoPE), which offers unified 3D positional encoding for both video and motion tokens. By providing a synchronized coordinate sys- tem for the dual-modal latent sequence, MVS- establishes inductive bias that fosters temporal alignment between the two modalities. We also propose a Motion-Videe Two-Stage Training Strategy. This strategy enables the model perform both the joint generation of complex human action videos and their cor- responding motion sequences, as well as versatile cross-modal conditional gen- eration tasks. To facilitate the training of a model with these capabilities, we construct HuMoVe, a large-scale dataset of approximately 80,000 high-quality, human-centric video-motion pairs. Our findings reveal that explicitly represent- ing human motion is complementary to appearance, significantly boosting the coherence and plausibility of human-centric video generation. Project page at: https: :/yos:///uxtt aoyang23.
woo.anng3.giiiee. git..bittbbbb.......... io a/o/chooottttooooo ion -erettiionwwwbbbbee wwwbnngebp/////
INTRODUCTION
Recently, video generation models have witnessed remarkable progress, driven in particular by the
rapid evolution of diffusion models (Ho et al. [2200 Song et al. 2020 Peebles & Xie 2023
Rom-
bach et al. 2022) and VLM caption models (I ju
2023a Wang al. [024b] Team al. 2024
Existing video generation models (Hong et al. 2022] Wan et al. 2025 Kong et al. 2024 Lin et al.,
2024 have achieved promising results in terms visual fidelity and temporal consistency. Nonethe- less, even state-of-the-an generators still face significant challenges in synthesizing complex human motions. The resulting videos often suffer from severe anatomical artifacts and unnatural joint artic- ulations, exemplified in aa.. This deficiency primarily arises from the inherent limitations of pixel-only-regression training objectives, which tend to prioritize visual fidelity over the under- lying kinematic principles governing human articulation. Concretely, the pixel-level reconstruction loss commonly used in diffusion models dominated by static appearance and background details, not temporal kinetics. This deficiency is particularly pronounced for human subjects, whose high degrees of freedom make even subtle kinematic errors glaringly unnatural.
Project leader. Corresponding author.

---

Published as a conference paper at ICLR 2026
Bave moodet
EchoMooon
Motoasto- Padeo (h) Cencitional Generntion via Croo-Modiil Completion
Podoo-to-Motion
(+) Frtvooodd Flummn Amsericaa Geverstion
Figure Overview of EchoMotion capabilities: (a) Improving anatomical integrity in human- centric video synthesis and (b) enabling bidirectional control between video and motion. By pro- cessing visual and motion sequences within a unified dual-branch Diffusion Transformer, the model learns joint distribution of human appearance and kinematics.
Prior research (Chefer et al. 2025 Huang et al. 2024) has highlighted that human video synthesis problems persist even with basic motion types that are well-represented in training data. This find- ing echoes the insight that the lack of anatomical plausibility is not merely a matter of data scale, but rather points to inherent difficulties in modeling fine-grained kinematic dynamics. To address this issue, some existing works focus on conditioning generation on explicit structural guidance, employing techniques like 2D keypoints priors(Jang al. 2025 Ma et al.) 2024b) or 3D poses pri- ors(Zhou et (0244, BBuchheim et al., 2025). While this approach offers direct control, it introduces two significant limitations. First, it creates a dependency on control signals that are often unavail- able in real-world applications. Second, even when 3D priors like human poses are used, they are typically projected into the 2D image plane to align with the video frames. This projection process inevitably discards crucial 3D geometric information, leading to diminished understanding of the underlying body structure and potential motion inconsistencies.
Motivated by these findings, propose EchoMotion, innovative framework that natively models the joint distribution of video and human motion modality. It features a dual-modality diffusion transformer, which adopts dual-branch architecture to uniformly process tokens from different modalities. Unlike the previous MMDiT architecture (Esser et al. (004)), which solely focuses on denoising video inputs, EchoMotion takes further step by explicitly denoising parametric motion jointly. Benefiting from this explicit motion modeling, our method can significantly reduce motion artifacts and facilitate the generation of complex human motion videos.
In contrast to common approaches that condition on motion videos (Zhou et al. 2024 Hu 2024), our parameterized motion representation not only more token-effiiint but also preserves na- tive 3D structural information, which is more conducive to the model learning kinematic patterns. Specifically, visual and motion tokens are concatenated along the sequence dimension and processed by our proposed Motion- Video Synchronized RoPE (MVS-RoPE) mechanism. This mechanism ap- plies token-wise position embeddings to both modalities, enforcing precise temporal correspondence while preventing spatial positional collisions between the video and motion streams. To ensure ef- ficient model convergence and fully exploit the multi-task capabilities of our model, we design a Motion- Two-Stage Training Strategy. This strategy employs two-stage training recipe: initial motion-only training phase followed by motion-video multi-task training phase. In the sec- ond phase, we further construct three paradigms: joint generation, motion-to-vided generation, and video-to-motion generation, to efficiently capture the cross-modal interactions. Furthermore, we introduce new high-quality, human-centric video dataset, which name HuMoVe. This dataset comprises approximately 80,000 video clips, each featuring distinct and clear human motions. For every video, we provide granular textual description that details (1) the subject's appearance and attire, (2) the background context, and (3) a precise description of the action being performed. To enable multi-modal joint training, we also extract and include the corresponding SMPI. motion pa- rameters for each video clip, offering rich structural guidance.
Our experiments demonstrate that by holistically modeling the joint distribution of video and mo- tion, EchoMotion significantly enhances the synthesis of human videos in terms of temporal coher- ence and structural integrity. Comprehensive evaluations validate its superiority over state-of-the-an art baselines, showcasing drastic reductions in motion artifacts and superior preservation of physical plausibility. Furthermore, this unified approach inherently enables versatile cross-modee control- lable generation, capability confirmed through extensive qualitative and quantitative assessments.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORKS
2.1 VIDEO DIFFUSION MODEL
Diffusion models have become the de facto standard for visual synthesis, scaling from high-fidelity
image generation (Rombach et al. 2022, Esser et al. 2024 [Peebles & Xiel [023 Peng et al. 2024
Li et al. [025]] Wang et al. 2024a 2025b) to video Blattmann et al. 2023; Lin et al. 2024 Wan
et al. 2025 Zhang al. 2025 Wang et al. 2025c; Wu et al. 2025 Wang et al. (025a) and 3D
assets Long et al.
2024 Xiang et al. 2025 Yang et al. 2025b] Liu et al. 2023b Yang et al. 2025a
Zhao et al. 2025 Wang et al. 2025e d) The core architecture, typically Diffusion Transformer
(DiT) (Pebless & Xie (023) operating in the latent space of VAE, was originally designed for static
images. By extending the 2D attention to 3D full attention (Wan et al., 2025 Kong et al. 2024) or inserting additional temporal attention layers (Blattmann et 20033) Gao et al. 2025). the diffusion
model adapted for the temporal coherent video generation. Regarding conditioning, instead of using conventional cross-attention to inject text prompts, MMDiT (Esser et al.
(024) employs a distinct approach. It processes visual and textual tokens with separate weights and then concatenates them as input to the attention mechanism. This design facilitates a bidirectional flow of information between the two modalities. To further improve the temporal coherence, VideoJAM (Chefer et al., 2025) introduced to incorporate explicit motion prior to video models through predicting optical flow in addition to appearance. Instead of focusing on dense, low-level motion like optical flow to model short-term temporal motion, our work focuses on leveraging the SMPL (Loper et al. 2023) parameter as a high-level, structured human kinematics.
2.2 CONDITIONAL HUMAN VIDEO GENERATION
Recent advancements in diffusion models have significantly improved the quality of human video generation. Building upon pre-trained diffusion models, methods like DisCo (Wang et al.] 2024c) and Follow-Your-Pon eMa 2024b) adapt the ControlNet architecture to guide generation using 2D human keypoints. Other works, such MagicAnimate (Xu et al. 2024) and Animate Anyone (Hu 2024), employ dedicated pose guidance modules to encode 2D pose sequences and inject them conditioning. Another branch of research uses rendered 3D human models for conditioning. For instance, Champ (Zhu et al.) (024) utilizes rendered SMPL (Loper et al.) (202)) models as guidance frames for video generation. Following this direction, RealisDance (Zhou al., 2024
2025) guides generation by concatenating multiple visual pose representations. such as outputs from HaMeR (Pavlakos et al. (024) and DWPose (Yang al. 2023), alongside rendered SMPL. models-along the channel dimension. Similarly, Human4DiT (Shao et al., 2024) generates free- view human videos conditioned on rendered SMPL models and camera poses. While powerful, these methods share a fundamental architectural limitation: they are strictly conditional generators. In contrast, we introduce a unified architecture that treats human motion parameters and video as coupled modalities, enabling both joint generation and cross-modal completion.
3 METHOD
This paper introduces EchoMotion, a system designed to generate videos with corresponding motion sequences from an input text prompt. We first propose a Dual-Modality Diffusion Transformer, augmented with a parametric motion representation and a Motion- Synchronized RoPE, to effectively model the intricate interaction between these two distinct modalities in Sec. 33. Then, we tailor a Multi-Modal Two-Stage Training Strategy facilitate mutual promotion and completion within this multi-modal system, as detailed in Section 3.2 Finally, to support this research and the broader community, we construct and release the HuMoVe dataset, large-scale collection of paired video, 3D human motion parameters, and text data in Section 3.3
3.1 DIFFUSION FOR JOINT VISUAL-MOTION GENERATION.
Architecture. Towards high-fidelity video generation, we select Wan Wan et al. 2025) as our backbone model, owing to its superior performance. Unlike approaches that model the video-only distribution p(z whicc can prioritize appearance fidelity at the expense of motion principles, we instead model the joint distribution of human movement and video p(x, mly). Here, y denotes the
3

---

Published as a conference paper at ICLR 2026
Video Decoder
Human Mocion Decoder
(5 (8.1.0) (82.0)
Visio Expertick
Expertoo
Cross Attetion
Attemion
(40.0%
(4.00
(82.1)
Self-Attcation
(020)
22 (8222)
MVS RoPE
1. (OL. (22 (1222)
(0,0.2) (0.1.2 (0..2))2222)))
Q
Piito Latean
Maston Lelents
His
Tempprral
Video Encoder
Human Motion Encoder
(4) Dual-Modality DIT
(b) Metion-Vidon Synchronized RoPE
Figure 2: Overview of EchoMotion. (a) The dual-modality DiT block for joint video-motion mod- cling. (b) Our MMVS-RoPH to serve as synchronized coordinate for dual-modal token sequence.
given text prompt, while and represent the video and human motion parameters, respectively. This unified modeling approach allows our model to effectively learn both visual and motion dy- namics. As illustrated Figure 2 our Dual-Modality Diffusion Transformer first processes the input video by parameterizing human motion representations, derived from SMPL pose and shape parameters (Loper et al. 2023). These motion tokens are concatenated with visual tokens form Diffusion encodes the precise position of each token within the unified multi-modal context. This ensures that during the joint self-attention process, both modal and cross-mode information is exchanged properly.
unified multi-modal context sequence, which is then fed into a series of Dual-Modale Transformer blocks. Within each block, our proposed
Parametric Human Motion Representation. Given a video of person in motion, we use the SMPL (Loper et al. 2023) model to parameterize the pose and shape of the human body. It captures articulated human body configurations through a low-dimensional set of pose and shape parameters, making it a widely adopted foundation in computer vision tasks involving human mesh recovery. For single frame, the SMPL model facilitates the extraction of human body representations. Specif- ically, it provides shape parameters B E TR. 10 that define the overall body shape, pose parameters 0 2 that capture human joint angles, global body orientation E R and the human root joint position v E R Following DART (Zhao et al. 2024). we further utilize the 3D joint position represent each human joint.
y 224x3
To construct a unified representation that integrates human motion modalities, we design multi-head projectors build a bidirectional mapping between the parameter and the latent spaces. For each frame, we categorize the motion parameters into three groups: {v, m7} for 3D position, {0, } for 6D rotation, and / for human shape. Three independent MLPs then project these parameters sets into the target transformer hidden dimension, generating 51 motion tokens per frame. Similarly, another three MLPs map the generated motion tokens back the original parameter space for reconstruction. Crucially, to better model rapidly changing motion patterns, preserve the temporal structure of these motion tokens. This contrasts with the typical time-level down- wwwn-sampling applied to visual tokens, and importantly, these refined and compact motion tokens retain crucial temporal motion information with only a minimal increase in computational overhead.
Dual-Modality Diffusson Transformer Block. This block processes and integrates multi-modh information. begins by passing the embeddings from the video and human motion modalities through modality-specife projections, implemented as distinct sets of learnable matrices. The projected features are then concatenated along the sequence dimension to form as:
Quan Kmm Vmm QQ: Qml. [K; Km,) [V:; Vnl.
(1)
where [ signifies the sequence-level concatenation operation. Thereafter, joint self-attention layer is applied to capture dependencies and correlations across both modalities. Following self- attention, the attended features are disentangled, with each modality's features subsequently pro- cessed independently through separate cross-atie cross-attention layers (to interact with text information) and FFNs. This architecture enables detailed interaction between modalities and with textual guidance.
4

---

Published as a conference paper at ICLR 2026
Decadd rausic ouape
Dscadd oike-
Vision Branch
Human Madion
Vision Branch
Iuman Motion
Vision Branch
luman Motion
Vision Branch
Human Mation
Branch
Branch
Branch
Branch
No voleo
98 RS
Nany RISSAN spod
May rde apgal
Nany upaa
Mady oppar
Dloe Aution Poolurityy
Cloa kideo
Nany SESNSN sypa
(+) Motion-Dilly Protmining
(b) Molien-Will-Viloo Triniig
(6) Motion to Video Triining
(d) Video to Mocion Training
Training Phase I
Training Phase II
Figure 3: Overview of our Motion- Two-Stage Training Strategy. In Phase 1, the model is pretrained on motion-only data. In Phase 2, we conduct multi-task training on paired motion-vides data, regarding "motion-win with- with--vdeo"", dattio-to-video"s and Wideo-to-motion three distinct tasks to be learned simultaneously.
Within self-attention layers, we propose specialized multi-modal positional embedding to inject features with precise position information. This enables the model to effectively reason about token relationships both within and across modalities, as formulated below.
Motion-Vides Synchronized RoPE. Existing MMDiT (Esser et al. 2024 Kong et al., 2024) ar- chitectures typically incorporate positional information either through inherent positional IDs from the text encoder by employing M-RoPE (Wang et al.) (024b), which extends relative positional encoding to jointly handle both text and visual modalities in a unified sequence. However, neither of these approaches directly accounts for the inherent temporal alignment between human motion and videos. Consequently, they are not directly suitable for representing the positional information of parameterized motion tokens, necessitating dedicated design accurately capture this crucial motion positional information.
As illustrated in Figure bb), our MVS-RoPE is designed to handle the distinct spatio-temporal nature of our multi-modal context sequence, which is composed of both video and motion latent tokens. Spatially, we partition the coordinate space: video tokens occupy base (h, w) region, while motion tokens are treated as a "diagonal extension" by offsetting their spatial index. This ensures modality distinction. Temporally, recognizing the direct temporal correspondence between motion tokens and visual tokens (4x temporal compression from video VAE), we assign scaled indexing scheme. While video tokens are indexed by time t, the corresponding coarse-grained motion tokens are assigned a scaled index of t/4. This directly embeds their temporal synchrony into the positional encoding. The processed feature after MVS-RoPE is given by:
fr. = MVS- t, h, w) = R(t, h, w) fi, fm MVS MVS-RoPE(E...
(2)
( W + fl:
(3)
Here, is the temporal index, is the motion token index, and (h, w) represents the spatial indices for visual tokens. H and W denote the spatial range of the visual tokens. The function R(-) represents the RoPE encoding, which applies rotation based on the input temporal and spatial indices.
This design ensures that 1) Preserves pretrained knowledge: video tokens receive the exact same 3D RoPE as used during pre-training, ensuring that valuable learned representations are not dis- turbed; 2) Enforces correct temporal alignment: the 1/4 scaling explicitly encodes the multi-rate relationship, resolving ambiguity and ensuring video and motion are perfectly synchronized in time. and 3) Guarantees modality distinguishability: the spatial diagonal extension prevents "positional collisions", allowing the model to easily differentiate between video and human motion tokens.
3.2 MOTION- VIDRO TWO-STAGE TRAINING STRATEGY.
Due to the divergent feature representations of visual and motion tokens, a phased training approach is necessary for the model to effectively process and align these two modalities. To achieve this alignment and ensure stable learning, we propose two-stage strategy initialized from pretrained DiT parameters:
Phase 1: Motion-only Pretraining. The motion branch is trained independently using motion-only datasets, while the video branch is frozen and deactivated (inputs omitted). This stage focuses on generating motion sequences.
5

---

Published as a conference paper at ICLR 2026
Lift Right Swimning eakdancing Tranpoling in sour
Vajer
Right hand
Wor kau ESt Estling
Golcal ouan
ing
Createce
maskkeoblll sthletice Cc Equestrian
Roller Skating
(n) The Distribation of Motion Categorics
(3) Word Cloud Caption
(c) Darnect Visuaization
Figure 4: Overview of our HuMoVe dataset. (a) Voronoi treemap of the dataset's composition. (b) Word cloud of the text captions. (c) Sample frames paired with their 3D mesh reconstructions.
Motion-video Multi-task Training. Subsequently, the model is trained on motion-video paired datasets with both branches unfrozen and active, enabling the gen- eration of both visual and motion sequences.
Phase 2:
The rationale for motion-only pretraining is allow the motion branch first converge on its own domain, preventing the dominant computationally expensive of video branch. Once stable, the architecture is readily extensible both joint generation of video and motion, as well as bi- directional cross-modal control. Therefore, in phase 2, we train the model using motion-video paired datasets with focus on the three complementary interaction paradigms.
As illustrated in Figure each paradigm is randomly sampled: 1) Joint training: generate both video and motion sequences concurrently. the conditioning input for video generation. 3) Video-to-motion training: video sequences are used to condition motion generation. When one modality serves as the conditioning input, its features are preserved and not subjected noise injection during the forward diffusion process. Additionally, lightweight MLP projects the task embedding to the latent space. This task hint then added to the latents to guide conditional token prediction. Under this framework paradigm, the model naturally achieves both pure text-guided motion-video generation and cross-modal conditional generation during inference.
Motion-to-videe training: motion sequences serve as
In-Context Classifier-Free Guidance (ICCFG), Our ICCFG implementation employs distinct con- ditioning strategies tailored During Phase 2, we apply a paradigm-specific conditional dropping strategy. For joint generation, the text condition is randomly dropped. For motion-to-vides generation, both text and motion con- ditions are randomly dropped. For video-to-motion generation, the text condition is always dropped, while the video condition dropped randomly. Accordingly, to leverage this capability during in- ference, we define uo(+) as the diffusion model. For the Joint Generation mode:
three training paradigms in Phase 2, differing from standard CFG.
or, o un(x,, me; 0) + wi (uo(Ie, me; y) un (3t, my, 0)),
where of and of represent the video and motion predictions timestep I, respectively. 0 signifies the absence of specific condition and wi is the guidance scale for text condition. For the Motion- to- Video Generation mode, the output can be expressed as:
o uo (are, 0, 0) + wi (u10 me y/ un (xt, m,, 0)) + (a my 0) - (x,, 0,0)) (5)
where we the guidance scale for motion condition. For the Video-to-Motion Generation mode: (Z,, 0 up m,, 0) + w2(ug (xz, m4, 0) - us (0, m, 0)).
(6)
This allows the model to leverage the rich context provided by various modalities while selectively enabling specific generation capabilities.
3.3 HuMoVe DATASET
A significant gap exists in current open-source resources for joint video-motion generation. Datasets from prior work (Mahmood et al. 2019 Lin et al. 2023 Fan et al.) 2025) are often unsuitable, typ- ically because they codality-speeiffe (lacking visual context) or consist of low-quality videos with redundant backgrounds and human characters. To address this limitation, we construct Hu- Mo Ve: large-scale, high-quality dataset featuring diverse scenes and human characters. Each video

---

Published as a conference paper at ICLR 2026
Table 1: Comparison with baseline models on video generation. We report both human evaluation and automatic metrics. Bold indicates the best performance, and underline indicates the second best.
Auto Metrics
Human Eval
Human Anatomy
Motion Smoothness
Dynamic
Aesthetic Quality
Vidco Quality
Prompt
Posture Plausibility
Degree
Following
CogVideoX-222
61.7
97.0
49.4
51.6
55.3
52.1
53.6
Wan-1.3B
78.1
98.2
60.6
60.1
68.2
70.3
64.0
Video Tuning(Wan- (ann-..3B)
77.4
98.3
61.6
59.7
69.3
73.2
63.5
EchoMotion (aantoonnaa--1-BB
79.6
98.9
61.9
60.0
71.3
73.2
66.1
Cog VogVicoXX1..55
65.3
98.5
54.4
53.2
62.5
60.4
59.4
Wan-5B
83.0
98.9
62.2
58.3
72.8
78.9
68.9
Video Tuning(Wann Wan-5B)
83.1
98.7
63.1
57.9
72.3
79.6
70.2
EctoMotion( Wan-5B)
85.1
99.3
64.0
58.3
$1.0
81.5
81.6
"The video shows a skateboarder quickly sliding down statr railing macnntonnnng balance dhroughout, with hold and smooth novements.
'I gymmast is streching the mot before triining lilling her logr over her hend and showing off her inc redelbblle redibbil
space standing position, with handy clorped from quickly squats compuiet squat Immediutely liftr leg
fomale performing dynamic comhuung squats
comples mreiching
Wan 2.2-5B
EchoMotion| Wan-5BB
Figure 5: Qualitative comparison with the 5B baseline. Our model (EchoMotion, right) generates anatomically correct and semantically coherent human motions, resolving the severe artifacts and compositional failures present in the baseline (left).
in HuMoVe is paired with descriptive textual caption and its corresponding 3D SMPL motion pa- rameters, forming a tightly aligned, multi-modal corpus for advanced generative modeling.
We implement comprehensive data processing pipeline that involves automated collection from various sources, followed by rigorous filtering for aesthetic quality and subject focus, and culmi- nates in precise 3D human mesh recovery and annotation, resulting in a final dataset of approxi- mately 80,000 entries. A detailed breakdown of our pipeline is provided in Appendix AA. The resulting HuMoVe dataset exceptionally diverse. As visualized by the Voronoi treemap in Figure Ia), spans 9 major categories (represented by distinct color groups) and 38 subcategories (rep- resented by shades within each group), ranging from subtle, everyday actions to dynamic, complex performances. A full categorical breakdown is available in the Appendix A.4 This careful curation provides a clean, comprehensive, and challenging foundation for developing kinematically-awwe aware generative models.
EXPERIMENTS
4.1
IMPLEMENTATION DETAILS
We perform experiments on two variants of the open-sourced base model, Wan2.1-1.3B and Wan2.2- 5B, to validate the effectiveness of method. The initial motion-only training phase leverages composite motion dataset of our proposed HuMoVe dataset alongside the extensive HumanML33 dataset (Guo al. 2022). public repository containing 14,616 human motions in SMPL.

---

Published as a conference paper at ICLR 2026
sing roeky sachiur gow, mooking bush waring lna cocdlartd
was tapuus with side maros olgas love dustiig right keas)
Prevenny the conenets
spaien de
The ragryhive caweffatserognneg Aew ensliat-
oabed pardurgann i Ae rane dape and madkeny excatd tepoouth amall sanaant...
Figure Tixt-to-video results from EchoMotion, demonstrating both strong prompt alignment and high kinematic plausibility across a diverse range of human-centric scenarios. Please refer Appendix and the supplementary material for additional examples and video results.
Figure 7: EchoMotion jointly generates an SMPL motion sequence (left) and video (right), demon- strating learned joint distribution.
format, and is conducted for 15k steps. Following this, the motion-video multi task training phase is performed our collected motion-video paired data for additional 12k steps. These step counts were determined empirically by monitoring both loss convergence and qualitative quality on a held- validation The Wan2.2-5B-EchoMootie whole training procedure is conducted on 32 NVIDIA A100 GPUs for about 4 days.
model trained for 4000 A100 GPU hours. The
4.2 TEXT VIDEO GENERATION
Evaluation Protocol. To enable comprehensive evaluation, build new benchmark that covers a wide spectrum of human motion, from daily activities to extreme athletic feats. Our benchmark includes diverse of prompts (30 per category) covering: precise, high-momentum movements from gymnastics and athletics: fluid, expressive motions from dance; reactive and interactive scenar- ios from ball and combat sports; and the natural gestures everyday life. For quantitative analysis, we use the automatic metrics from VBench (Huang et al. 2024) and VBench-2.0 (Zheng et al.) 2025), alongside human user studies to collect numerical ratings. These quantitative findings are complemented by qualitative visualizations that illustrate the performance of our method.
Quantitative Results. We evaluate our method against the video-only baselines at both 1.3B and 5B scales with baseline open-source models. The results, summarized in Table [ demonstrate that our joint modeling approach yields substantial gains in motion fidelity. On automatic met- rics, EchoMotion markedly improves scores for Motion Smoothness and Anatomical Consistency over the baselines. Notably, this improvement achieved without any reduction in the Aesthetic Quality score. These results strongly corroborated by our human evaluation, where participants
8

---

Published as a conference paper at ICLR 2026
pupte domeee das dermier qurc
peree vall is dee he Aerooee yous, Suline
Saare Taloo
Soure Matia
Bicoand Natin
Greeraee Yide
Generate Human Vidzo Given Conditional Motion Text
Recovery Human Motion Given Video
Figure 9: Cross-modal completion by EchoMotion. (a) Motion-to-Viie Video synthesis from motion and
text. (b) Video-to-Motion recovery (inverse kinematics).
assign EchoMotion significantly higher ratings for Video Quality, Motion Plausibility, and Prompt- Following for human-centric prompts.
Qualitative Results. Figure [ highlights the
Video
Humanmotion
Video
Human motion
critical limitations of Video-only training complex, human-centric prompts. While the Wan2.2-5B baseline generates reasonable thetics, it consistently violates kinematic con- straints, producing severe anatomical artifacts (e.g., the tangled gymnast, the distorted skate- boarder). Furthermore, the baseline struggles with semantic compositionality. failing to exe- cute the multi-step workout sequence. In con- trast, by jointly modeling video and human mo- tion, our model generates subjects with plausi- ble anatomy and successfully follows compositional instructions. Our model generates coherent and physically plausible motions across all challenging cases, demonstrating that joint distribution modeling leads to a superior internal representation of human kinetics.
Video
Video
Human motion
Human mitiinn
distribution on temporal level w/MVS-RoPE
(b) temporal level
wottiee-eevllsynec
Figure 8: Effect of MVS-RoPE.
In addition to outperforming baselines, our method demonstrates remarkable versatility and a sophis- ticated capacity for synthesizing a wide variety of high-quality, kinematically plausible actions- from a golf swing to a snowboarding aerial-across diverse scenes. Moreover, the efficacy of our joint modeling approach directly illustrated in Figure 7 where the model simultaneously generates a video and its corre- sponding, temporally-alignes
generation. Figure showcaess its generative breadth, successfully
SMPL sequence from one prompt. This co-generation demonstrates that the model's output is not simply pixel-level synthesis but is fundamentally conditioned on an internal representation human kinematics.
4.3 CROSS-MODAL COMPLETION
Our model's unified design enables powerful cross-modal capabilities, shown in Figure 9 By forming tasks as modal completion, EchoMotion operates bi-directionally: (a) it can synthesize high-fidelity video that precisely follows a given motion sequence (motion-to-video), and (b) it can recover the underlying SMPI. motion from an input video (video-to-motion), This flexibility to perform both generation and inverse kinematics with a single model highlights the significant advantage of our joint modeling approach. We provide further quantitative evaluations for cross- modal completion tasks in Appendix A.7.2 and A.7.3
4.4 ABLATION STUDIES
In this section, we conduct ablation studies to validate the key components of our framework. Ad- ditional ablations detailed in Appendix A.8

---

Published as a conference paper at ICLR 2026
Joint Modeling vs. Video-Only Modeling. We compare our joint training approach against baseline fine-tuned exclusively on the video data from our dataset (Video Tuning). As shown in Table while the Video-only tuning approach offers only marginal improvements and fails to hance key kinematic metrics, our EchoMotion model demonstrates substantial gains in both Human Anatomy and, notably, Posture Plausibilliy. This result confirms our central hypothesis: The key to high-quality human motion synthesis lies in the joint modeling of appearance and kinematics during training, whereas simply adding more human-centric video data offers marginal benefits.
MVS- RoPP Design. To validate our MVS-RoPE design for aligning video and motion modalities, we visualize the self-attention score in Figure Crucially, the motion temporal sequence is four times the length of the video temporal sequence, requiring the model to learn non-trivial 4:1 temporal mapping. In our model with MVS-RoPE (a), the attention map reflects this perfectly. The video-to-motion attention forms a clear, shallow diagonal, while the Motion-to-Vitee forms corresponding steep diagonal. This asymmetrical structure is direct evidence that the model has learned the correct temporal alignment. In contrast, the baseline without MVS-RoPH (b) fails completely; the attention is scattered, and the required diagonal structures are absent, demonstrating its inability to synchronize the two modalities.
attention
CONCLUSION
We present EchoMotion, an innovative framework for generating human videos by jointly modeling appearance and kinematics within Dual-Modality DiT. Our approach utilizes a SMPL-based para- metric representation for human motion, concatenating its tokens with video tokens into a unified multi-modal sequence. To enable effective information exchange within this sequence, we intro- duce MVS-RoPE, a novel positional embedding that enforces temporal alignment between video and motion modalities during multi-model joint self-attention. Furthermore, a Motion-Vides Two- Stage Training Strategy endows the model with versatile, bi-directional capabilities: controllable motion-to-vide and by our introduction of HuMoVe, a new large-scale dataset comprising approximately 80,000 high- quality video-motion pairs.
generation. The development of this framework is supported
Limitations. Our framework is currently limited to single-person generation. Extending it to multi- person scenarios, while architecturally feasible by concatenating SMPL tokens, would require creat- ing a new, large-scale dataset with per-person annotations. As this represents significant resource commitment, prioritized perfecting the single-person generation model in this work and leave multi-person generation as promising direction for future projects.
ACKNOWLEDGMENT
This work is supported by the NSFC fund (62576190), in part by the Shenzhen Science and Tech- nology Project under Grant (KJZD20240003 1032100144 JCYJ2C220812111 1801001004)
REFERENCES
Gimini 2.5.
Kling 2.5.
https://kiigai..omt
Veo 3.
Stephen Batifol, Andreas Blattmann, Frederie Boesel, Saksham Consul, Cyril Diagne, Tim Dock- horn, Jack English, Zion English, Patrick Esser, Sumith Kulal, et al. Flux. kontext: Flow match- ing for in-context image generation and editing in latent space. arXiv e-prints, pp. arXiv-2506. 2025.
Andreas Blattmann, Tim Dockhom, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models large datasets. arXiv preprint arXiv:2311.15122 2023.
10

---

Published as a conference paper at ICLR 2026
Benito Buchheim, Max Reimann, and Jürgen Döllner. Controlling human shape and pose in text- to-image diffusion models via domain adaptation. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 3688-3697. IEEE, 2025.
Hila Chefer, Uriel Singer, Amit Zohar, Yuval Kirstain, Adam Polyak, Yaniv Taigman, Lior Wolf, and Shelly Sheynin. Videojam: Joint appearance-motion representations for enhanced motion generation in video models. arXiv preprint arXiv:2502.02492. 2025.
Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu. Executing your commands via motion diffusion latent space. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 18000- 18010. IEEE, 2023, doi: 10.1 1100//CCPPR777992220000222 URL htthtt::: /ww.. http///t.......... I1G9/CVPR52729 2023 01726
Gang Cheng, Xin Gao, Li Hu, Siqi Hu, Mingyang Huang, Chaonan Ji, Ju Li, Dechao Meng, Jin- wei Qi, Penchong Qiao, et al. Wan-animate: Unified character animation and replacement with holistic replication. arXiv preprint arXiv:2509. 14055, 2025.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference machine learning, 2024.
Ke Fan, Shunlin Lu, Minyue Dai, Runyi Yu, Lixing Xiao, Zhiyang Dou, Junting Dong, Lizhuang Ma, and Jingho Wang. Go to zero: Towards zero-shot motion generation with million-scale data. arXiv preprint arXiv:2507.07080
2025.
Yao Feng, Jing Lin, Sai Kumar Dwivedi, Yu Sun, Priyanka Patel, and Michael J. Black. Chat- pose: Chatting about 3d human pose. In IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pp. 2093-2103, IEEE, 2024. doi: (1009//CPPR2733.2022..000222
/tLhhtts:: wttp://ogg/....1//////../e//
00204
Yu Gao, Haoyuan Guo, Tuyen Hoang, Weilin Huang, Jiang, Fangyuan Kong, Huixia Li, Jiashi Li, Liang Li, Xiaojie Li, et al. Seedance 1.0: Exploring the boundaries of video generation models. arXiv preprint arXiv:2506.091 2025.
Shubham Goel, Georgios Pavlakos, Jathushan Rajasegaran, Angjoo Kanazawa, and Jitendra Ma- lik. Humans in 4d: Reconstructing and tracking humans with transformers. In IEEE/CVF In- ternational Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pp. 14737-14748. IEEE, 2023, doi: 10. 11110/CCC111070222..011111
UtLhttss::/ orr//
10.1109 R01.100//CCV11000000 2023. 01358
Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang. Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5152-5161, 2022.
Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary position embedding for vision transformer. In European Conference on Computer Vision, pp. 289-305. Springer, 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-683 2020.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pre- training for text-to-video generation via transformers. arXiv preprint 2.XX::2205..5866 2022.
Li Hu. Animate anyone: Consistent and controllable image-to-video synthesis for character anima- tion. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 8153-8163, 2024.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianx- ing Wu, Qingyang Jin, Nattapol Champaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 21807-21818, 2024.

---

Published as a conference paper at ICLR 2026
Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Hu- man motion as a foreign language. In Alice Oh, Tristan Naumann, Amir Glober- son, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural In- formation Processing Systems 36: Annual Conference on Neural Information Pro- cessing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http:://ppppeee 3fbf0clea( 77660 3dea Bbb 6be 78dd6f Abstract Conference html
wwps. nips.cc/papprrrr...eeeepppepppee files//aperr22233/1e
Zeyinzi Jiang, Zhen Han, Chaojie Mao, Jingfeng Zhang, Yulin Pan, and Yu Liu. Vace: All-in-one video creation and editing. arXiv preprint arXiv:2503.07598 2025.
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zere Text-to-image diffusion mod- els are zero-shot video generators. In IEEE/CVF International Conference on Computer Vi- sion, ICCV 2023, Paris, France, October 1-6, 2023, pp. 15908-1599 15918. IEEE, 2023. doi: 1100ICCCV5107002233.06666 URL https: bttsps:/doi..om 10. 1109/ 1090111CCCC ICCV5 1000. 200.222..2222222 01462
Nikos Kolotouros, Georgios Pavlakos, Michael J. Black, and Kostas Daniilidis. Learning to re- construct 3d human pose and shape via model-fitting in the loop. In 2019 IEEE/CVF In- ternational Conference on Computer Vision, ICCV 2019, Seoul, Korea (South). October 27 November 2. 2019, pp. 2252-2261. IEEE, 2019. doi: 1011009/CCV22011000224 URL
https://dof org. 10. 110. 201 00234]
Weijic Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.036033 2024.
Huaqiu Li, Yong Wang, Tongwen Huang, Hailang Huang, Haoqian Wang, and Xiangxiang Chu. Ld-rps: Zero-shot unified image restoration via latent diffusion recurrent posterior sampling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13684-13694, 2025.
Bin Lin, Yunyang Ge, Xinhua Cheng, Zongjian Li, Bin Zhu, Shaodong Wang, Xianyi He, Yang Ye, Shenghai Yuan, Liuhan Chen, et al. Open-sora plan: Open-source large video generation model. arXiv preprini arXiv:2412.0013/ 2024.
Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang. Motion-x: A large-scale 3d expressive whole-body human motion dataset. Advances in Neural Information Processing Systems, 36:25268 252800 2023.
Jing Lin. Yao Feng, Weiyang Liu, and Michael J. Black. Chathuman: Chatting about 3d humans with tools. In IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025, pp. 8150-8161. Computer
Vision Foundation / IEEE, 2025. doi: 10. (0.1099CCVPR337742222500000
URL https:
//openaccess. theov com http://contaaat//////22////////a Cttp:/itaaananataaaaaaamaaaa
Chatting_about SaDD_Humasswsaa wi www.wwwwwhwwwtto.eeooooooowwoest httpooossssssso___e111__1
satpss__c__________22 20t____a2__________2 htm]
html
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in neural information processing systems, 36:34892-34916, 2023a.
Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang- Syncdreamer: Generating multiview-consistentt images from a single-view image. arXiv preprint arXiw:2309.03455. 2023b.
Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Mare Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9970-9980, 2024.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pp. 851-866. 2023.
12

---

Published as a conference paper at ICLR 2026
Yue Ma, Yingqing He, Xiaodong Cun, Xintao Wang, Siran Chen, Xiu Li, and Qifeng Chen. Fol- low your pose: Pose-gun fose-guided text-to-video generation using pose-free free videos. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan (eds.). Thirty-Eightte AAAI Conference Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artifi- cial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intel- ligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pp. 41 4117-4125, AAAI Press, aaai.
2024a. doi: N0.1609/AAALLV385....1 28.006.
URL
V 28215
Yue Ma, Yingging He, Xiaodong Cun, Xintao Wang, Siran Chen, Xiu Li, and Qifeng Chen. Follow your pose:
Pose-guided text-to-video generation using pose-free videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 4117-4125, 2024b.
Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black. Amass: Archive of motion capture as surface shapes. In Proceedings of the IEEE/CVF interna- fional conference on computer vision, pp. 5442-5451, 2019.
Priyanka Patel and Michael J Black. Camerahmr: Aligning people with perspective. In 2025 Inter- national Conference on 3D Vision (3DV), pp. 1562-1577 IEEE, 2025.
Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing hands in 3d with transformers. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 9826-98888 2024.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp.
2023.
Long Peng, Wenbo Li, Renjing Pei, Jingjing Ren, Jiaqi Xu, Yang Wang, Yang Cao, and Zheng- Jun Zha. Towards realistic data generation for real-world super-resolution. arXiv preprint arXiv:2406.07225,, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- the IEEE/CVF confer-
resolution image synthesis with latent diffusion models. In Proceedings the ence on computer vision and pultern recognition, pp. 10684-10695, 2022.
Ruizhi Shao, Youxin Pang. Zerong Zheng, Jingxiang Sun, and Yebin Liu. Human4dit: Free-view human video generation with 4d diffusion transformer. arXiv e-prints, pp. arXiv-2405, 2024.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint Stti:220330555.00 2024.
Timo Von Marcard, Roberto Henschel, Michael J Black, Bodo Rosenhahn, and Gerard Pons-Moll. Recovering accurate 3d human pose in the wild using imus and a moving camera. In Proceedings of the European conference computer vision (ECCV), pp. 601-617, 2018.
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wet Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models. aixiv preprint arXiv:2503.20314. 2025.
Jiahao Wang, Caixia Yan, Haonan Lin, Weizhan Zhang, Mengmeng Wang. Tieliang Gong, Guang Dai, and Hao Sun. Oneactor: Consistent subject generation via cluster-conditioned guidance. NeurlPS, 2024a.
Jiahao Wang, Hualian Sheng, Sijia Cai, Weizhan Zhang, Caixia Yan, Yachuang Feng, Bing Deng, and Jieping Ye. Echoshot: Multi-shot portrait video generation. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025a
Jiahao Wang. Caixia Yan, Weizhan Zhang, Haonan Lin, Mengmeng Wang, Guang Dai, Tieliang Gong, Hao Sun, and Jingdong Wang. Spotactor: Training free layout-controlled consistent image generation. In AAAI, 2025b.
13

---

Published as a conference paper at ICLR 2026
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing vision- languaee model's 's perception of the world at any resolution. arXiv preprint 24Xv:2409.12291,,, 2024b.
Tan Wang, Linjie Li, Kevin Lin, Yuanhao Zhai, Chung-Ching Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, and Lijuan Wang. Disco: Disentangled control for realistic human dance generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 9326 9336, 2024c.
Weitao Wang, Zichen Wang, Hongdeng Shen, Yulei Lu, Xirui Fan, Suhui Wu, Jun Zhang, Haoqian Wang, and Hao Zhang. Dreamswapy: Mask-guided subject swapping for any customized video editing. arXiv preprint wiXiv:2508.14465 2025c.
Weitao Wang, Haoran Xu, Jun Meng, and Haoqian Wang. Make your move: Make your 3d contents by adapting multi-view diffusion models to external editing. arXiv preprint arXiv:2508.07700, 2025d.
Weitao Wang, Haoran Xu, Yuxiao Yang, Zhifang Liu, Jun Meng, and Haoqian Wang. Mvreward: Better aligning and evaluating multi-view diffusion models with human preferences. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 7898-7906, 2025e.
Bing Wu, Chang Zou, Changlin Li, Duojun Huang, Fang Yang, Hao Tan, Jack Peng, Jianbing Wu, Jiangfeng Xiong, Jie Jiang, et al. Hunyuanvideo 1.5 technical report. arXiv preprint arXiv:2511.18877 2025.
Jianfeng Xiang, Zelong Lv. Sicheng Xu, Yu Deng, Ruicheng Wang. Bowen Zhang. Dong Chen, Xin Tong, and Jiaolong Yang. Structured 3d latents for scalable and versatile 3d generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 21469-214888 2025.
Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Jia-Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1481-1490, 2024.
Yuxiao Yang, Peihao Li, Yuhong Zhang, Junzhe Lu, Xianglong He, Minghan Qin, Weitao Wang, and Haoqian Wang. Nova3d: Normal aligned video diffusion model for single image to 3d generation. In 2025 IEEE International Conference on Multimedia and Expo (ICME), pp. 1-6. IEEE, 2025a.
Yuxiao Yang, Xiaoxiao Long, Zhiyang Dou, Cheng Lin, Yuan Liu, Qingsong Yan, Yuexin Ma, Haoqian Wang. Zhiqiang Wu, and Wei Yin. Wonder3d++: Cross- diffusion for high- fidelity 3d generation from a single image. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025b.
Zhendong Yang, Ailing Zeng, Chun Yuan, and Yu Li. Effective whole-body pose estimation with two-stages distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4210-4220, 2023.
Chuannui Zhang, Zhengxian Wu, Guanxing La, Yansong Tang, and Ziwei Wang. imowm: Taming interactive multi- modal world model for robotic manipulation. arXiv preprint arXiv:2510.09036, 2025.
Kaifeng Zhao, Gen Li, and Siyu Tang. Dartcontrol: A diffusion-based autoregressive motion model for real-time text-driven motion control. arXiv preprint arXiv:2410.05220 2024.
Zibo Zhao Zeqiang Lai, Qingxiang Lin, Yunfei Zhao, Haolin Liu, Shuhui Yang, Yifei Feng, Mingxin Yang, Sheng Zhang, Xianghui Yang, al. Hunyuan3d 2.0: Scaling diffusion models for high resolution textured 3d assets generation. arXiv preprint arXiv:2501. 12202, 2025.
Dian Zheng, Ziqi Huang, Hongbo Liu, Kai Zou, Yinan He, Fan Zhang, Yuanhan Zhang, Jingwen He, Wei-Shi Zheng. Yu Qiao, et al. Vbench-2.0. Advancing video generation benchmark suite for intrinsic faithfulness. arXiv preprint arXiv:2503.21755. 2025.
14

---

Published as a conference paper at ICLR 2026
Jingkai Zhou, Benzhi Wang. Weihua Chen, Jingqi Bai, Dongyang Li, Aixi Zhang. Hao Xu, Mingyang Yang, and Fan Wang. Realisdance: Equip controllable character animation with real- istic hands. arXiv preprint 240::22409.06222,2000
Jingkai Zhou, Yifan Wu, Shikai Li, Min Wei, Chao Fan, Weihua Chen, Wei Jiang, and Fan Wang- Realisdance-dit: Simple yet strong baseline towards controllable character animation in the wild. arXiv preprini arXiv:2504. 14977, 2025.
Shenhao Zhu, Junming Leo Chen, Zuozhuo Dai, Zilong Dong, Yinghui Xu, Xun Cao, Yao Yao, Hao Zhu, and Siyu Zhu. Champ: Controllable and consistent human image animation with 3d parametric guidance. In European Conference on Computer Vision, pp. 145-162. Springer, 2024.
A APPENDIX
A.1
USE OF LLMS
We utilized Large Language Models (LLMs) in several capacities to assist in this research. The core intellectual contributions, including the formulation of the research problem, the design of the EchoMotion architecture, the MVS-RoPE mechanism, and the mixed multi-modal in-context learning strategy, were conceived and developed entirely by the human authors. The roles of LLMs were confined to the following assistive tasks:
Writing and Manuscript Refinement: We employed gencral-purpose LLMs, such as OpenAl's ChatGPT, for grammatical corrections, stylistic improvements, and enhancing the clarity and readability of the manuseript. The scientific narrative, structure, and all claims remain the original work of the authors.
2. Data Curation Support: To aid in the collection of our HuMoVe dataset, we used an LLM to generate a broad set of keywords and search queries related to complex human motions. This helped to systematically identify relevant video content from public-domain sources.
3. Automated Video Annotation: A significant part of the annotation process for the Hu- MoVe dataset was facilitated by Qwen-VIz -L-Narrrr (qwe). This model was tasked with generating the initial draft of the granular textual descriptions for each video clip, covering (1) the subject's appearance and attire, (2) the background context, and (3) a description of the action.
A.2 PRELIMINARY
replace the conventional U-Net backbone in diffusion models with pure transformer architecture, demonstrat- ing superior performance and scalability. A DiT operates on latent space and composed of three main stages:
Diffusion Tranformers(DiT), Diffusion Transformers (DiTs) (Peebles & Xie 2023
An input encoder that "patchifies" the noisy latent variable into a sequence of tokens.
A series of transformer blocks that process these tokens. These blocks are conditioned the diffusion timestep & and other context, such as text embeddings, via adaptive layer normalization (adaLN).
A final decoder that "unpatchifies" the output tokens back into the predicted noise map.
Flow Matching. During training, video latents n, is disturbed by random noise zo No(0, T),
according to the timestep [0, [0,1],
It tx + (1 f)
The model is trained to predict the velocity iy I1 2o by minimizing the MSE loss of the model prediction and ut,
L = Ezo.zi. l y, 0) 12,
where y is the input text description, denotes the model weights, and u(2, y. t; 0) is the prediction by the DiT model.
15

---

Published as a conference paper at ICLR 2026
Table 2: Mapping of Major Categories their corresponding Fine-grained Categories.
Major Category
Fine-grained Categories
Urban & Outdoor
Archery, Cycling, Equestrian, Highlining, Motocross,
Parkour, Rock Climbing, Skateboarding, Street Workout
Water Sports
Kayaking, Kitesurfing, Paddlcboardingg Surfing, Swimming, Wakeboarding, Windsurfing
Athletics & Fitness
Athletics, Olympics
Ball Sports
Basketball, Golf, Soccer, Tennis, Volleyball
Combat Sports
Boxing, Judo, MMA, Taekwondo, Wrestling, Wushu
Dance & Artistic
Artistic Roller Skating, Breakdancing, Dancing
Gymnastics & Acrobatic
Gymnastics, Trampoline
Ice & Snow Sports
Figure Skating, Skiing & Snowboarding
General & Others
Daily Activity, Other
Rotary Position Embedding (RoPE). Rotary Position Embedding al., 2024) encodes abso- lute positional information by applying position-ddpennentt rotations to the query and key vectors, which typically adopted by recent Transformers to inject positional information within self- attention layers. Given d-dimention vision embedding &r, with a 3D positional index (4, h, w) in the sequence, 3D RoPE is added dimension-wise,
Tt R. Rt,,,,
(9)
where k, denotes a 3D rotation matrix determinded by the given 3D positional index and a pre-defined base frequency e.
A.3 THE JOINT DISTRIBUTION OF VIDEO AND HUMAN MOTION.
Instead of modeling the video-only distribution, propose that the distribution of human move- ment video, denoted as (2),,, can be modeled by joint distribution of its video and corresponding human motion parameters. Specifically, given text prompt y, the distribution p(zly) be formu- lated as
(10) where x and m represent the video and the human motion parameters, respectively. The goal of our model f is learn the joint distribution (2(z, mly). During inference, we generate video and corresponding human motion parameters by iteratively denoising the video and human motion latents jointly.
p(2ly) = (z, m mly).
f(v)
(11)
Revisiting Eq 10 10
we could derive that
Po (a(ly)
pa(a 9)
(12)
(ar mly)
where m2) y) corresponds the video conditional distribution given the human motion. There- fore, sampling the video from the pa mn, y) is equivalent to generating a video that matches the given human motion and text prompt y, i.c., controllable video generation. Similarly, given video, we can recover the human motion parameters within the video by sampling from the motion conditional distribution:
Po = 0)
(m 8)
(13)
(= m 0)
A.4 DETAILS OF HuMoVe DATASET
Fine-grained Categories. Table Table details the two-level classification scheme used to organize our dataset. We group specific, fine-grained activities (e.g., Skateboarding, Swimming, Boxing) into
16

---

Published as a conference paper at ICLR 2026
broader, thematic major categories (e.g., Urban & Outdoor, Water Sports, Combat Sports). This taxonomy provides a structured framework for analyzing the dataset's ' diversity and for evaluating model performance across different motion types.
Raw Data Collection. To ensure our raw data encompasses wide range of human movements, our collection process began with structured, top-down approach. We first manually defined nine major motion categories, as detailed Table 2 (e.g.. Urban & Outdoor, Water Sports, and Combat Sports). These high-level categories were then provided as input to Large Language Model (LLM) (gim), which we tasked with generating diverse list of specific search keywords for each category. An example the prompt used for this task shown below. The resulting keywords were subsequently used to collect a raw data pool from diverse sources, including open-source datasets, movies, and the internet. The prompt used for this task shown in Listing
You are a data sourcing expert for computer vision research. Your task to expand high-levei categories of human motion into specific, diverse, and Fine-grained search keywords suitable for video platforms.
For each major category provided, generate a 1ist of 20-30 search terms.
Here is an example of the desired input-output format:
.IInput:**
Major Category: Combat Sports
**Desired Output Keywords:**
boxing training drills
MMA sparring session slow moticn
cinematic Taekwondo high kick
Judo threw tutorial
female wrestler practice highlights
Mushu performance competition
Now, please generate keywords for the following 1ist of major categories:
Urban Outdoor
Water Sports
Athletics & Fitness
Ball Sports
Combat Sports
Dance Artistic
Gynnastics & Acrobatics
Ice & Snow Sports
General Others
Listing 1: The prompt used expand major categories into specific search keywords via LLM.
Data Filtering, Raw video data often exhibits frequent scene translation and contains low aesthetic and resolution data, which will negatively impact the performance of the model. To mitigate these challenges, we employ scene detection model to segment the raw video into clips and leverage an aesthetic scoring model filter out video clips with low aesthetic quality. As this work primarily focuses on the movement of the main characters in the foreground, we employed bounding box de- tection model to identify the individuals in the video, and filtered out those containing multiple main characters by comparing the number and area of the bounding boxes. Moreover, the DW OWPose(Yan Yang al. 2023) employed detect the number of visible key points within the video. Those data with relatively low number of visible key points filtered avoid artifacts and jitter during the 3D human pose detection stage.
Human Mesh Recovery and Post Process. Given the track bounding boxes input, to obtain frame-wise wise high-quality SMPL parameters including 3 E 100 0 E Rº I E IR. and v E R3 we the CameraHMR(Pate) & Black 2025) for the human mesh estimation, due to its high stability and accuracy. Moreover, we perform temporal smoothing on the trime-wise SMPL parameters and obtain the positions of the 3D keypoints J RR21x3 through motion retargeting. Finally,
17

---

Published as a conference paper at ICLR 2026
Table Quantitative comparison with state-of-the-art models. Our model is evaluated against both leading closed-source models and open-source baselines. Bold indicates the best performance and second- best results are underlined.
Metrics
Human Eval
Human Anatomy
Motion
Dynamic
Aesthetic Quality
Vidco Quality
Prompt
Posture Plausibility
Smoothness
Degree
Following
Veo 3.1
85.5
99.3
67.5
59.9
90.2
89.9
92.1
Kling 2.5 Turbo
84.7
99.0
70.0
59.7
90.4
89.3
91.0
Wan-5B
82.3
98.7
62.2
58.3
72.7
77.5
69.1
EchoMottion (Wan55))
83.2
99.1
64.0
58.2
80.4
81,4
80.8
leverage Qwen- VL-Narrator (qwe) to caption the videos. Our construction mechanism yields about
80k high-quality paired datasets.
A.5 COMPARISON WITH TOP-TIER COMMERCIAL MODELS.
We conduct quantitative comparison with leading commercial models, evaluating on 40 prompts specifically selected for their motion complexity (Table 3 While gap exists compared to top-tier closed-source models like Veo 3.1 (veo) and Kling 2.5 Turbo (kli), this is an expected outcome given the vast disparities in model scale, training data, and computational resources. Despite this, results highlight two key takeaways: First, EchoMotion achieves a competitive Motion Smoothness score, confirming the effectiveness of our joint modeling approach for kinematic quality, Second, it significantly outperforms its video-only baseline (Wan-5B), especially in Posture Plausibility and Human Anatomy. This empirically validates that our method provides substantial gains in human video generation.
A.6 MORE QUALITATIVE RESULTS
In this section, provide expanded gallery of qualitative results to further demonstrate the capabilities of EchoMotion.
Figure showccess diverse of text-to-video generation results, highlighting the model's ability to handle complex prompts across various activities and environments. Figure IT focuses on the notion-to-videy task, illustrating how EchoMotion precisely translates 3D motion sequences into realistic videos while adhering to textual descriptions of appearance and context.
A.7 MORE QUANTITATIVE RESULTS
A.7.1 TEXT-TO-VIDEO.
To provide a more granular understanding of our model's performance and robustness, we present detailed quantitative comparison across various motion categories. Figure 12 visualizes the per- formance of our proposed model, EchoMotion nine Attelriiesttlltticc, Arts, Ball, Daily, Fight,
against several baselines and ablations.
The evaluation conducted on diverse Gymnastics, Ice Show Sports, Outdoor, and Water Sports-in addition to an overall average perfor- mance. Models are assessed on four key metries: Aesthetic Quality, Dynamic, Human Anatomy, and Motion Smoothness.
consistent trend emerges from the results: while all models achieve competitive scores in Aes- thetic Quality, our final model, EchoMotion (WaniiB), establishes significant lead in the metrics most critical to motion fidelity: Human Anatomy and Motion Smoothness. For instance, the " "Av- erage Performance" comparison, in these two areas. This strongly indicates that our approach particularly effective at generating anatomically correct human figures and ensuring temporally coherent, smooth movements, which are common challenges in human video generation.
model outperforms the next-best baseline by clear margin
The strength of our model is further validated by its consistent high performance across wide spectrum of motion types. From the intricate and precise movements "Gymnastics" and "Arts" to the large-scale, dynamic actions in "Water Sports" and "Fight," model consistently ranks first.
18

---

Published as conference paper at ICLR 2026
jownn fod conciied poined clonnal honkir sodbwry riger porvrr from ite lean lcape awe are rouio the beniinhal abow Aeal and dioppic o
Yo rablig weooing secoiiers witk
Jinai ailee atter vi'k tayaag.."
crivede sooggh for donghng soleweh rie proichee hiss she sllgwehh frosh
wide vossr ecopler Ausii Acrionaaal Aerane
Aereyldty Aast hasoriom alra lay Abe sosa and paw vare makes was pollo kav Steigg
Oolignl droud
pootot troght reving fright dereeigg fyfver gooch folleg
neeet eaapesloun Novr Jemevr evantess dhund Anghtt the boodet
de Cpdrealond woske Aox refict delice dless voxtivaped and iooaas in unl bei and of sax cos dlps snoudh emuutiig bor aidick
angleuu yoeec fhwns pracing eut
cuntitser gtlle noully....
C..4jo jounng wilit musk saffe musk bucdir light tha cnyload Alt bod sigit) lise fox. avs qwead wide vagge Actiace side ore dussed shood suaitlly payfooiig
and skor lvv nening rually frum Jowe right mway bookig pres reiggng sige frewey
"A polis aosest underiaas abllii Bel axlone
fowsing sautec.
rother muann Aer anderaar
étoies
vatrd tatt veto 2. fronet naw craiing She rover for And resaent or daal for rles masehaad bude Akyko,
coant sonas wat ahovee antir Accke ds conte her eash proucilly esten the clnol
oary astoants apen pustee cuttral brngg regeteen clest Piea)
ECA
MEAA
Figure 10: Additional text-to-video generation results from EchoMotion. These examples showcase the model's capability to generate diverse range of high-quality videos, spanning various activities (e.g., kayaking, skateboarding, skiing) and environments. The results demonstrate faithful adher- ence complex and detailed textual descriptions.
This demonstrates the generalizability and robustness of method, proving it is not overfit specific type of motion but can handle a diverse range of human activities effectively.
A.7.2 COTTON-TO-VIDBO
We provide a quantitative comparison on the motion-to-video task against several leading methods on the case from VACE-Benchmark (Jiang al. 2025). including Tex(2Video-Zert (Khachatryan et al. 2023), Follow- Your- (Ma et al.) 2024a) VACE-14B (iang et al. 2025), and the special- ized animation model
14B (Cheng al.) 2025). As shown Table EchoMo- tion achieves highly competitive results, demonstrating excellent video quality and strong kinematic plausibility with qualitative results provided in Figure As shown Table EchoMotion achieves highly competitive results, demonstrating excellent video quality and strong kinematic plausibil- ity. The qualitative results Figure [3 visually corroborate these metrics, showcasing the model's ability to generate high-fidelity videos where the character's 's movement is highly faithful to the in- put motion, while the character's 's appearance and the background scenery strictly adhere to the text prompt.
is important to clarify that our task a cross- modsl completion challenge (generating from both text and motion), which is distinct from standard image animation (driving source image with
19

---

Published as a conference paper at ICLR 2026
young man rinnling caster of a eliely ln goon weaning gray kusdir and tight iestning pastr, ragaged strangh trainieg backgroned fentores noutly werangel dark monal racks, which bolld dunibtelle und colooed roii bulle-wuungs. shop blus, and darkk grase spheres ure introperrall wilk clear' virbbee brick wall testue in the back...
'I young South Azian male dollg physical traininng on raral pash dusk. Hi dus a losn and suseesaar build, short halk and a light beard. He is wearing dark gray sleeveiess blue shorts, decorated with light hem. He barefoot the dur path The bachground feanures open, fallow
faraland, with dried grousud..."
young
lail lowse-fiting insovrs, inductrial-attle indhor sparse bankgrrundd
dextoral concoste a lurge Bounstsciilnng windon mudion stable and fixel, blackeaaii-nitt touer autiniiiiin original Rigfocuntrust stile, and the light and shadow str arsucures resaai anchunged. creating caln yet tense overall stmophere....
"I deep skiuned, othlens Affican fonale atllcn is the Bool phase of her run up for long jump an ouidoor track and folld verue. She deep rod rubler rack with ligie yellere wather blocks vooble the olse the track. The background fouturer slightly bisrred palm and shrubs, interpprree with soattere grap
ropical flowers
Figure 11: Additional results for the notion-to-video task. These examples illustrate EchoMotion's s ability accurately render a given 3D motion sequence into visually coherent video. Crucially, the model follows the kinematic guidance from the motion input for the action, while simultaneously adhering to the textual prompt for the subject's appearance, attire, and scene context.
tion). For the comparison with the first frame of the ground truth video and transformed it by Plux-Kontext (Batifol al. 2025) as its input. Since this setup does not use text prompt to control for the Wan2. A-iimat-- 14B model, the Prompt Following metric not applicable for this method.
14B, which requires a source image, we provided
The result highlights key advantage of EchoMotion: its ability to jointly understand and synthesize from both textual and motion inputs, offering greater flexibility. It is particularly noteworthy that despite being more compact (5B) and versatile motion-video multi-task model, EchoMotion deliv- ers performance on par with larger models specialized for single task. Crucially, this versatility is achieved with remarkable parameter efficiency. Unlike approaches that require training auxiliary control module (c.g., a ControlNet branch), EchoMotion natively supports the motion -oviiioo task through its unified motion-viee video multi-task training, without introducing any additional parameters.
20

---

Published as a conference paper at ICLR 2026
Won-1.38
EchoMstiontWam 1.338
Wen:5B
Z EchoMotionWan 3B:
Average Performance
Category: Arts
Category:
Category:
Score
3
Category: Daily
Category: Fight
140
3/0
Qnety
Category: Cymnastices
Category: Ice Show Sports
Category:
Category: Sports
Score
Figure 12: Per-category quantitative comparison. We provide a detailed performance breakdown of model (Wan2.2 5B EchoMotion) and other baselines across nine distinct motion categories, plus an overall average. Models are evaluated on four metrics: Aesthetic Quality, Dynamic, Human Anatomy, and Motion Smoothness. Our model consistently achieves superior performance, partic- ularly in Human Anatomy and Motion Smoothness, across all tested scenarios, demonstrating its robustness and figh-iidelity motion generation capabilities.
A.7.3 IIDEOTT--MTTIO.
Following the evaluation protocol of recent works like ChatHuman LLn et al.) 2025), performance of our Video-to-Motion capability on 200 samples from the 3DPW test set (Von Mar- card et al. 2018). We report the Mean Per-Joint Position Error (MPJPE) and Mrorrstess-Alignel
) we assess the
21

---

Published as a conference paper at ICLR 2026
abou tedit porr ure lanas alas exparaie Aloned
"At
Joain the bookk manniy atiint
jour tooad dech weatag clis white axplans wil
abootionaa liee' Aggh
ward for wart nulessng ad senu leis The andiee Alinz ls wil porr sn wadle
mabist abibbdyy Aggage tpolhsoo her and fady lacs Her uproutan edlond rish Aesle dinaso, and the di doos on pluns paye pary morkin giv.."
Crubitre
aepale palkid
anta
of
bhy pedigiiiee sid, widk woldow des workin woandt
Ohrra
Figure 13: Qualitative comparison with the Motion-to-Videe task with baseline methods.
Table 4: Quantitative comparison on human motion generation. Best results are highlighted in bold
and second-best results are underlined.
Video Quality
User Study
Aesthetic
Quality
Motion
Smoothnest
Overall
Consistency
Temporal
Flickering
Prompt
Following
Pose
Consistency
Overall
Quality
Texi2Video-Zero
57.6
79.7
23.9
76.6
44.2
70.1
40.9
YallowYYourPare
48.8
90.1
26.1
88.0
50.9
79.2
42.1
VACE-14B
60.2
98.6
26.4
97.3
79.2
82.4
66.9
Wan2.2-Animate- 14B
61.4
98.9
28.1
99.2
87.2
68.9
BchooMtiion(Wan-5B)
59.2
99.1
26.9
98.4
78.2
82.2
65.2
MPJPE (PA- (P)M))) and categorize the compared methods into two groups: reconstruction-base. and geeeeettve--baaee based, as shown in Table
The results show that while our method does not surpass specialized, reconstruction-baat models like MMRR2.GGell et al.) (2023), this an expected outcome. Reconstruction based methods are typically optimized for the single task of human mesh recovery and often rely on strong supervi- sion from 2D keypoint annotations during training. In contrast, our EchoMotion model is designed as versatile, motion-video multi-task generative framework, which inherently involves trade-off between specialization and flexibility. Within the more comparable category of generative- based approaches, method achieves performance par with the state-of ofce-----hhhtt ChatHuman, demon- strating the effectiveness of our unified model, which supports not only motion recovery but also broader generative tasks.
A.7.4 MOTION QUALITY.
While EchoMotion is primarily designed for video generation, we also evaluate its motion synthesis quality. Table 6 compares its performance with that of specialist motion generation models. No- tably, a direct comparison using the Frechet Inception Distance (FID) with baseline methods is not applicable, this metric is highly dependent the training data distribution. Our model is trained
22

---

Published as a conference paper at ICLR 2026
Table 5: Quantitative comparison of Human Mesh Recovery performance the 3DPW dataset Von Marcard et al., 2018). We categorize methods into aeconstruction-based and generative-basel approaches. Best results are highlighted in bold.
Method
PA-MPJPE І
MPJPE +
SPIN (Kolotouros et al.
2019
62.9
102.9
Rec-Based
HMR2.0 (Goel al. 2023
58.4
91.0
ChatPose (Feng al. 2024
81.9
163.6
Gen-Based
ChatHuman (Lin et al. 2025
58.7
91.3
Cehooootion(Videooooooo
59.8
94.1
Table 6: Quantitative comparison of motion generation quality. The metrics are: Frechet Inception
Distance (FID), Pose Plausibility (PP), Prompt Following (PF), and Motion Smoothness (MS).
Method
FID 4
PP f
PF 1
MS f
MLD
71.4
63.7
91.5
MotionGPT
80.1
72.3
93.9
EchoMotion (Wan 1.3B)
10.9
79.5
78.8
92.2
EchoMotion (Wan 5B)
11.3
80.8
82.3
92.4
on our proposed Ve dataset, whereas the baseline models (e.g. MLD (Chen et al. 2023). Mo- tionGPT (Ting al. 2023) ) are trained on distinct, motion-only datasets like HumanML3D (Guo 2022). Consequently, we report FID scores only for the model trained on HuMoVe Ve dataset.
ctt al.
For fair and practical comparison, generated a set of 50 diverse prompts using LLM (Team et al. 2024), rendered the generated motion parameters into mesh videos, and then asked human annotators to score them three key aspects: Pose Plausibility (PP), Prompt Following (PF), and Motion Smoothness (MS). The results show that EchoMotion achieves competitive performance on Pose Plausibility and Motion Smoothness, and excels in Prompt Following. We attribute this superior cernuuctionfoollowiii ability to our joint training and sampling paradigm, where the motion module benefits from the rich semantic understanding of the large pre-trained video model.
A.8 MORE ABLATION STUDIES
In this section, we present additional ablation studies to further validate the architectural design choices of EchoMotion.
A.8.1 IMPACT OF POSITIONAL. COLLISIONS MVS-RoPE
To demonstrate the necessity of the spatial extension design within our proposed MVS-RoPE, we investigate the effect of spatial index overlap between modalities. We compare two distinct configu- rations: (a) EchoMotion (Proposed): Motion tokens are treated as a diagonal extension of the visual latent space to ensure unique spatial identifiers. The encoding is formulated as:
MVS- RoPE(J..)... i, R(t, H + + W f
(14)
where and W denote the spatial dimensions of the video latents.
(b) Positional Collision (Baseline): Motion tokens assigned spatial indices starting from the origin, identical to the initial visual tokens. This results "collision" of coordinates:
fM MVS-RoPE t, R(t, f.m
(15)
Both models trained for 1,600 iterations under identical settings. The qualitative comparison is illustrated Figure [4] The results reveal stark contrast: the model utilizing our proposed MVS- RoPE (Left) produces visually appealing and temporally coherent videos. Conversely, the Positional
23

---

Published as a conference paper at ICLR 2026
we/ position collisions (ours)
w/ position collisions
Figure 14: The negative effect of positional collision of the positional embedding.
All Notio-ONy Pestraning Resctin
Single stram DIT (Danere)) Dual stream DIT (Durs
Motion-Crle Prot anigg
0.50
0.8
Flow Matchine Loss
0.65
Flow Matching
0.5
0.35
0.4
0.25
0.20
OK
5K
Gr
24
4k
6k
Training
Training
(a) Ablation Study on Motion-Only Pretraining
(b) Ablation Study on Dual-Stream Architecture
Figure 15: Ablation studies stage training scheme (w/ Motion-Only Pretraining) and a one-stage baseline (w/o Motion-Only Pretraining). (b) Comparison of training performance between Dual-stream DiT and Single- stream DiT baseline. Our architecture achieves a consistently lower loss and converges to a better final value.
key designs. (a) Comparison of training loss between two-
Collision baseline (Right) suffers from severe visual degeneration, generating output with collapsed structures and noise.
This degradation occurs because identifying the modality of token relies heavily on its positional embedding in the attention mechanism. When motion and video tokens share the same spatial coordinates (collision), creates positional ambiguity. This forces the motion tokens to interfere with the visual tokens in the attention mechanism, disrupting the pre-trained generative priors of the video backbone and preventing the model from effectively distinguishing between the latent representations of the two modalities.
A.8.2 TWO-STAGE TRAINING SCHEME.
To validate the effectiveness of our motion-only pretraining, we conduct an ablation study comparing the Phase 2 (motion-videe joint training) convergence of two models. Our full model starts this phase from the checkpoint obtained after motion-only pretraining, while the baseline model skips Phase 1 and begins joint training with motion branch initialized by copying the weights from the video branch. As illustrated in Figure 15]a), the benefits are clear and significant. Our model with pretraining (orange curve) not only (1) begins with a substantially lower Flow Matching Loss, but also (2) converges much more rapidly and (3) achieves better final loss value compared to the baseline (blue curve). This demonstrates that establishing a strong motion prior in Phase provides a superior initialization for the subsequent joint training. This pretraining prevents the motion branch's learning signals from being overwhelmed by the computationally dominant video branch, leading to more stable and efficient joint distribution learning.
24

---

Published as a conference paper at ICLR 2026
A.8.3 DUAL-BRANCH ARCHETECTURE.
We conduct an ablation study verify the superiority of our dual-branch DiT architecture over single-stream alternative. The single-stream baseline processes the concatenated video and motion tokens through a shared set Q, K, projection and FFN layers. In contrast, our dual-branch design employs separate set of weights (i.e., experts) for the projection and FFN layers of each modality.
As illustrated in Figure 15 b), the training loss curves for both architectures reveal clear perfor- mance gap. Our dual-branch model (orange curve) consistently achieves lower flow matching loss throughout the training process compared to the single-stream baseline (blue curve). Notably, our model converges to a significantly lower final loss, while the baseline plateaus at higher value, as highlighted in the magnified inset. This result suggests that providing dedicated processing paths for video and motion modalities allows the model to learn more effective and specialized represen- tations. This approach avoids the potential feature interference that can occur in shared-wer weight, single-stream architecture, ultimately leading to superior model performance and a more robust joint distribution model.
A.9 EXPERIMENTS DETAILS
We adapt two pre-trained video foundation models, Wan2.1-1.3B and Wan2.2-5B, by integrating our dual-modality blocks.
For the 1.3B model, replace all original DiT blocks with video-motion blocks. This results a final model with 2.6B parameters.
For the larger 5B model, we adopt a hybrid strategy, replacing half of the video blocks with our dual-modality blocks, which yields final model with 7.5B parameters.
All models were trained on NVIDIA A 100 80GB GPUs. The 2.6B model required approximately 2,300 GPU hours for pre-training, while the 7.5B model required 4,000 GPU hours. Detailed train- ing hyperparameters are provided in Table
Table 7: Experimental settings of EchoMotion 1.3B model.
Parameter
Value
Transformer dim
1536
Numbers of heads
24
Numbers layers
30
Number of video-only blocks
0
Number of video-motion blocks
30
Video height
480
Video width
832
Video frame
81
FPS
16
Batchsize
2
Train timesteps
1000
Train shift
8.0
Optimizer
AdamW W
Learning rate
8e-6
Weight decay
0.001
Sample timesteps
50
Sample shift
8.0
Sample guidance scale
6.0
A.10 COMPUTATION AND PARAMETER ANALYSIS
As illustrated in Figure 16 it is crucial to distinguish between model's parameter count and its computational cost. While introducing the branhb architecture significantly increases the to- tal number of parameters, the growth in computational overhead (i.e., FL.PPs) is disproportionately
25

---

Published as a conference paper at ICLR 2026
Table 8: Experimental settings of EchoMotion 5B model.
Parameter
Value
Transformer dim
3072
Numbers of heads
24
Numbers of layers
30
Number of video-only blocks
15
Number of video-motion blocks
15
Video height
708
Video width
1280
Video frame
121
FPS
24
Batchsize
1
Train timesteps
1000
Train shift
8.0
Optimizer
AdamW
Learning rate
8c-6
Weight decay
0.001
Sample timesteps
50
Sample shift
8.0
Sample guidance scale
6.0
Parameters
(PetaFLOPs)
Model Comparison: Parameters Computation
300
261.4
237.0
250
Parameters (Billions)
FLOPs (PetaFLOPS)
200
150
92.2
81.7
1.3
50
Wan-1.38
(chaMotionnWaan11.33
Wan-58
CchaMotiooWWan-SBe
Model Architectures
Figure 16: Visualization of our model scale and computational cost.
small. For instance, our 1.3B dual-branch model has double parameters than its video-only counter- part but requires only 12.9% more FLOPs for a single forward pass. This trend holds true for our larger 5B models: the hybrid architecture introduces 51 % more parameters but only increases the computational cost by 10.3% (from 237.0 to 261.4 PFI.OPs).
This disproportionately small increase in FLOPs is a direct consequence of our architectural design. The additional parameters are exclusively dedicated to processing motion tokens, which, despite their rich information content, represent a small fraction of the total sequence length (e.g., 4,131 motion tokens vs. 32,760 video tokens in the 1.3B model). As a result, the impact on practical inference latency is marginal. This demonstrates that our dual-branch approach is highly efficient method for incorporating multimodal capabilities, significantly expanding model functionality with minimal increase in its computational footprint.
26

---

Published as a conference paper at ICLR 2026
A.11 SOCIETAL IMPACTS AND SAFEGUARDS
The advancements in controllable human video generation grounded in kinematic models, as exem- plified by our work Ectoon present profound societal impacts. By enabling the creation of high-quality, kinematically plausible videos with direct control over human motion, our frame- work democratizes access sophisticated animation and visual effects tools. This innovation can dramatically streamline workflows in industries such as filmmaking (for pre-visualization) video games (for realistic character animation), virtual reality (for lifelike avatars), and even in specialized fields like sports science and physical therapy for visualizing complex biomechanics. The ability to generate videos from structured motion data offers unprecedented creative flexibility and new paradigm for digital human creation.
However, the power such generative technologies also introduces significant challenges. The au- tomation of high-fidelity animation could lead to job displacement for traditional 3D animators and motion capture artists, necessitating industry-wide adaptation and focus on creative direction over manual execution. More critically, the potential for misuse poses serious ethical concern. The abil- ity to generate realistic videos of individuals performing complex actions they never took could be exploited create highly convincing deepfakes, impersonations, ing public trust. Furthermore, ensuring that our MoVe e dataset and the models trained on it are free from demographic physical biases is essential prevent the generation of content that reinforces harmful stereotypes.
misinformation, thereby erod-
To address these issues, we implement robust safeguards. Our models inherit the safety mechanisms from their foundational bases (Wan2.1 and Wan2.2), which include filters to detect and prevent the generation of inappropriate or harmful content. We are committed adhering to strict ethical guidelines regarding the use of our technology. By open-sourcing our models and the MoVe e dataset, we aim foster transparency, enable independent auditing, and encourage community-drin approach to developing responsible and ethical human-centric generative Al.
27


Reviews:
[{"review_id": "doG4vqSPDL", "reviewer": "Reviewer_ehBs", "summary": "This work proposes EchoMotion that accepts both video and human motion modalities for video generation with a mixed multi-modal in-context learning strategy during the training. This work also introduces a new human-centric video dataset HuMoVe that includes paired video, 3D human motion parameters, and text data.", "questions": "1. Why is there no video tuning result for Wan-1.3B at Table 1?\n2. I saw there is a parameter named FPS in Table 4. Is that the FPS of the input video or something else?\n3. Section 4.3 mentions that EchoMotion can operate bi-directionally. Can you provide quantitative results for this part to see the performance comparison with other state-of-the-art models?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work proposed a Dual-Modality DiT architecture that accept input and output with different modality.\n2. This proposed Motion-Video Synchronized RoPE is an interesting idea to add motion information to the model.\n3. This paper proposed a new high-quality dataset for video, human motion and text.", "weaknesses": "1. The novelty of the Dual-Modality DiT and Motion-Video Synchronized RoPE is limited. The notion of multi-modality DiT is not new and the idea of adding motion information is well studied in human mesh and skeleton generation tasks.\n2. There are only baseline model results of Wan-1.3B and Wan-5B which are not enough to give accurate evaluation of the proposed architecture. \n3. There is no ablation study to show the effectiveness of each proposed block in the architecture.\n4. The model efficiency evaluation can add metrics like average generation fps for a more direct comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0ufoCDLfKF", "reviewer": "Reviewer_pvjq", "summary": "This paper introduces EchoMotion, a new framework designed to solve a critical problem in video generation: the synthesis of complex and kinematically plausible human motion. The authors argue that existing models, trained on pixel-only objectives, prioritize appearance fidelity at the expense of learning the underlying physical principles of human articulation, leading to anatomical artifacts and unnatural movements. To address this, EchoMotion's core idea is to model the joint distribution of video (appearance) and 3D human motion (kinematics), rather than just the video distribution conditioned on text. MVS-RoPE is proposed as a unified 3D positional encoding for both video and motion tokens and establishes an inductive bias for video-motion temporal alignment. A large-scale dataset HuMoVe with 80,000 video-motion pairs is constructed for training and achieves better human-centric video generation results.", "questions": "I believe this paper is substantial, demonstrates improved results, and serves as a positive contribution to advancing the field of controllable video generation. Please refer to the weaknesses to further improve this paper.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper clearly identifies a fundamental weakness in current human-centric video generation models for kinematic correctness and proposes to explicitly model the joint distribution of video and motion as a strong inductive bias to enhance the video generation performance;\n2. The MVS-RoPE design is clear and well-justified to the non-trivial problem of aligning modalities with different temporal resolutions.\n3. The creation of the 80,000-pair HuMoVe dataset is a substantial contribution to the field. The lack of large-scale, high-quality, paired video and 3D motion data has been a major bottleneck.\n4. The experiments are thorough and well-designed.", "weaknesses": "1. The paper does not provide a clear description of the specific \"open-source datasets, movies, and the internet\" used to build the HuMoVe dataset. Furthermore, the extracted motion could be noisy as the ground truth;\n2. The framework's reliance on the SMPL model as its parametric motion representation creates an inherent bottleneck for fine-grained realism. SMPL is a whole-body model that offers very limited, or no, supervision for highly articulated and expressive areas like individual hand gestures and facial expressions.\n3. Is the strong inductive bias harmful for those physical disabilities or significant bodily variations, such as amputees, as the underlying parametric model does not support this topology.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KTYFJVN3N4", "reviewer": "Reviewer_T7MT", "summary": "This paper aims to overcome the limitations of motion generation based on pixel-level supervision in previous studies by proposing joint modeling of human appearance and motion. \n\nThe authors propose a DiT-based architecture that processes tokens from two modalities. The SMPL parameters are used to represent human poses, and to emphasize the dual-modality nature, Query, Key, and Value extracted from both the video and motion are concatenated and processed through self-attention. This structure enables attention to consider multiple modalities, which is advantageous for joint modeling. A motion-video synchronized RoPE (MVS-RoPE) which is an encoding method applicable to both modalities, is also proposed. Specifically, a diagonal extension is proposed to prevent interference between motion latents and video latents. \n\nIn addition, the authors propose the HuMoVe dataset, containing over 80,000 video-motion pairs. This dataset includes descriptive textual captions, 3D SMPL motion parameters, and video pairs, making it valuable for multi-modal generative modeling that considers vision, text, and motion jointly. \n\nThe experimental results present various metrics and human evaluations, showing performance improvements over baselines. Furthermore, ablation studies for each module are provided to analyze the effectiveness of the proposed methods.", "questions": "- p.2 L66: The authors mention that previous works are limited because, even with a 3D prior, supervision is applied after projecting it into 2D, which constrains accurate 3D (motion) generation. However, since the proposed method is also trained through a video diffusion process, hasn’t it still failed to overcome the problem of losing 3D information?\n\n- p.4 L189: Motion tokens are generated as 51 dimension. What is the specific reason for this number?\n\n- Motion tokens are added diagonally to visual tokens. Since maintaining temporal alignment is sufficient, there seems to be no strict reason for using the diagonal arrangement. Is there experimental evidence supporting the \"positional collisions\" mentioned in the text?\n\n- Eq. 6: If the reviewer's understanding is correct, the last term must be u_{\\theta}(\\phi, m_{t}, \\phi)\n\n- Quantitative comparison is provided only as self-evaluation. Although direct quantitative comparison with previous studies may be difficult, joint modeling is expected to enhance the performance of both the video and motion decoders. Therefore, a quantitative comparison between the videos and motions generated by the proposed framework and those produced by conditional generation methods (e.g., VideoJAM), given GT as condition, could better highlight the advantage of joint modeling (even if the performance does not surpass that of conditional generation).\n\n\n- Minor Comments:\n\n-- p.5 L231: i is the motion token -> i is the motion token index ?\n\n-- Fig.3: The distinction between \"noisy\" and \"clean\" is described only in text. It would be clearer and easier to understand if visual symbols were added to indicate the presence or absence of noise.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper proposes the large-scale HuMoVe dataset. Since the dataset includes test captions, videos, and motion parameter pairs, it is highly useful for multi-modal modeling tasks.\n\n- MVS-RoPE that can be jointly applied to visual and motion embeddings is proposed. This encoding technique utilizes diagonal positioning to prevent interference between vision and motion latents, which is a reasonable approach (although more experimental evidence is needed to support this).\n\n- The paper is easy to follow.", "weaknesses": "- The deep network structure is only a simple extension of existing networks. Except for MVS-RoPE, the network mainly uses self-attention on concatenated features for joint modeling, which is quite simple and straightforward. Discussion on whether other components could be improved to better support joint modeling would strengthen the paper.\n\n- The quantitative evaluation relies only on self-evaluation. Even if direct comparison with prior studies is difficult, the paper should include analyses comparing the video and motion decoder performance improved from joint modeling with existing conditional generation methods (e.g., VideoJAM) to show the degree of improvement or equivalence.\n\n- The explanation of how text descriptions were generated for the HuMoVe dataset needs to be clarified. In particular, since the initial data were created using an LLM, the paper should provide more detailed information about the prompts used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mm1hGvVpZ5", "reviewer": "Reviewer_GwaP", "summary": "Inspired by VideoJam , this work establishes a model for the joint distribution of video and motion. It explicitly denoises parametric motion and performs text-to-video & motion generation. The results demonstrate an improvement in motion smoothness and human evaluation scores compared to the baseline (Wan video ).", "questions": "1. In the supplementary materials, specifically in sample 6 (especially the last frame) and sample 15, some very unnatural or incomprehensible human poses appear. What are the possible reasons for this?\n\n2. As mentioned in the paper, EchoMotion can perform video-to-motion and motion-to-video  tasks. Could you provide quantitative metrics to demonstrate the performance of these tasks? Specifically, for motion-to-video, could you compare it with models like Champ , Animate Anyone, or WanAnimate (since its base model is also Wan video)?\n\n\n\n3. The VBench metrics  used in the comparison are not specifically designed for human motion. Would it be possible to compute an FID (Fréchet Inception Distance) on the generated SMPL motion parameters?\n\n4. It is suggested to also include comparisons with closed-source models, such as Kling, Veo3, etc.\n\n5. If you were to use SMPL-X as the motion representation instead of SMPL, would this lead to an improvement in the representation of hands?\n\n\n6. Could an ablation study be provided for the complex training process described in Section 3.2 ?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "Originality:\n\nIt designs and establishes a modeling framework for the joint distribution of video and motion.\n\nQuality:\n\nThe quality is acceptable.\n\nClarity:\n\nThe paper is well-structured and clearly articulated.\n\nSignificance:\n1. This work proposes a solution for modeling the joint distribution of video and motion.\n2. Community Contribution: The authors commit to open-sourcing their code, which will be a valuable public resource for advancing the field", "weaknesses": "1. Limited quantitative experiments: The paper only compares results with its base model using metrics that are not specialized for human motion. It lacks comparisons with closed-source models like Kling or Veo3 (it doesn't necessarily need to surpass them, but at least show the gap with SoTA models). The evaluation metrics are not focused on human motion.\n\n\n\n2. Lack of necessary ablation studies: The effectiveness of the video-to-motion and motion-to-video  capabilities is unknown, as no quantitative results are provided. This is crucial for validating the joint distribution modeling. Furthermore, there is no ablation study for the complex training process .\n\n\n3. The visual quality demonstrated in the supplementary materials is still subpar. There are instances of impossible human poses, and the characters' hands are very blurry.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)