Animating the Uncaptured: Humanoid Mesh Animation with Video Diffusion Models
Authors: Marc Benedí San Millán
Angela Dai
Matthias Nießner
Venue: ICLR 2026 Poster (2026)
Keywords: Motion generation
Motion Tracking & Transfer
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: A method to animate humanoid meshes from a text prompt by transferring motion generated by video diffusion models to the mesh.

Abstract: Animation of humanoid characters is essential in various graphics applications, but require significant time and cost to create realistic animations. We propose an approach to synthesize 4D animated sequences of input static 3D humanoid meshes, leveraging strong generalized motion priors from generative video models -- as such video models contain powerful motion information covering a wide variety of human motions. From an input static 3D humanoid mesh and a text prompt describing the desired animation, we synthesize a corresponding video conditioned on a rendered image of the 3D mesh. We then employ an underlying SMPL representation to animate the corresponding 3D mesh according to the video-generated motion, based on our motion optimization. This enables a cost-effective and accessible solution to enable the synthesis of diverse and realistic 4D animations

Published as a conference paper at ICLR 2026
ANIMATING THE UNCAPTURED: HUMANOID MESH ANIMATION WITH VIDEO DIFFUSION MODELS
Mare Benedí San Millán, Angela Dai, Matthias Nießner Technical University of Munich, Germany (marc. bereed,eedd,,.. angel la dai, n easner r @ttumdee
ABSTRACT
Animation of humanoid characters is essential various graphics applications. but requires significant time and cost to create realistic animations. We propose an approach to synthesize 4D animated sequences of input static 3D humanoid meshes, leveraging strong generalized motion priors from generative video mod- els -a--as as such video models contain powerful motion information covering a wide variety of human motions. From input static 3D humanoid mesh and text prompt describing the desired animation, we synthesize a corresponding video conditioned on rendered image of the 3D mesh. We then employ underly- ing SMPL representation to animate the corresponding 3D mesh according to the video-generated motion, based our motion optimization. This enables cost- effective and accessible solution to enable the synthesis of diverse and realistic 4D animations.
Project Website: https://marcb.pro/att
INTRODUCTION
Character animation is fundamental in computer graphics enabling lifelike, expressive, and en- gaging virtual characters for applications such as movies, video games, mixed reality, robotics, and many more. Such characters portrayed with realistic motions help to drive storytelling and interac- tivity, making crafted content more engaging and immersive.
Traditionally, character animation requires significant manual labor from highly-traines artists, who manually craft character rigs, define keyframes for motions, and fine-tune detailed motion behavior. This is both costly and requires significant amount of tedious effort from skilled artists Thus, leveraging learned motion priors to inform character animation would enable much more efficient animation synthesis. However, ground-truth capture of human motion is technically demanding and expensive to acquire, resulting in very limited data available (Mahmood et al. 2019 Liu et al. 2020) Zou al. 2020) for training such motion priors, strongly limiting the diversity and generalization capability of methods fully supervised with such 4D data (Tevet et al. 2023
Shafir et al. 2024).
Recently, video diffusion models (Ho al. 2022 Blattmann al. 2023 Yang et al. 2024), trained on large-scale datasets with abundant vidco data, have demonstrated the ability to generate diverse and realistic videos conditioned on textual prompts. This suggests that these models implicitly learn motion priors that capture how the world evolves over time. Building on this insight, we propose a general approach for animating a 3D humanoid mesh based on text description of the intended motion. Instead of relying on small and constrained 4D human motion capture datasets, we leverage motion priors learned by video generative models, which possess strong representational capacity and can synthesize diverse, high-fidelity human motion sequences.
Given text prompt describing the intended motion, we generate a synthetic video of 3D humanoid mesh performing the specified motion using text-to-video (T2V) diffusion model. This model is conditioned on both the rendering of the 3D mesh and the text prompt. To facilitate motion tracking, employ the SMPL body model (Loper et al. 2015) as deformation proxy for the input mesh, enabling us to estimate the SMPL parameters and track the 3D mesh throughout the generated video frames.
Our approach begins by registering the SMPL body model to the input mesh, leveraging estimated body joint locations from multiple views. We then reparameterize the input mesh's vertex coor-

---

Published as a conference paper at ICLR 2026
"playing tennis"
( Ot)
VDM
L
S
Render
+
D(S,9, s) D ( 94) D (( ((5)0.12))
Figure 1: Animating the Uncaptured: novel approach for text-driven animation of 3D humanoid meshes. Given an input mesh and text prompt, we use video diffusion model to generate a 2D video of the mesh performing the specified motion. This motion is subsequently extracted and transferred to the 3D geometry via sparse and dense tracking.
dinates into barycentric coordinates relative to the SMPL mesh faces. To transfer motion from the generated video to the input mesh, we extract 2D body landmarks, silhouettes, and dense DI- NOv2 (Oquab al. 2023) features from the video frames, which serve as tracking cues for accurate motion reconstruction. This provides an easy, accessible approach to generate wide range of real- istic 4D humanoid animations.
Our contributions are summarized as follows:
We tackle the task of mesh animation by leveraging the strong generative capabilities of text-to-video (T2V) diffusion models.
We propose pipeline to robustly track the motion from the generated video by combining body landmarks, silhouettes, and dense features.
RELATED WORK
Text-to-Human Motion Generation Human motion generation is pivotal area in computer graphics, focusing on synthesizing realistic human movements for applications such as animation and virtual reality.
Although human motion can be captured using motion capture (MoCap) systems ( O'RRurkee & Badler 1980 1980 [onescu al. 2014 Von Marcard et (208)), this process requires specialized setups and actors which is expensive and tone-consuming. This motivates the development of data-driven methods that can generate human motions from different input signals, such actions (Guo al.
2020), and audio (Fukayama & Goto, 2015; Crnkovic- -riis & Crnkovic-Friis 2016 Deng et al.
2025 enabling the creation of diverse and realistic human animations without need manually
capture every new motion.
In particular, text-to-hune -ott-to-human motion generation aims synthesize human motion sequences based on text prompts describing the desired action in natural language. Sagguagg.ZZhngggg et al. (2024b) proposed the first text-conditioned diffusion model (Ho al. 2020) for human motion generation. Such text-to- motion methods typically require paired text and body pose parameters for training, which are often obtained from motion capture data. However, such data is limited in quantity and diversity, and may
not generalize well to unseen actions (Guo al. 2022
Plappert et al. 2016 Guo et al.] 2020).
Similar to our approach, MotionDreammrr (Uzolas et al. 2025 proposes to extract motions from video diffusion models to animate meshes. As proposed by Tang al. (2023); Zhang et al. (002a)); Luo et al. (2023), they extract semantic features from the intermediate activations the diffusion model and perform feature matching between frames. This is expensive process and limits their
2

---

Published as a conference paper at ICLR 2026
pipeline to low resolution features. In contrast, employ both sparse and dense features for mesh registration and tracking, and our focus on humanoid meshes enables us to use stronger body priors for regularization. Similarly, AnyMoLe (Yun et al. 2025) leverages video diffusion mod- els for motion in-betweening of arbitrary characters. Their approach involves fine-tuning the video model on rendered context frames and training scene-speeific joint estimator which is computa- tionally expensive. In contrast, our method is designed for text-to-motion generation for humanoid meshes and does not require any fine-tuning of existing models.
Human Pose and Shape Estimation from Images and Videos Monocular HPS estimation re- mains particularly challenging due to the inherent ambiguity in the 2D-to-3D mapping and the ab- sence of depth information. To solve this, methods often rely on statistical body models (Loper itt al. 2015; Osman et al., [020 Xu et al. 22200; Pavlakos et al., 2019; Tiang et al., (024) which provide a prior of shapes and poses and a low-dimensional parameterization human body.
Two main approaches have been widely adopted to address this problem: optimization- based meth- ods and regression- based methods. Optimization-ban based methods iteratively refine 3D pose and shape parameters by minimizing reprojection error between the 3D model and 2D image observations, such as silhouettes or body keypoints (Lugaresi et al., 2019; Cao et al. 2019 Pishhhulin et al, 2016 Khirodkar et al. 2024). Bogo et al. (2016) proposes to predict 2D joints location from an image and then optimize the SMPL body prameters to minimize the reprojection error. To seek a stronger body prior, Pavlakos al. (2019) trains variational autoencoder (Kingma & Welling 2013) to learn the representation of human poses. Regression- -aased methods leverage deep learning directly estimate 3D pose and shape parameters from images or videos. Frameworks such as HMR (anaaaaaa et al. (018) and SPIN (Kolotouros et al. 2019) employ neural networks to predict SMPL parameters by learning from paired image data with eseudo-ground truth annotations. While these methods achieve real- time inference and improved robustness compared to optimization- approaches, their performance is limited by the quality and diversity of the training datasets. As result, regression-basee models often struggle to generalize to cut-of-distributin data, such as syn- thetic videos generated by text-to-video models.
Recent advancements have extended the image-based formulation into the video domain. By lever- aging temporal information, these methods significantly improve the consistency and realism of estimated human motion across frames (Sun al. 2024 Shin et al. 2024 Goel et al. 2023).
Such human pose estimation from images and videos have also been used to synthesize 3D and 4D human-objee interactions (Li & Dai 2024, Li et al., 2024). In this work, use optimization- based approach to focus on humanoid animation. Our approach robustly handles the synthetic videos generated by text-to-video model to effectively generate diverse, realistic motion for various hu- manoid meshes.
3 METHOD
Our method tackles the task of text-to-motion for humanoid meshes. Given a text prompt (P) describing the desired motion and untextured humanoid mesh (8) without any rigging skeleton in an arbitrary pose, the method generates the deformation parameters (θ) to animate the mesh over time. For this, we leverage the motion priors of Video Diffusion Models (VDMss by generating a video conditioned on the prompt and the rendering of the mesh (TRGB E TR. #x33). We use the SMPL body model model[oper al. (2015) as deformation proxy to animate the input mesh through its parameters (section 3.3.1). We then optimize the deformation proxy parameters to match the motion in the generated video (section 3.3.2 The overview of our method illustrated figure
3.1 PRELIMINARIES
SMPL The Skinned Multi-Person Linear (SMPL) model (Loper al. 2015 is parametric body model that encodes body shape and pose variations using a learned low dimeeninsi representation. It is defined by the shape parameters coefficients correspond to the principal components of the body shape and the pose parameters encode the rotations of 23 skeletal joints in axis-angle representation.
R and the pose parameters 0 2333 where the shape
3

---

Published as a conference paper at ICLR 2026
"practicing karate moves"
Video Diflssion Model
Video
Misia Reudering
e
D(S, Ox)
T(v) Vertex
Arub
Regarameieriiatiion
Ti-k
L3
Txu
Cal
t=
0NIL
SMPL
Registration
C
Re-
RALE
Figure 2: Method overview. Given input mesh in arbitrary pose and text prompt describing the desired motion, we generate video conditioned on the text prompt and the rendering of the mesh. We leverage the SMPL body model as a deformation proxy to track the motion from the video and transfer it to the input mesh. Specifically, we fit the SMPL model to the input mesh and associate its vertices with the SMPL faces (3.3.I). Finally, we optimize the SMPL parameters to match the video motion using estimated body landmarks, silhouettes and DINOv2 features extracted from the frames (3..22.
this work, we use the SMPL model as deformation proxy to animate the input mesh. To represent the pose parameters,
leverage the pre-trained variational autoencoder VPoser (Pavlakos et al. 2019). which provides a strong prior on valid human poses. Instead of directly optimizing the pose parameters E 2R23x3 then decoded to obtain the corresponding pose parameters. To simplify the notation, we use the symbol refer both to the pose parameters and the VPoser encoding.
optimize the latent encoding 7 E R-32 of VPoser. This latent vector
3.2 VIDEO GENERATION AND FRAME FEATURES
We start by normalizing S to unit scale and centering at the origin. We define P (S, C*) + I e R H x W x3 the rendering function of the mesh S from camera C obtain the frame JRGB
Video Generation We use Video Diffusion Model (VDM) generate video [IRGB)FF F-1 with F frames depicting the mesh performing the motion described by the prompt. For this, we condition the VDM with the frame TRGB and the prompt P. Note that the first frame of the generated video is the same as the rendered image IRCB
Body Landmarks Estimation We use MediaPipe Pose Landmarker (Lugaresi et al. 2019 to es- timate the body pose landmarks from the video. For each frame, MediaPipe provides 33 landmarks, (ji, W:), where ji are the normalized pixel coordinates and are the confidence scores. We rear- range them to be in the same order the jMinttooiss and apply a smoothing filter to mitigate the effect of noisy predictions.
Dense Features Due to the sparse and noisy nature of landmark estimation, we additionally extract dense per- pixepi features from the frames using a pre-trained DINOv2 model (Oquab et al., 2023) and obtain [T [ Following Dutt et al. 2024), we annotate the vertices of the mesh V = (vx) with the features y {u,²}. where v. is the feature vector of vertex v;. To compute these features, we render the mesh from 100 uniformly spaced cameras positioned on a sphere centered around the mesh extract the features from each view, and back- pack-project them for accumulation onto the mesh vertices. Finally, due to memory constraints, we apply PCA to reduce the dense features and keep the first 64 components.
4

---

Published as a conference paper at ICLR 2026
Silhouette
We extract the silhouette I of the mesh by thresholding the white background.
3.3 OPTIMIZATION
We reduce the task of transferring the motion from the video to the mesh as video tracking problem. First, we register the SMPL (Loper al. 2015) model the input mesh and reparameterize V"s coordinates with respect to the SMPL faces JFMPI. (3.3.1. Finally, we optimize the parameter θ to deform the input mesh match the motion in the video( 3.3.2).
Model Parameters The deformation function D (8, O2 St is function that deforms the input mesh using the deformation parameters O, for frame Note that for D(S, θ,) = S due to the video generation process being conditioned on I The parameters θ are defined as (s. B, fe, Te, R4, At). where s is the scale of the SMPL model, 3 is the shape parameters, and 8,, Tr. Re, and Â the pose, translation, rotation parameters and per-vertex offsets for each frame t.
Note that the shape and scale parameters are shared across all frames. For simplicity, we use O to refer to the deformation parameters for frame t: θ {
To mitigate the influence of noisy signals during optimization, such as the jitter in the estimated landmarks, we opt to use shallow Multi-Layer Perceptrons (MLPs) to parameterize 9, Te, and R That is, we use MLP, TMPP, and RMLP to regress their corresponding SMPI. parameters for each frame. As input, they take the Sinusoidal Positional Encoding (Vaswani et al., 2017) of the frame index. For simplicity, we use the same notation to refer to the SMPL parameters produced by the MLPs. For instance, O2 (MLp(PE(t)) represents the pose parameters predicted by the MLP for time step t.
3.3.1 SMPL REGISTRATION
We choose to leverage the SMPL model (Loper al. 2015) for the following reasons: (1) due to monocular tracking being inherently ambiguous, we benefit from the body prior in SMPL and VPoser (Pavlakos et al. 2019). which ensures that optimization remains within the plausible space of human poses; (2) the input mesh does not incorporate any deformation model such as skeleton or blend shapes.
The first term ensures that the SMPL joints j are close to the estimated 3D joint locations J from the input mesh while the second term minimizes the distance between the SMPL vertices and their nearest neighbors on the input mesh. To obtain J, we triangulate the 2D landmark predictions from MediaPipe (Lugaresi) et al. 2019 using views sampled around the mesh.
We optimize for 5, B, Oo. To. Ro; and minimize both terms in 1 and 2
Ji 12
L,
Wi J
Cypp (V)
Σ
||0- NN((,, LSMPL ||
(2)
[
DEV
We also include squared L2 priors for the shape and pose parameters, as defined in
La (B)
||
Lo (0.) = 110.112
SMPL as Deformation Proxy Finally, we reparametrize the input mesh vertices V with respect to the SMPL model. For this, after the registration process (3.3.I. we find for each vertex E 1 its closest SMPL face /SMPL E JSMPL Then, each vertex v can be represented using the barycentric coordinates of the corresponding SMPL face, along with the distance to the face plane:
7 SMPL + 720 72P5 SMPL + T3uSMPL
SMPL + dn
where 7 are the barycentric coordinates of the input mesh vertex v with respect to the SMPL face SSMPL formed by the vertices uSMPL MPLL and 03
SMMLL d is the distance from the input mesh vertex
5

---

Published as a conference paper at ICLR 2026
to the SMPL face, and is the normal of the SMPL face. We refer to this differentiable function, which computes the location of the input mesh vertices from the SMPL vertices as r L/MPI. + V.
This ensures that the imput mesh vertices are attached to the body model, allowing us optimize the model parameters to match the motion in the video. To address potential outliers, we apply a robust filtering mechanism that identifies and excludes mismatches. Outliers detected by combining multiple criteria: (1) absolute distance between the input mesh vertices and the SMPL model; (2) the angular deviation between vertex normals and face normals of the closest SMPL triangle, ensuring alignment within a threshold of 45 degrees; (3) neighborhood statistics, which analyze the mean and standard deviation of distances between vertices identify points that deviate significantly from their neighbors.
3.3.2 VIDEO TRACKING AND MOTION TRANSFER
We transfer the motion from the generated video to the input mesh by optimizing the following parameters of the deformation model θ₁: T, R2, dr, and A,, keeping fixed the shape and scale parameters 3 and s. We formalize the deformation model as follows:
sS 92 = (s R₁ SMPL SM,L((,,0) +
(5)
Loss terms
We optimize the parameters described above to minimize equation
1 Lioil F Σ (C, + Lui Lb) fregs
(6)
The first data term, Lj. shown in equation minimizes the distance between the re- re-projected SMPL. joints (). and the predicted 2D landmarks ()), where is the confidence score of the predicted land- marks, p is the Berman-McClun loss function (Geman 1987), and N the number of landmarks.
C, wip (3: - j..)
The second data term, Lai, shown in equation 8 minimizes the binary cross-entropy loss between the rendered silhouette (Tel)) and the silhouette extracted from the generated video (T5il), where N HW is the number of pixels.
N Σ
Ssi N
pall t.
log (T + (1 P log(1 r
(8)
The final data term, Cg. shown equation 9 minimizes the cosine similarity between the rendered features (I$) and the dense features extracted from the generated video (I+).
N
C.
N
|
The last term, Legss includes all regularization terms: Crmmp Lex and Co. which are defined in equations and respectively.
Temporal regularizers are used to ensure smooth motion across frames and mitigate the impact of landmark jitter. In particular, we penalize abrupt changes in translation, rotation, pose parameters, and 3D joint locations between consecutive frames. The temporal regularization terms are defined as follows:
T
Liemp (x)
Σ
|| xt-11l2
(10)

---

Published as a conference paper at ICLR 2026
"dancing K-pop"
"kicking soccer ball"
Figure 3: Qualitative results. Visualization of generated mesh animations with our method. Each row shows: the text prompt, the input mesh, and intermediate frames of the generated motion. For all generations, we visualize the frames from the front and side views.
Inspired by Bogo et al. (2016), we also include term to penalize extreme bending of the knees and elbows. See equation [1] This term is defined as the sum of the squared angles between the upper and lower limbs, ensuring that implausible poses with excessive bending are avoided.
Lex. ben. (0)
Σ
exp {0t)
(11)
iE (elbowws knees)
Additionally, as described in equation 3.3. we penalize deviations from the manifold valid
human poses by using the VPoser regularization term defined in equation 3
Finally, we employ E--Rigid-as-Posiiie (Sorkine Alexa 2007 regularization to ensure that the resulting mesh deformation is smooth and preserves the mesh's intrinsic structure.
Feature Mapper Because the appearances of the mesh S and the generated video { F-1 1== diverge time, optimize learnable projection that maps the vertex features y into space better aligned with the features extracted from the video 7 This projection is optimized in a self- supervised manner.
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
In our implementation, we use PyTorch 2.0.1 (Paszke et al. 2017) and CUDA 11.7 Nickolls et al., 2008), and perform differentiable rendering with PyTorch3D 0.7.7 Ravi et al. 2020). We use SMPL (Loper et al. 2013) and VPoser (Pavlakos et al. 2019) as body priors. For frame processing, we use MediaPipe Lugaresi et al. 2019 and DINOv2 Oquab et al. 2023) extract landmarks and dense features respectively. Our neural parameterization consists of shallow multilayer perceptrons (MLPs) with four layers and 128 hidden units per layer. We use a 64-dimensional positional encod- ing of the frame index as input. We used Kling AI (Platform) (202) as a Video Diffusion Model. It generates 5-second videos at a resolution of 768 x 1280 pixels, which we downsample to 384 x 640 pixels during optimization due to memory constraints. For SMPL registration,
described in sec- tion 3.3.I we set the batch size to 8 and optimize for 1, 000 iterations or until convergence, using learning rate of 0.01 with the Adam optimizer (Adam al. 2014). Similarly, for video tracking, as described in section 3.3.2 we use the same batch size and optimize for 4000 iterations or until convergence, with learning rate of 0.001. The complete optimization process takes approximately
7

---

Published as a conference paper at ICLR 2026
Table Pose fitting performance comparison with SMPLIFY-X smoothed version SMPLIFY-X*, WHAM (Shin et al. 2024), Multi-HMR (Baradel* et al. 2024).and our proposed method on untextured sequences from the CAPE dataset. Metries include Per--eeet--oossitteiee Lower values indicate better performance across all metrics.
(Pavlakos et al. 2019). it's
(MPJPE), Per- Vertex-Erro (PVE), and acceleration emror (Accel).
Method
MPJE
PVE
Accel
SMPLIFY-X
0.054
0.057
20.57
SMPLIFY-X
0.053
0.056
02.18
WHAM
0.051
0.054
08.61
Multi-HMR
0.045
0.510
09.03
Ours
0.036
0.041
01.49
1.5 hours NVIDIA RTX 2080 Ti GPU (12 GB VRAM), 16GB RAM, and 4 CPU cores Intel(R) Reon(R) Gold 6240.
4.2 VIDEO TRACKING
We evaluate the tracking performance our method using rendered videos of animated untextured meshes, as this enables evaluation with ground truth while also mimicking the output of text-to- video model conditioned on untextured meshes.
Datasets For evaluation, we use the CAPE dataset (Ma et al. 2020 Pons-Moll et al. 2017). which provides 4D sequences of clothed humans in motion. Each sequence includes a 3D human body mesh with the corresponding SMPL+D registration. We use these SMPL parameters ground truth for evaluation. We evaluate random set of 26 sequences from the CAPE dataset, totaling 7,000 frames, and render their untextured SMPL meshes.
Baselines We compare method with two learning-base baaeed methods, WHAM (Shin et al. 2024 and Multi-HMR (Baradel et al., 2024). and an optimization-bann method, SMPLIFY-X (Pavlakos et al. 2019). Since SMPLIFY-X does not enforce temporal regularizations, filter to its results to improve the performance on video sequences. Finally, as our method uses the input mesh get initial alignment for the first frame of the sequence, align the results of WHAM and SMPLIFY-X to the input mesh using Procrustes alignment as well for fair comparison.
apply smoothing
Evaluation Metrics We evaluate the methods using three metrics: Peaa--er-Joit-Posittee Error (MPJPE). Per- Sortex-Errov (PVE), and acceleration eтrоr, which quantifies the fiter-frame smooth- ness of the reconstruction. We follow established evaluation from Pavlakos et al. 2019), and include acceleration error as in (Shn et al. (024 to measure the temporal consistency.
As shown in table I our method outperforms SMPLIFY-X, SMPLIFY-X with smoothing, Multi-HMR and WHAM in all metrics. We note that Multi-HMR and WHAM. as learning- based methods, are not trained for tracking untextured videos, which causes its performance to degrade from its original setting. Our approach is better suited for tracking videos generated by text-to- video model conditioned on untextured meshes.
Results
4.3 MOTION FROM GENERATED VIDEOS
Perceptual Study We evaluate the quality of the motions generated by our method by conducting perceptual study. A total of 30 participants took part in the study and were asked to answer the following questions: (Q1:) "Which motion best aligns with the given text prompt?", (Q2) "Which method generates more natural motions?". (Q3) "Which motion do you prefer overall?". Addition- ally, we evaluate the perceived quality the motions using a Likert scale asking participants to rate the motions realism and prompt alignment from 0 (poor) to 5 (excellent).
For each test case, we generated motion using our method, MDM Tevee et al. 2023 FineMoGen (Zhang et al. 2023ch, MotionDiffuse (Zhang et al. 2024b), ReMoDiffuse (Zhang al.
8

---

Published as a conference paper at ICLR 2026
Preferred Ours (%)
Perceived Quality
40-
ReMoDifluse
MDM
ReMaDiffuse MotianCLIP
20
HineMoGeen
MationCLIPP
FineMoGen
MotionDiffuse
aligns generates
you
adheres the text prompt Is natural and realistic.
given text prompt? natural mation?
prefer overall?
User study results. (Left) User study indicating the percentage of users that prefer our method over baselines. (Right) Perceived quality of the generated motions, where 5/0 indicate strong agrremmen/ddisagreemennt with the statement: "The video..."
Figure 4:
"bouncing a ball"
Ours
MDM
Figure 5: Qualitative evaluation. We compare the motions generated by MDM Tevet et al. 2023) and our method for some of the prompts used in the perceptual study. We show two views (front and side) of the generated motions for multiple frames.
2023b), and MotionCLIP (Tevet et al., 2022) presented from two different views: front and side. To mitigate bias, the order of the motions was randomized for each participant. Note that the questions aim to not only evaluate prompt alignment but also the overall quality of the generated motion.
Participants were shown 17 such motion pairs, each
Figure 3 shows that participants prefer our method over the other baselines across all questions. We also observe that our method achieves higher scores the Likert scale for both realism and prompt alignment.
Qualitative Results In figure show comparison of the generated motion using our method our method generates motions that align better with the text prompt and look more realistic. Specifically, by leveraging the wide prior of the VDM, we can generate more diverse and realistic motions.
and MDM (Tevet et al. (202)) for the same motion prompts. We observe that
In figure show additional qualitative results of our method. We generate motions given text prompt and visualize the frames from the front and side views.
4.4 LIMITATIONS AND FUTURE WORK
While video diffusion models (VDMs) show great promise in generating diverse and realistic hu- man motions, they remain susceptible to artifacts such as morphing effects. However, with the rapid advancement of this field, we anticipate that future iterations of these models will address these shortcomings and further enhance motion realism. Monocular tracking remains an inherently un- derconstrained problem, often leading to ambiguities and inaccuracies in the reconstructed motion.

---

Published as a conference paper at ICLR 2026
Video with Landmarks
Animated Mesh
Figure 6: Failure case. Example of a failure case for the action "bouldering" where the generated video suddenly morphs the front of the body into the back. Our method solves this by smoothly rotating the mesh to accommodate the sudden change. At the end, the generated video fails to represent the mesh accurately which can seen in the last column. However, this still does not collapse the final mesh.
To mitigate these challenges, future work could explore the integration of depth predictors Khi
rodkar et al. 2024 leverage multi-view diffusion models (Xie et al., 2024 Zhang al. 2024a
Wu et al." 2025). Our proposed approach opens up exciting possibilities for generating 4D datasets of human motion. These datasets could serve valuable resources for training and benchmarking
models in human motion analysis. Moreover, our method has significant potential for practical appli- cations, such as creating animations for virtual characters in video games, movies, and mixed-reality experiences.
4.4.1 FAILURE CASES
As discussed in section 4.4 our method is limited by the capabilities the underlying video diffu- sion model. This includes generating animations that adhere closely to the provided prompt when the input video does not accurately reflect the desired action. This can lead to tracking failures, especially in scenarios where the video contains morphing effects. We show example in figure where the sudden morphing of the body from front to back becomes 180 degrees rotation in the generated animation.
CONCLUSION
In this work, we propose Animating the Uncaptured, novel text-to-motion generation method for humanoid meshes. By leveraging the strong priors of video diffusion models, our approach gen- erates realistic and diverse human motions, which are then transferred to 3D meshes. We use the SMPL model as deformation proxy, anchoring the vertices of the input mesh to their closest SMPL. face and optimizing the SMPL parameters to track the motion depicted the generated video. This process is guided by extracting 2D body landmarks, silhouette information, and dense semantic fea- tures from the video frames. Experiments on the CAPE dataset demonstrate that our method quan- titatively outperforms baseline approaches, particularly in tracking videos with untextured meshes. Finally, our user study highlights a strong preference for the motions produced by our method, both in terms of realism and alignment with text descriptions.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work was supported by the ERC Consolidator Grant Gen3D (101171131) of Matthias Nießner and the ERC Starting Grant SpatialSem (101076253) of Angela Dai.
REFERENCES
Kingma DP Ba Adam et al. A method for stochastic optimization. arXiv preprint arXiv:1412.6988. 1412(6), 2014.
Fabien Baradel#, Matthieu Armando, Salma Galaaoui, Romain Brégier, Philippe Weinzaepfel, Grégory Rogez, and Thomas Lucas* Multi-hmr: Multi-person whole-body human mesh re- covery single shot. In ECCV, 2024.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominík Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.5112,, 2023.
Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J Black. Keep smpl: Automatic estimation of human pose and shape from single image. In European conference computer vision, pp. 561-578. Springer, 2016.
Z. Cao, G. Hidalgo Martinez, T. Simon, S. Wei, and A. Sheikh. Openpose: Realtime multi-pon multi-person person 2d pose estimation using part affinity fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019.
Hila Chefer, Uriel Singer, Amit Zohar, Yuval Kirstain, Adam Polyak, Yaniv Taigman, Lior Wolf, and Shelly Sheynin. Videojam: Joint appearance-motion representations for enhanced motion generation in video models. arXiv preprint arXiv:2502.02492, 2025.
Luka Crnkovic-Friis and Louise Crnkovic-Friis. Generative choreography using deep learning.
arXiv preprint arXiv: (605506992111 2016.
Xiang Deng. Youxin Pang, Xiaochen Zhao, Chao Xu, Lizhen Wang, Hongjiang Xiao, Shi Yan, Hongwen Zhang, and Yebin Liu. Stereo-talker. Audio-driven 3d human synthesis with prior- guided mixture-of-experts. IEEE Transactions Pattern Analysis and Machine Intelligence, 2025.
Niladri Shekhar Dutt, Sanjeev Muralikrishnan, and Niloy J Mitra. Diffusion 3d features (diff3f): Decorating untextured shapes with distilled semantic features. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4494 4504, 2024.
Satoru Fukayama and Masataka Goto. Music content driven automated choreography with beat-wise motion connectivity constraints. Proceedings of SMC, pp. 177-183, 2015.
Stuart Geman. Statistical methods for tomographic image restoration. Bull. Internat. Statist. Inst., 52:5-21, 1987.
Shubham Goel, Georgios Pavlakos, Jathushan Rajasegaran, Angjoo Kanazawa, and Jitendra Malik. Humans in 4d: Reconstructing and tracking humans with transformers. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 14783-14794, 2023.
Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. Action2motion: Conditioned generation of 3d human motions. In Proceedings of the 28th ACM International Conference on Multimedia, pp. 2021-2029, 2020.
Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang. Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/VV Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5152-5161, June 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.

---

Published as a conference paper at ICLR 2026
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv:2204.03558 2022.
Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3, 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(7):1325-133 1339, 2014. 2013.248.
doi: 10. 109/TAMI.
Yujiao Jiang, Qingmin Liao, Zhaolong Wang. Xiangru Lin, Zongqing Lu, Yuxi Zhao, Hanqing Wei, Jingrui Ye, Yu Zhang, and Zhijing Shao. Smplx-lite A realistic and drivable avatar benchmark with rich geometry and texture annotations. In 2024 IEEE International Conference on Multime- dia and Expo (ICME), pp. 1-6. IEEE, 2024.
Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 7122-7131, 2018.
Rawal Khirodkar, Timur Bagautdinov, Julieta Martinez, Su Zhaoen, Austin James, Peter Selednik, Stuart Anderson, and Shunsuke Saito. Sapiens: Foundation for human vision models. In Euro- pean Conference on Computer Vision, pp. 206-228. Springer, 2024.
Diederik P Kingma and Max Welling.
Auto-encoding variational bayes. arXiv preprint
arXiv:1312.6114 14,661111000 2013.
Nikos Kolotouros, Georgios Pavlakos, Michael Black, and Kostas Danillidis. Learning to Re- construct 3D Human Pose and Shape via Model-Fitting in the Loop. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2252-2261, Seoul, Korea (South), oct 2019, IEEE ISBN 978-1-7281-4803-8.
doi: 10. 10.100///CV222099000344. URL https:
orq org. //ieeexplore.
Hongjie Li, Hong-Xingg Yu, Jiaman Li, and Jiajun Wu. Zerohsi: Zero-shot 4d human-scene interac- tion by video generation. arXiv preprint arXiv:2412.18600, 2024.
Lei Li and Angela Dai. Genzi: Zero-shot 3d human-scene interaction generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20465-20474, 2024.
Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling- Yu Duan, and Alex C. Kot. Ntu rgb+d 120: A large-scale benchmark for 3d human activity understanding. IEEE Transactions Pattern Analysis and Machine Intelligence, 42(10))2684-2701 2020. doi: 10.11099/PMM..019 2916873.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia), 34(6):248:1 148:116
October 2015.
Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, al. Mediapipe: A framework for building perception pipelines. arXiv preprint 1906.: 08/72, 2019.
Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell. Diffusion hyperfeatures: Searching through time and space for semantic correspondence. Advances in Neural Information Processing Systems, 11447500-45114 2023.
Qianli Ma, Jinlong Yang, Anurag Ranjan, Sergi Pujades, Gerard Pons-Moll, Siyu Tang, and Michael J Black. Learning to dress 3d people in generative clothing. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6469-6478, 2020.
Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Black. AMASS: Archive of motion capture as surface shapes. In International Conference on Computer Vision, pp. 5442-5444440 5451, October 2019.
Mixamo. Mixamo: Online 3d character animation platform, 2025. Available at: https: /wwww
i xamo com
12

---

Published as a conference paper at ICLR 2026
John Nickolls, lan Buck, Michael Garland, and Kevin Skadron. Scalable parallel programming with cuda. In ACM SIGGRAPH 2008 Classes, SIGGRAPH '08, New York, NY, USA. 2008. Association for Computing Machinery. ISBN 9781450377451. doi: 11114//1401132.1411115 URL htRl.ttts:///. patttp::/doi..rggee 11451/1001111111111111 32. 100152
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fermandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-War Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Ar- mand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
Joseph O'Rourke and Norman I. Badler. Model-based image analysis of human motion using con- straint propagation. JEEE Transactions on Puttern Analysis and Machine Intelligence, PAMI-2 (6):522-536, 1980. doi: 10. 1199/TPAMI.1880044477000
Ahmed AA Osman, Timo Bolkart, and Michael Black. Star: Sparse trained articulated human body regressor. In European Conference on Computer Vision, pp. 598-613. Springer, 2020.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vetito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W: W, 2017.
Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black. Expressive body capture: 3d hands, face, and body from a single image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10975-10985, 2019.
Leonid Pishchulin, Eldar Insafutdinov, Siyu Tang, Bjoem Andres, Mykhaylo Andriluka, Peter V Gehler, and Bernt Schiele. Deepcut: Joint subset partition and labeling for multi person pose estimation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4929-4937, 2016.
Matthias Plappert, Christian Mandery, and Tamim Asfour. The KIT motion-language dataset. Big Data, 4(4):236-252. dec 2016. doi: 10. 1.108//ii.22016.0222 URL httt:://dx doi org/ 10. 1089/bibb 6..66.060002
Kling AI: Al-Powered Video Generation Platform. Kling ai, 2024. URL httph: .ii
Gerard Pons-Moll, Sergi Pujades, Sonny Hu, and Michael J. Black. ClothCap: seamless 4D cloth- ing capture and retargeting. ACM Transactions on Graphics, 36(4):1-15, aug 2017. ISSN 0730- 0301, 1557-7368. doi: 10. 0155533729599377777
URL https::/t///. acm. wrg..orr//ooo /arg//oo//e 1145/ /00//37229559 31011799999977777771 1073711 294 citations (Crossref/DOO) 12024-10-111 350 citations (Seman- tic Scholar/DOI) [2024-03-18].
Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin John- son, and Georgia Gkioxari. Accelerating 3d deep learning with pytorch3d. arXiv preprint arXiv:2007.08501, 2020.
Yoni Shafir, Guy Tevet, Roy Kapon, and Amit Haim Bermano. Human motion diffusion gener- ative prior. In The Twelfth International Conference on Learning Representations, 2024.
Soyong Shin, Juyong Kim, Eni Halilaj, and Michael J Black. Wham: Reconstructing world- grounded humans with accurate 3d motion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2070-2080, 2024.
Olga Sorkine and Mare Alexa. As-Rigid-AAs-Posibbl Surface Modeling. The Eurographics Associ- doi: 10.2312/SGPPSSGP77/00999 URL https://
ation, 2007. ISBN 978 978-3-905673-46-33
or:: 2014400-299100443110 ISSN: 1827-77788
2ttp://11222222.077000000999 Accepted:
13

---

Published as a conference paper at ICLR 2026
Qingping Sun, Yanjun Wang, Ailing Zeng, Wanqi Yin, Chen Wei, Wenjia Wang, Haiyi Mei, Chi- Sing Leung, Ziwei Liu, Lei Yang. et al. Aios: All-in-one-stage expressive human pose and shape estimation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 1834-1843, 2024.
Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. Advances in Neural Information Processing Systems, 36: 1363-1389, 2023.
Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or, Motionclip: Ex- posing human motion generation to clip space. In European Conference on Computer Vision, pp. 358-374. Springer, 2022.
Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffusion model. In The Eleventh International Conference on Learning Repre- sentations, 2023. URL openreview net
Lukas Uzolas, Elmar Eisemann, and Petr Kellnhofer. Motiondreamer: Exploring semantic video diffusion features for zero-shot 3d mesh animation. In International Conference on 3D Vision 2025, 2025. URLLhttts:://opeeeeeet
net wett/formmm.d==xaannnm
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Timo Von Marcard, Roberto Henschel, Michael J Black, Bodo Rosenhahn, and Gerard Pons-Moll. Recovering accurate 3d human pose in the wild using imus and a moving camera. In Proceedings of the European conference computer vision (ECCV), pp. 601-617, 2018
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang. Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng. Shiwei Zhang, Siyang Sun, Tao Fang, Tianxing Wang. Tianyi Gui, Tingyu Weng. Tong Shen, Wei Lin, Wei Wang. Wei Wang, Wenmeng Zhou, Wente Wang, Wenting Shen, Wenyuan Yu, Xianzhong Shi, Xiaoming Huang, Xin Xu, Yan Kou, Yangyu Lv, Yifei Li, Yijing Liu, Yiming Wang. Yingya Zhang, Yitong Huang, Yong Li, You Wu, Yu Liu, Yulin Pan, Yun Zheng, Yuntao Hong. Yupeng Shi, Yutong Feng, Zeyinzi Jiang, Zhen Han, Zhi-Fan Wu, and Ziyu Liu. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv: 2503.00344. 2025.
Rundi Wu, Ruiqi Gao, Ben Poole. Alex Trevithick, Changxi Zheng, Jonathan T Barron, and Alek- sander Holynski. Cat4d: Create anything in 4d with multi-view video diffusion models. In Pro- ceedings of the Computer Vision and Pattern Recognition Conference, pp. 26057-26068, 2025.
Yiming Xie, Chun-Han Yao, Vikram Voleti, Huaizu Jiang, and Varun Jampani. Sv4d: Dy- namic 3d content generation with multi-framm and multi-view consistency. arXiv preprint arXiv:2407. 17470, 2024.
Hongyi Xu, Eduard Gabriel Bazavan, Andrei Zanfir. William T. Freeman, Rahul Sukthankar, and Cristian Sminchisescu. GHUM & GHUML: Generative 3D Human Shape and Articulated Computer Vision and Pattern Recognition
Pose Models. In 2020 IEEF/CVF Conference (CVPR), pp. 61992 Seattle, WA, USA, jun 2020. IEEE. ISBN 978-1-7281-7168-5 doi:
10.1109/CVPR4260000. 202..0000..200002.....22 9157563/ 241 citations (Semantic Scholar/DOI) [2024-03-17].
peattps://ieexxee w/teeeeeeeeeeeeeeeeeeeeeeeeeeeee leee org dwwpg///ccmme/////////
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Xiaotao Gu, Yuxuan Zhang, Weihan Wang, Yean Cheng, Ting Liu, Bin Xu, Yuxiao Dong, and Jie Tang. Cog Videox: Text-to- Video Diffusion Models with An Expert Transformer, aug 2024. URL
06072 LrXiv:2408.0601 [cs].
Kwan Yun, Seokhyeon Hong, Chaclin Kim, and Junyong Noh. Anymole: Any character motion in- intbeewenning leveraging video diffusion models. arXiv preprint arXiv:2503.08417 2025.
14

---

Published as a conference paper at ICLR 2026
Haiyu Zhang, Xinyuan Chen, Yaohui Wang, Xihui Liu, Yunhong Wang, and Yu Qiao. 4diffusion: Multi-view video diffusion model for 4d generation. Advances in Neural Information Processing Systems, 37:15272-15295 2024a.
Junyi Zhang, Charles Hermaan, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. Advances in Neural Information Processing 3 36:45333-455777, 2023a.
Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, and Ziwei Liu. Remodiffuse: Retrieval-augmente motion diffusion model. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 364-373, 2023b.
Mingyuan Zhang, Huirong Li, Zhongang Cai, Jiawei Ren, Lei Yang, and Ziwei Liu. Finemogen: Fine- grainer spatio-temporal motion generation and editing. Advances in Neural Information Processing Systems, 36: 13991-13999) 2023e.
Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. IEEE transac- lions pattern analysis and machine intelligence, (166):4115-4125 2024b.
Shihao Zou, Xinxin Zuo, Yiming Qian, Sen Wang, Chi Xu, Minglun Gong, and Cheng. 3d human shape reconstruction from a polarization image. In Proceedings of the European Conference Computer Vision (ECCV), 2020.
A SUPPLEMENTARY MATERIAL
We provide additional details about our method and results in this appendix. Additionally, we in- clude a video in the supplementary material and an anonymous link to the source code: https:
//anonynous 4open, sci
A.1 ABLATIONS
Table 2: Ablation study on the effect of various components on performance. Metrics include MPJPE (Mean Per Joint Position Error), PVE (Per Vertex Error), and Accel (Acceleration Error). Lower better for all metries.
Method
MPJPE
PVE
Accel
w/o Lo
0.0486
0.0556
1.7121
w/o Lex
0.0393
0.0453
1.5699
w/o Lo
0.0392
0.0447
1.5895
w/o Liemp
0.0403
0.0458
3.2005
Opt. Parameters
0.0453
0.0533
2.6494
Ours
0.0362
0.0411
1.4981
We conduct different ablation studies to evaluate the impact of our various design choices. To evalu- ate setting where ground truth is available, we run the same evaluation as for tracking (section 4.2) and show ablation results table We observe that the temporal loss Limmp clearly improves per- formance, particularly in acceleration error. Opt. Parameters refers to directly optimizing the SMPL parameters instead of using an MLP to predict them. This confirms that by using a neural param- eterization, take advantage of the inductive bias of the MLP improve the tracking per- formance, particularly, the smoothness. Les ben. and Co allow us to leverage the body prior, which helps maintain anatomically plausible poses.
15

---

Published as a conference paper at ICLR 2026
Figure 7: Example of a video generated with Cog VideoX YYagg et al. 2024 conditioned on the rendering of mesh and the prompt "The person is running".
A.2 FULL PROMPT
We provide the prompt used to generate our results, where we replace the ACTION placeholder with the action want to generate. The prompt emphasizes dynamic motions, human body realism, static camera and bright lighting. These are to avoid static videos, body morphing, camera zooms and dark lighting respectively. The prompt is follows:
"An award winning documentary about person ACTION. Energetically ACTION. Dynamic move- ment. Light grey person. Realistic movement. Realistic motion. Realistic human body. Wide angle shot showcasing the man, dynamic movement, this video incredibly detailed and high resolution, the uniform light is impressive, masterpiece. Clear illumination. Bright light. No dark light. Fixed camera. No zoom in."
A.3 PERCEPTUAL STUDY
total of users participated in the study and gave consent for their data to be used for research purposes. We include screenshot of the user study interface in figure 9
A.3.1 LIST OF PROMPTS
We list the prompts used in the perceptual study:
"The person is taking cover and shooting"
"The person is practicing with nunchaku" "
"The person is playing tennis"
"The person is practicing karate moves"
"The person is taking cover"
"The person is adjusting her glasses" "
"The person is stopping the traffic"
"The person is digging hole with shovel"
"The person is dancing disco fox"
"The person is opening door"
"The person is tying her shoes"
"The person is brushing her hair"
"The person is mining with a pickaxe"
"The person is smoking"
16

---

Published as a conference paper at ICLR 2026
"playing tennis"
Video
Ours
Figure 8: Example of video generated with Wan2.2 (Wan al. 2025) and the tracked mesh.
"The person is getting ready to fight"
"The person is bouncing ball"
"The person is practicing kickboxing" "
A.4 JUSTIFICATION OF THE CHOICE OF VDMS
The results of our method are generated using Kling Al (Platform) 2024). a closed-source model. While we believe that academic research should publish code and models ensure reproducibility. we chose to use this model due to its superior performance on our task. However, we want to emphasize that our method agnostic the choice of VDM and be used with any model that generates videos from text prompts, therefore, with the fast pace of research in this area, we expect that our method can be easily adapted to future models such VideolAM (Chefer show an example of the generated video obtained from Cog VideoX (Yang et al. 2024) in figure 7
2025). We
A.4.1 WAN2.2
With the recent release of Wan2.2 (Wan et al. 2025), open-source model. We provide an example of the generated video and tracked mesh in figure B
able
method with this
We used similar environment as in section 4.1 with the exception of using an NVIDIA A.00 GPU with 80GB of memory.
A.5 ETHICAL CONSIDERATIONS
We present method for generating mesh animations from text prompts by leveraging video diffu- sion models. As result, our approach inherits the ethical considerations associated with these mod- els, particularly the potential for generating deepfakes of people performing actions. However, since our method requires 3D mesh as input- which costly and challenging to obtain for arbitrary individuals- we do not anticipate its for creating deepfakes. Nevertheless, acknowledge the importance of considering potential misuse and emphasize the need to raise awareness of the broader ethical implications.
A.6 ADDITIONAL RESULTS
A.6.1 VIDEO TRACKING
We provide additional results of the video tracking on the Mixamo (Mixamo) 2025) dataset, which contains variety of characters and motions for rigged humanoid meshes. For this,
used 50
17

---

Published as a conference paper at ICLR 2026
Table 3: Ver-Vertex-Enne Error (PVE) riggable untextured sequences from Mixamo (Mixamo 2025
dataset for WHAM (Shin et al. 2024. SMPLIFY-X (Pavlakos et al. 2019) and our method.
Method
PVE
SMPLIFY-X
0.099
WHAM
0.066
Ours
0.058
randomly selected animations from the dataset, rendered them as videos, and tracked them using pipeline. We report the average per-vertex-error (PVE) in table The results demonstrate the effectiveness of our video tracking method on diverse characters and motions.
A.6.2 QUALITATIVE RESULTS
provide additional results of our method in figures 10 and additional comparisons against
MDM in figure 12
Animating the Uncaptured User Study
(Total: 2/17)
Text prompt: The person is practicing with nunchaku.
Restart (Synd) Videos
Method view)
Method (side view)
Which method best with the given text prompt?
Figure 9: Perceptual Study Visualization. A screenshot the user study interface. The is presented with a text prompt and two methods to compare. Each method has two videos, one from the front and one from the side. The user can play the videos and compare them side by side, At the bottom of the page, the user has answer set of questions.
18

---

Published as a conference paper at ICLR 2026
"dancing flamenco"
"doing side lunges"
"playing hopseotch"
"shooting with a bow"
"protecting his eyes from the sun"
Figure 10: Additional qualitative results. Visualization of generated mesh animations with our method. Each row shows: the prompt, the input mesh and the generated mesh animations. For all generations, we visualize the mesh from the front and side views.
19

---

Published as a conference paper at ICLR 2026
"laughing hysterically"
"exercising with battle ropes"
"energetically dancing"
"dancing waltz"
"riding horse"
Figure 11: Additional qualitative results. Visualization of generated mesh animations with our method. Each row shows: the prompt, the input mesh and the generated mesh animations. For all generations, we visualize the mesh from the front and side views.
20

---

Published as a conference paper at ICLR 2026
"practicing with nunchaku"
Oun
MDNA
"opening a door"
Ours
MDM
"getting ready to fight"
Ours
MDM
Figure 12: Additional comparisons with MDM (Tevet et al., 2023). We show two views (front
and side) of the generated motions for multiple frames.
21


Reviews:
[{"review_id": "X1d9pEYjdP", "reviewer": "Reviewer_eAnh", "summary": "This paper presents Animating the Uncaptured, a method for animating 3D humanoid meshes from text prompts. Given an input mesh and textual description, the approach first generates motion videos via a text-to-video (T2V) diffusion model, then leverages the SMPL parameterized human model as a deformation proxy to track and reconstruct character motion from the generated video, which is subsequently transferred back to the 3D mesh.\nTo enhance reconstruction quality, the authors integrate multiple cues of body keypoints, silhouettes, and dense DINOv2 features as optimization constraints. Experiments on the CAPE dataset show that the method outperforms SMPLify-X, WHAM, and Multi-HMR baselines. A user study further indicates that the generated animations achieve higher perceived realism and better text-motion consistency.", "questions": "1. Limited quantitative evaluation:\nThe CAPE dataset provides only narrow evaluation scenarios on untextured meshes. \nWhy not assess performance on broader human pose and shape benchmarks such as 3DPW, RICH, or EMDB?  Additionally, would textured meshes influence generation quality or tracking efficiency?\n\n2. Tracking accuracy:\nHas the tracking accuracy been quantitatively evaluated on editable or rigged humanoid meshes to validate applicability in real animation pipelines?\n\n3. Use of SMPL motion generation:\nSince the method already performs mesh-to-SMPL registration, why not leverage existing SMPL-based motion generation techniques to animate the mesh directly, instead of relying on text-to-video tracking? This would seem to be a more straightforward way to drive the mesh using well-established motion priors.\n\n4. Failure cases: \nWhat are the observed failure scenarios?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Novelty: \nThe paper explores a promising direction by introducing a generalized motion prior from large-scale video diffusion models (VDMs) to animate static 3D meshes. The proposed “generation–tracking–deformation” pipeline bridges generative video synthesis and 3D motion reconstruction, leveraging the strong expressive capacity of modern VDMs. This cross-domain integration offers an extensible framework for text-driven 3D animation.\n\n2. Experimental Evidence: \nThe paper provides abundant qualitative examples, and quantitative results in Table 1 demonstrate clear improvement on CAPE sequences, particularly in the Accel metric, compared with existing registration and reconstruction baselines such as SMPLify-X, WHAM, and Multi-HMR.", "weaknesses": "1. Unclear motivation: \nWhile the paper claims to focus on animating humanoid meshes, the methods and experiments seem more centered on registration and tracking from images or videos.\nThe distinction between animation generation and motion fitting is not clearly articulated, making the actual novelty somewhat ambiguous.\n\n2. Method clarity: Several parts of the method are under-explained.\n- line 159 mentions \"we use the encoding $Z \\in R^32$ of the variational autoencoder VPoser\", \nbut no further elaboration is given. \n- lines 256–269 are vague: it is unclear whether $v_i^SMPL$ refers to the corresponding face or vertex on the SMPL template.\nThe function $\\Psi$ is introduced without a precise definition or computational description.\nalso lacks a clear definition or computational description. \n- Equation (3) refers to L1, but the written form corresponds to an L2 prior, indicating inconsistency between text and formulation.\n\n3. Resource requirements: \nThe paper mentions 1,000 iterations for registration and over 4,000 for video tracking, yet omits device type, runtime, or memory requirements. Without such details, the practical feasibility and scalability of the method remain unclear.\n\n4. Lack of ablation on MLP optimization: \nThe implicit MLP for temporal modeling appears to be a design simplification rather than a fundamental requirement. Since the MLP is optimized per sequence and does not generalize across meshes, it limits both efficiency and scalability. A shared temporal model (e.g., RNN, Transformer, or motion prior) might offer better generalization and faster inference.\n\n5. Strong Dependence on the Video Diffusion Model: \nThe presented animations rely heavily on the pretrained Kling AI VDM. The authors neither fine-tune the VDM for animation-related content nor test robustness across different video generators. This raises concerns about reproducibility and generalization to varied video outputs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3jP7oKPItU", "reviewer": "Reviewer_FtqR", "summary": "This paper presents a novel method to animate a static 3D humanoid mesh from a text prompt, addressing the limitations of costly MoCap datasets by instead leveraging the rich, generalized motion priors from large-scale Video Diffusion Models (VDMs). The pipeline first generates a video by conditioning a VDM on a rendered image of the mesh and the text prompt. Then, it performs a robust motion transfer by first registering a SMPL body model to the input mesh as a \"deformation proxy\" and re-parameterizing the mesh vertices. A tracker then optimizes the time-varying SMPL parameters to match the generated video, guided by a combination of sparse 2D landmarks, dense silhouettes, and DINOv2 features. To ensure temporal smoothness, these motion parameters are predicted by shallow MLPs, and experiments show this approach outperforms baselines in tracking and is significantly preferred by users over traditional MoCap-based methods for realism and prompt alignment.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper's primary strength is its hypothesis that generative video models, trained on massive, diverse, \"in-the-wild\" video data, contain superior and more generalizable motion priors than the small, clean MoCap datasets currently used by most text-to-motion methods. The strong user study results (Fig. 4) convincingly validate this hypothesis.\n- The method for transferring 2D video motion to the 3D mesh is very well-designed. It correctly identifies that regression-based pose estimators (like HMR) would fail on synthetic VDM-generated videos, and thus wisely opts for a more robust optimization-based tracking approach. The use of multiple, complementary tracking cues (sparse landmarks, dense silhouettes, and semantic DINOv2 features) provides strong guidance for the optimization.\n- The paper does an excellent job of evaluating its claims. The authors wisely isolate and evaluate their tracking component on a controlled task (recovering GT motion from the CAPE dataset). The results show it is not only more accurate (lower MPJPE/PVE) but significantly smoother (much lower \"Accel\" error) than strong baselines. The perceptual user study is the main payoff. The fact that users overwhelmingly preferred this method's animations to those from a strong MoCap-based model (MDM) on realism, prompt alignment, and overall quality is a very strong result.\n- While the method combines several existing tools (VDMs, SMPL, DINO), it does so in a novel pipeline that solves a practical problem. The \"SMPL-as-proxy-rig\" approach makes the method applicable to a wide range of static humanoid meshes that lack their own skeletons or rigs, which is a common use case.", "weaknesses": "- The method is a \"Garbage In, Garbage Out\" system that places full trust in the VDM's output. The paper acknowledges that VDMs can produce artifacts or \"morphing effects,\" but does not fully address how the tracker would handle them. If the VDM generates a physically impossible motion, a distorted body part, or a character that morphs into the background, the optimization-based tracker will likely fail or produce an equally nonsensical 3D animation. (I am personally a believer of VDM as world simulators, so hopefully this will be less of an issue over time).\n- The tracker works from a single 2D video, which is an inherently ill-posed 2D-to-3D problem. While the SMPL/VPoser prior helps, the method is still susceptible to ambiguities in depth and self-occlusion. The (very specific) prompt engineering in Appendix A (e.g., \"Wide angle shot,\" \"Fixed camera,\" \"No zoom in\") suggests that the VDM output must be carefully constrained to be \"trackable,\" which limits the range of dynamic camera motions that can be animated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JF6nYgQ5df", "reviewer": "Reviewer_xL5i", "summary": "This paper described a method to generate 3D humanoid motion using 2D video generation models. Video models have seen significantly more data compared to motion generation models (that are restricted to motion-capture data typically), and hence seem to be less expressive. The method is well engineered, with individual steps that make perfect sense, and produces convincing results.", "questions": "- How do you not get motions that are too flat in terms of depth?\n- Why use VPoser, which is a rather old prior instead of newer ones?\n- Why not use the texture of the mesh as well, wouldn't that help the video model to be more expressive?", "limitations": "", "rating": 10, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- excellent results.\n- clear and reasonable method.\n- some nice steps, such as the combined modalities of the tracking.\n- seems to surpass the SOTA even when compared to dedicated motion generation models.", "weaknesses": "- Applies to humanoid characters only.\n- Depth is not explicitly addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YBu5VKbTCC", "reviewer": "Reviewer_xUv5", "summary": "This paper proposes a new method for animating static 3D humanoid meshes using video diffusion models. Instead of relying on costly motion-capture datasets, proposed approach leverages motion priors learned by large-scale text-to-video diffusion models, which inherently capture diverse human movements. Given a 3D mesh and a text prompt, the model generates a synthetic video of the mesh performing the motion, then reconstructs 3D motion by fitting and optimizing a SMPL body model to track motion cues such as 2D landmarks, silhouettes, and dense DINOv2 features. The optimized SMPL parameters are transferred to the mesh. Experiments on the CAPE dataset show that this method outperforms existing baselines in motion tracking accuracy and smoothness, and a perceptual study confirms that users find the generated animations more realistic and better aligned with textual descriptions.", "questions": "Why are untextured meshes rendered for video diffusion model inference? It seems that leveraging video diffusion models with textured meshes could yield better performance due to a smaller domain gap with the models’ training distribution. This untextured setting could also effect largely to the pose fitting performance.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This introduces a simple yet effective framework that leverages the motion priors of existing video diffusion models to animate static 3D humanoid meshes without relying on expensive motion-capture data. Its integration of generative video priors with SMPL-based optimization enables realistic, temporally coherent, and diverse motion synthesis from simple text prompts. This combination, including feature-based optimization, registration, and reparameterization, makes the approach both scalable and generalizable, offering a practical and accessible solution for creating 4D humanoid animations. Also, its performance on pose fitting outperforms the previous works.", "weaknesses": "A main concern is that its novelty and contribution are somewhat limited, as the overall concept of using video diffusion models for motion generation has already been explored in prior works such as MotionDreamer [1] and AnyMoLe [2]. Similar to MotionDreamer and AnyMoLe, the proposed approach extracts dense features and generated videos to guide motion reconstruction. Although this paper integrates these components into a clean and unified framework for humanoid, it primarily extends existing ideas rather than introducing a fundamentally new mechanism for motion extraction or representation. Furthermore, while the concepts overlap with [2], this work is not referenced in the paper.\n\n\n[1] MotionDreamer: Exploring Semantic Video Diffusion Features for Zero‑Shot 3D Mesh Animation. Uzolas et al., 3DV 2025\n\n[2] AnyMoLe: Any Character Motion In‑betweening Leveraging Video Diffusion Models. Yun et al., CVPR 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)