How to Move Your Dragon: Text-to-Motion Synthesis for Large-Vocabulary Objects
Authors: Wonkwang Lee
Jongwon Jeong
Taehong Moon
Hyeon-Jong Kim
Jaehyeon Kim
Gunhee Kim
Byeong-Uk Lee
Venue: ICML 2025 poster (2025)
Keywords: motion synthesis for diverse objects
motion synthesis for large-vocabulary objects
text-to-motion synthesis
animal motion synthesis
non-human motion synthesis
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: In this work, we address text-to-motion synthesis for a wide range of object categories, including bipedal, quadrupedal, and winged animals, as well as fish and insects.

Abstract: Motion synthesis for diverse object categories holds great potential for 3D content creation but remains underexplored due to two key challenges: (1) the lack of comprehensive motion datasets that include a wide range of high-quality motions and annotations, and (2) the absence of methods capable of handling heterogeneous skeletal templates from diverse objects.
To address these challenges, we contribute the following:
First, we augment the Truebones Zoo dataset—a high-quality animal motion dataset covering over 70 species—by annotating it with detailed text descriptions, making it suitable for text-based motion synthesis.
Second, we introduce rig augmentation techniques that generate diverse motion data while preserving consistent dynamics, enabling models to adapt to various skeletal configurations.
Finally, we redesign existing motion diffusion models to dynamically adapt to arbitrary skeletal templates, enabling motion synthesis for a diverse range of objects with varying structures.
Experiments show that our method learns to generate high-fidelity motions from textual descriptions for diverse and even unseen objects, setting a strong foundation for motion synthesis across diverse object categories and skeletal templates.
Qualitative results are available on this [link](https://t2m4lvo.github.io).

How to Move Your Dragon: Text-to-Motion Synthesis for Large- Vocabulay Objects
Wonkwang Lee" 12 Jongwon Jeong *2 Tachong Moon +2
Hyeon-Jong Kim Jaehyeon Kim f3 Gunhee Kim Byeong-UL UK Lee²
Abstract
to create realistic 3D motions for such a diverse array of entities has long been a shared aspiration across creative communities and industries, unlocking profound potential for animation, gaming, virtual reality, and beyond. However, achieving such realism has traditionally required significant manual effort and expertise, making the process both labor- intensive and time-consuming (Baran & Popović, 2007).
Motion synthesis for diverse object categories holds great potential for 3D content creation but remains underexplored due to two key challenges: (I) the lack of comprehensive motion datasets that include a wide range of high-quality motions and annotations, and (2) the absence of methods capa- ble of handling heterogeneous skeletal templates from diverse objects. To address these challenges, we contribute the following: First, we augment the Truebones Zoo dataset- a high-quality animal motion dataset covering over 70 species-by by an- notating it with detailed text descriptions, making it suitable for text- oxt-bbsed motion synthesis. Sec- ond, we introduce rig augmentation techniques that generate diverse motion data while preserv- ing consistent dynamics, enabling models to adapt to various skeletal configurations. Finally, we redesign existing motion diffusion models to dy- namically adapt to arbitrary skeletal templates, en- abling motion synthesis for diverse range of ob- jects with varying structures. Experiments show that our method learns to generate high-fider -ideliiyl motions from textual descriptions for diverse and even unseen objects, setting a strong foundation for motion synthesis across diverse object cate- gories and skeletal templates. Qualitative results are available this link.
Despite advancements in data-driven motion synthesis, two key challenges hinder progress in generating motions for diverse objects. First, the lack of high-quality motion datasets (Kapon al., 2024) spanning broad spectrum of objects with diverse skeletal structures and rich annotations limits the opportunity explore text-driven motion synthe- sis for large-vocabulary objects. Second, existing methods rely heavily on fixed skeletal rig templates, such as those designed for humans (Tevet et al., 2023; Guo et al., 2024), making them ill-suited for handling the heterogeneous skele- configurations found in real-world 3D motion data in a unified manner. From the quadrupedal stance of horse to the winged anatomy of a bird the fantastical form of dragon, synthesizing coherent and realistic motions across such diverse skeletal structures remains an open challenge.
To address these challenges, we propose novel framework for text-to-motion synthesis that enables motion generation across arbitrary skeletal structures, accommodating the di- versity of real-world object motion data.
Specifically, make the following contributions:
1. Introduction
To the best of our knowledge, this is the first work to tackle text-driven motion syn- thesis for a broad range of objects with significantly different skeletal structures within a unified framework.
Novel Problem Setup:
Imagine a world where creating lifelike motions for any crea- ture or object is effortless as describing it in words-a world where the majestic flight of dragon or the intri- cate crawl of centipede can be seamlessly brought to life with the power of intuitive text-based synthesis. The ability
High-Quality Text Descriptions: We curate com- prehensive set of human-labeled descriptions for True- bones Zoo dataset (Truebones, 2022), covering mo- tions over species with unique skeletal templates.
Equal contribution. "Work done KRAFTON. 'Seoul National University * KRAFTOON 'VVIDIA. Correspondence to: Bycong- Uk Lee <lview94@ gmail.com>>
Rig Augmentation: tation methods adjjstn-adjjssing bone lengths/numbers and rest poses to enhance the model's adaptability and generalization diverse skeletal templates.
We introduce novel rig augmen-
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

How to Move Your Dragon: Text-to-Motion Synthesis for Large-Vocabulary Objects
Wyvern attacks by diving down from the air.
A Tyrannosaurus struck and falls to the ground.
An elephant performs a headbutt attack.
A shark rotates its body 180 degrees to the right.
Figure 1. In this work, we address the task of synthesizing high-fidelity 3D motions for broad spectrum of objects with diverse skeletal structures, characterized by varying joint counts and differing dependencies, including animals, dinosaurs, and imaginary creatures, based textual descriptions. The synthesized motions can be applied rigged meshes generate visually appealing 3D animations.
Generalized Motion Diffusion Models: We extend motion diffusion models (Tevet et al., 2023), originally constrained to a single fixed skeletal template, by in- corporating tree positional encoding (TreePE) (Shiv & Quirk, 2019) and rest pose encoding (RestPE). allow- ing dynamic adaptation diverse joint hierarchies and skeletal structures.
skeleton-agnostic designs for humans and anthropomorphic biped characters (Zhang et al., 2024). Despite successes, most approaches either rely on fixed skeletal templates or exclusively focus on human or anthropomorphic motion, limiting their applicability to more diverse objects with het- erogeneous anatomical structures and motion patterns.
2.2. Non-Human Motion Synthesis
Extensive experiments on Truebones Zoo dataset demon- strate our framework's 's ability to generate high-fidelity mo- tions conditioned on textual descriptions, or even synthesize motions for novel objects downloaded from the web.
Non-human motion synthesis has received comparatively less attention, though recent efforts have started addressing this gap. SinMDM (Raab al., 2024) identified internal motion motifs within an arbitrary object motion, enabling the generation of diverse, flexible-length motions that pre- serve structural consistency. OmniMotton-GGP (Yang et al., 2024) leveraged human motion data to synthesize quadruped animal motions, demonstrating the transferability of human motion to non-human domains. Similarly, Zhao al. (2024) introduced learning-based retargeting method capable of adapting dog motions to a T-rex, horse, and hamster, though it requires training separate CycleGANs (Zhu et al., 2017) for each source-target pair. MAS (Kapon et al., 2024) ex- tended a 2D motion diffusion model with a 2D-to-3D lifting mechanism to synthesize 3D equine motions. However, all these approaches rely on fixed skeletal templates or object- specific components and designs, limiting their adaptability to arbitrary structures across large vocabulary of objects.
To inspire future work and further advancements, we will release the code for our data and model pipelines, along with the annotated captions, establishing comprehensive benchmark for motion synthesis across diverse objects with heterogeneous skeletal structures.
2. Related Work
2.1. Human Motion Synthesis
Most research in motion synthesis has focused on human motion with fixed skeletal template, forming the foun- dation of progress in this field. Early work emphasized text-driven motion synthesis (Plappert et al., 2016; Guo et al., 2022), generating human motion from textual descrip- tions. Recent work introduced fine-grained control (Zhang et al., 2023) for specific body parts, open-vocabeleeee vocabulary motion generation (Liang et al., 2024) for arbitrary text inputs, and
In contrast, CharacterMixer (Zhan al., 2024) and Skin- Mixer (Nuvoli et al., 2022) propose rig-blending techniques
2

---

How to Move Your Dragon: Text-to-Motion Synthesis for Large-Vocabuule Vocabulary Objects
Bone Length Aug.
Bone Number Aug.
[Short Description]:
P
A Tyrannosaurus swings its tail in a powerful strike.
[Mid Description]:
F=8
A Tyronnosaurus twists its massive body to the right and swings its powerful tail forward to the left, delivering a fierce strike.
Rest Pose Aug.
Combined & Retargeted
F=50 F=17
[Long Description::
A Tyrannosaurus, initially standing still, begins to twist its massive body to the right. As it completes the twist, it swings its powerful tail forward to the left, delivering a fierce and impactful strike.
Figure Left:
Given original skeleton template (white), the rig augmentation technique adjusts bone length (cyan), quantity (pink), rest poses (orange), or their combinations. The resulting skeleton template is then retargeted (green) the original poses (dark) each frame to generate data that maintains the original motion dynamics while incorporating diverse skeletal templates. Right: Our annotated captions accurately capture motion, delivering different levels of detail through short, mid, and long descriptions. Details added gradually, ranging from high-level actions (blue) to part-level dynamics (green) and initial postures (red).
that create hybrid characters and their motions constructed from multiple heterogeneous skeletons. While these meth- ods demonstrate the potential to generalize motion across structurally diverse rigs, their reliance motion retargeting limits flexibility in generating novel or diverse motions.
due to anatomical differences across objects and the rigging styles of different experts, leading variations in the num- ber of joints and rest poses. Second, even identical motion dynamics result in different joint rotations (R) depend- ing the underlying static features. Thus, the ability to comprehend and generalize across diverse skeletal config- urations crucial for accurately modeling motion while preserving consistency across different structures.
3. Background: Hierarchical Skeletal Rig
hierarchical skeletal rig forms the foundation for repre- senting 3D motion. It encodes object's motion as se- quence of 3D poses P glohal Fxx33, where F is the number of frames and J represents the number of joints. These 3D poses are reconstructed by integrating static fea- tures, which define the skeleton's topology and configura- tion, with dynamic features, which capture the temporal aspects of motion across frames.
4. Large-Vocal Text-to-Motion Dataset
vocgb-vocybulary object motion synthesis is challenging due to the diversity of objects and skeletal configurations. In this section, we discuss and address three key challenges for effective text-to-motion synthesis. First, it necessitates high-quality motion data that captures a variety of objects and motion patterns. Second, this motion data must be represented across diverse skeletal templates to account for real-worke world scenarios. Third, rich annotations with detailed text descriptions are essential to ensure that synthesized motions align accurately with the conditioned input text.
Static Features include tree hierarchy of J joints (8) that represents skeletal topology, and the rest pose (Pres E Rx33) that encodes joint offsets relative to their parents. Each joint j is associated with a parent joint P(j).
Dynamic Features describe motion via a sequence of joint translations (p R. xx33 and rotations (R IR F)) where is the number of frames and D the dimension of the rotation. p(f, j] and [f] define the local translation and rotation of joint frame ff relative to its parent. In this work, we focus on simplified setting that considers only the temporal evolution of rotations, R, as dynamic features.
4.1. High-quality Motion Data
To address the first challenge, we utilize the Truebones Zoo dataset (Trucbones, 2022), which contains over 1,000 artist-created animated armature meshes in FBX format, as illustrated in Figure 1. The dataset spans 70 unique animal species, including mammals, reptiles, birds, fishes, insects, and dinosaurs. Each species is uniquely characterized by its skeletal topology and rest pose, capturing the diversity of anatomical structures. Also, each animal object is associated
also important to note that working with skeletal rig representations of real-world 3D motions presents two key challenges. First, skeletal configurations vary significantly
3

---

How to Move Your Dragon: Text-to-Motion Synthesis for Harge-Vocabulary Objects
Skeletal Structure (S, P p
Tree Position Encoding
Linear Projection
Hips :10, 0,0,
Hips (Root)
Spine 1: [0, 1, 0, 0,
[0S1 4851-475 801s
Spine 2: [0, 0, 0,
Transformer Encoder
Spine
MLP
R.Clavicle : [0, 0, 1, 0,0,0,
Encoding
Spine
TreePE & RestPE
(S,
Hips [0.00, 0.00, 0.00)
11.90 239
Spine 1: [00551, 4.75, 0.01]
Encoder
Text
Linear Projaction
Spine 2: [00448 7,51, 0.02]
R.Clavicke
Text Prompe
R.Clavicle : [1.90, 4.16, 2.19]
Figure 3. Generalized Motion Diffusion Model. Left: Our model conditions skeletal information (S, Pres) to adapt to diverse skeletal templates and a textual prompt C for text-driven synthesis. At each denoising step. the textual prompt C' is encoded using the SigLip (Zhai al., 2023) text encoder and the denoising timestep is embedded form conditioning token zya that guides motion synthesis. Right: The topological (S) and rest pose information (Pies) are encoded by Tree Positional Encoding (TreePE) (Shiv & Quirk, 2019) and Rest Pose Encoding (RestPE) and added to the noisy latent motion x. (f.3) for the f-th frame and j-th joint. The enhanced representations, along with the conditioning token, are then processed by transformer blocks to estimate clean motions (#6 (=(f-)) 1.35
with multiple animations, ranging from to 70 sequences.
skeletal rig by changing the root orientations or joint angles in the rest pose. To determine a new rest pose, we randomly select frame from the motion sequence and set the pose that frame as the new rest pose.
From these FBX files, we extract armature animation data, capturing both the static structure and dynamic features, which are then preprocessed to construct our dataset. For preprocessing, we standardize each skeleton's 's rest pose by aligning its forward direction to the Y-axis and the upward direction to the Z-axis, and normalizing its scale and center to fit within a unit-length cube centered at the origin.
4.3. High-Quality Multi-Level Text Annotation
High-quality text annotations are essential for ensuring pre- cise alignment between textual descriptions and motion data. For this, we manually annotate the motion data in the True- bones Zoo dataset with detailed, high-quality descriptions. These annotations are delivered in a multi-level format, pro- viding varying degrees of detail to address different levels of abstraction and precision required by the model.
4.2. Rig Augmentation
Moreover, we introduce three rig augmentation techniques to simulate various skeletal rig configurations found in real- world motion data, as illustrated Figure 2. These tech- niques, Bone Length Augmentation, Joint Number Aug- mentation, and Rest Pose Resetting, expand the diversity of skeletal configurations in the Truebones Zoo dataset while preserving the original motion dynamics.
Examples of these multi-level descriptions are displayed in Figure 2. To elaborate, the captions are structured into three levels of detail, short, mid, and long, progressively increas- ing in the amount of information they provide about the motion. Short captions deliver only the high-level actions, while details on initial postures, and part-level or directional dynamics are gradually added in mid and long captions.
Bone Length Augmentation randomly adjusts joint-to- joint bone lengths to simulate anatomical variations. To ensure natural motion after the adjustments, we leverage GPT-4o to categorize joints into specific body parts based on hierarchical information, such joint names and parent- child relationships. Furthermore, GPT-4o determines ap- propriate scaling ratios for each part, ensuring the adjusted skeleton resembles anatomically similar objects (c.g., ad- justing dog's bone lengths to match those of a wolf). Each part is then scaled accordingly, with proportional and sym- metrical adjustments applied left and right counterparts to maintain anatomical balance and realistic motion dynamics.
For interested readers, detailed statistics the dataset are provided in the appendix. These include the full list of animal categories in Truebones Zoo dataset (Table 3), word count histograms for each description level (Figure 10), and statistics on the most frequent verbs and nouns used in the annotations (Figure 11 and Figure 12, respectively).
5. Generalized Motion Diffusion Model
We introduce generalized motion diffusion model that ex- tends single-object motion synthesis to diverse objects with arbitrary skeletal templates. Figure provides an overview of our framework. Unlike prior works (Tevet et al., 2023; Raab al., 2024), which assume a fixed skeleton topology, our key innovation is the explicit incorporation of skeletal
Joint Number Augmentation modifies the skeletal rig by altering its structural complexity and organization. To ad- just skeletal granularity, joints are randomly removed or subdivided, creating simpler or more detailed rigs.
Rest Pose Resetting alters the default configuration of the

---

How to Move Your Dragon: Text-to-Motion Synthesis for Large-Vocabulary Objects
configuration information through Tree Positional Encoding (TreePE) and Rest Pose Encoding (RestPE), enabling mo- tion generation across varying skeleton templates. We first present the background on single-object motion diffusion models before detailing our generalization extensions.
is modified to independently transform D-dimensional per- joint representations into D'-dimensional tokens:
(f.) -iinerProo(x? E ERD.
2t
(5)
(fd) represents the input for frame f and joint j. Here, zt
5.1. Motion Diffusion Models
To further encode the topological dependencies among joints, employ Tree Positional Encoding (TreePE) (Shiv & Quirk, 2019). For each joint, TreePE captures both abso- lute position and relative dependencies within the tree-like skeletal structure S by encoding its path from the root as a binary-like sequence of transformations that track parent- child relationships and preserve hierarchical information. To integrate this information, we apply simple MLP layer to each joint's tree encoding, then add the resulting representa- tion TreePE(j) E IR" to the corresponding token, ensuring the hierarchical structure is embedded into the model.
Diffusion models (Song & Ermon, 2020; Ho et al., 2020)
learn to generate realistic samples ze by progressively re-
versing a noising process that begins with random noise æT. At each intermediate step i, the data is represented as dit, gradually refined as the neural network Eg denoises it step by step, which is trained to predict and remove the noise at each stage. To further control the denoising process with conditioning variables C, classifier-fres free guidance (CFG) (Ho & Salimans, 2021) is often utilized.
In the motion synthesis literature, fixed skeleton topology S with constant rest pose Prea is typically assumed, focus- ing solely on synthesizing the dynamic features of motion for a single object. The task is then simplified to generat- ing the rotation sequence Tn = R E FxJDD from noisy latent variables Ty, given an additional input condition C.
(1-5) + PE(f) + TreePE(j). (f 2
(6)
5.3. Handling Arbitrary Rest Pose
In hicrarchical skeletal rig system, the dynamic features of motions are determined not only by the skeletal topology but also by the specific rest pose of each object. To ensure accu- rate modeling of motion dynamics, our framework incorpo- rates explicit encoding of rest pose information. Specifically, the rest pose offset P (j) E IR3 for joint j is transformed into positional embedding RestPE(j) E R.'' by simple ML. module with sinusoidal encoding (Mildenhall et al.,
Diffusion Transformers (DiTs) (Peebles& & Xie, 2023) serve as the backbone for parameterizing the denoising network co to capture spatiotemporal dependencies in motion data. Un- der the fixed skeleton assumption, the input 2t E TR. FxJxD is reduced via a linear projection (Tevet et al., 2023):
2t = LinearProj(a:)) E FxD'
2020), capturing its relative position and geometry within the skeleton in 3D space. These embeddings are added to the corresponding joint tokens:
where the J x D joint dimensions are compacted into D'- dimensional features per temporal frame, while spatial de- pendencies among joints are captured during this projection. To capture the temporal dependency among frames, posi- tional encodings (Vaswani al., 2017) are then added:
(f.) 2(f = =(((x) + PE(J) + TreePE(j) + RestPE(j). (7)
Through this integration, our model explicitly captures the skeletal topology and its rest pose configuration, enabling accurate modeling of motion dynamics for diverse objects.
( 5 =()
( + PE(J),
where 2/ is the nosiy latent token and PE(f) encodes position features for frame f. The resulting representation &, E IR FxD' is processed by the series of transformer blocks and re- re-projected back to the original motion space:
5.4. Two-Stage Learning Approach
While the proposed modifications enhance adaptability to diverse skeleton templates, they also increase computational costs due to the expanded joint dimension. To mitigate (3) this, we draw inspiration from text-to-video diffusion mod- els (Blattmann et al., 2023), adopting a two-stage learning (4) approach with factorized spatial-tempon attention to de- couple pose modeling from motion dynamics.
21 = Transformers($)).
60 = c0(2,, t, C) = LinearReproj(an) E FXJxD
5.2. Extension to Arbitrary Skeleton Topology
In the first stage, we train a pose diffusion model to learn joint rotation dependencies within each frame independently using spatial attention blocks. Multi-view rendered images of each pose are used as conditioning inputs (see Figure 9). Once trained, this model is frozen. In the second stage, we enhance the model by introducing temporal attention blocks
Motion synthesis for diverse objects requires the ability to handle skeleton topologies S with varying number of joints and their dependencies. To do so, our framework is designed to preserve the joint dimension throughout the denoising process. Specifically,, the linear projection layer in Eq. (1)
5

---

How to Move Your Dragon: Text-to-Motion Synthesis for Large-Vocabular Objects
Table 1. Ablation study on the positional encodings (TreePF. & RestPF) and the rig augmentation technique in pose-level diffusion models. @@ I represents top-1 R- R-Precision (retrieval accuracy), with the "+" symbol indicating evaluation on a dataset with rig augmentation applied. The first row represents the oracle performance, where retrieval and alignment are measured between paired ground-trum -ruth pose and image embeddings, reflecting the quality of the embedding space of the learned pose encoder.
Ablated Components
Train Set
Test Set
Test Set"
PEs
Rig Aug.
FID (L)
R@ 1 (1)
Align. (1)
FID (L)
R@ (1)
Align. (t)
FID (L)
R@ I (1)
Align. (1)
0.0000
0.9751
0.9850
0.0000
0.8822
0.9151
0.0000
0.8734
0.9311
X
x
0,9203
0.3602
0.6755
2.2564
0.2626
0.6870
1.0188
0.3705
0.7422
X
/
0,9813
0.3322
0.6627
2.2714
0.2562
0.6763
0.8628
0.3867
0.7454
x
0.0070
0.9739
0.9797
0.7704
0.5979
0.8734
0,6871
0,4435
0.7706
V
0,0072
0.9693
0,9700
0.6802
0.6045
0.8890
0.2636
0.6707
0.9276
after each spatial attention block, allowing it to capture mo- tion dynamics over time while leveraging the pose-level knowledge from the first stage. Additionally, we employ factorized
the original 125K poses and 1K motions. Moreover, we use S patchl143884 (Zhai et al., 2023) to extract image and text embeddings to condition our pose and motion diffusion models, respectively. For further details on data construction pipeline, please refer Appendix B.1.
attention: spatial attention cap- tures structural dependencies by enabling joints to attend to each other within a frame, while temporal attention oper- ates on a per-joint basis to model motion trajectories across frames. We set J = 150 and F' = 90 throughout the paper.
We evaluate the models using six automated metrics, along with a user study, to comprehensively assess their perfor- mance. The metrics include Fréchet Inception Distance (FID) (Lee et al., 2019), R-Precision (Guo et al., 2022), Alignment (Guo et al., 2022), Coverage (Li et al., 2022), Multimodality (Lee et al., 2019), and Motion Stability In- dex (MSD) (Kim et al., 2024). FID measures the similar- ity between the feature distributions of generated and real data, with lower scores indicating more realistic outputs. R-Precision and Aligmment evaluate how well generated samples match input conditions. R-Precision quantifies the proportion of correct matches, while Alignment measures cosine similarity, with higher values indicating better corre- spondence. Coverage assesses how much of the reference data is represented by the generated samples, considering reference covered if its cosine similarity with generated data exceeds a threshold. Multimodality captures the di- versity of generated outputs for a single imput, with higher scores reflecting greater variability. MSI measures temporal smoothness and stability, with higher scores indicating less jitter and stable motion over time. Finally, we conduct a user study to evaluate perceptual quality. Given rendered motion videos from each method, 30 participants compare the outputs in terms of realism, aligmment with the input prompts, and overall preference. We aggregate the results across participants and report the average preference scores.
6. Experiments
All models were trained on a Linux system equipped with ei- ther an NVIDIA RTX A6000 (48GB) or A100 (40GB) GPU. The pose diffusion model required approximately 29GB of VRAM with a batch size of 512 over 400K iterations, com- pleting training roughly 30 hours. The motion diffusion model used about 38GB of VRAM with a batch size of 4 and sequence length of 90, trained for IM iterations over approximately 4 days.
To ensure motion integrity of the augmented rigs, bone length adjustments are restricted to 0.8x-1.2x of the orig- inal and applied symmetrically when symmetry existed. Bone erasing was limited to distal appendages (toes, head tips, tail ends) and redundant spine/root bones. Under these constraints, visually verified 10K augmented results and found them suitable for training.
6.1. Dataset & Evaluation Metries
To evaluate the pose synthesis model, aggregate all mo- tion data for each object category and extract their poses. We then apply clustering, generating 30 distinct pose clus- ters per object. Three clusters are randomly selected as the test pose set, while the remaining clusters are used for train- ing. For motion synthesis evaluation, we randomly select one motion per object for the test set, with the remaining motions used for training.
For automated evaluation, we train and utilize a lese- encoder compute all metrics instead of a motion-level encoder due to the limited availability of motion data. To ensure the pose encoder captures nuanced and meaningful representations of poses, we align its embedding space with that of the pretrained SigLIP image encoder, enabling the pose encoder to inherit the rich semantic knowledge learned during large-scale pretraining. We refer the reader Table I
Next, we apply the data augmentation techniques introduced in Section 4.2 independently to each motion, expanding the dataset to include 2.5M poses and 26K motions, up from
6

---

How to Move Your Dragon: Text-to-Motion Synthesis for Objects
Ours
Ours w/ GPT-Caption
Ours w/o Rig Aug.
Single-Object MDMs
Retargeting
1.0
1.0
1.0
Coverage (1)
0.8
Coverage (1)
0.8
0.8
0.6
0.6
Novelty
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
0.60
0.68
0.76
0.84
0.92
1.00
0.48
0.52
0.56
0.59
0.63
0.00
0.08
0.16
0.25
0.33
0.41
Threshold
Threshold
Threshold
(a) Test on Varied Rigs
(b) Test on Novel Motion Dynamics
Figure 4. Quantitative comparisons on text-to-motion synthesis scenarios. A higher Area- Uneer-the--eeeee the-rve (AUC) indicates the better adaptability to varied rigs or motion dynamies. Our approach enjoys greater generalization capahility in synthesizing motions for diverse rigs (i.e. leftmost pane) and dynamies (Le. middle and right panes) compared to the baselines.
Reference
Ours
w/ GPT-Caption
w/o Rig Aug.
SO-MDMs
Figure 5. Adaptability to novel motion descriptions. We compare the synthesized motions different models for a dragon using the novel motion description from the ant: "An ant is knocked back and ends up lying its back, motionless." " Among them, only our method successfully generates dragon motion that closely aligns with the reference motion from the ant. Please watch the video in this link.
(i.e. 1st row) and Figure 13 for the quantitative and quali- tative evaluation of the learned pose encoder, respectively. Also find Appendix B.2 for additional details regarding the architecture and training protocol.
thesize novel motions dynamics. As this is the first work on text-to-motion synthesis for vocabuocayulary objects with varying skeletal templates, we compare our method against reasonable ablated baselines in this setup. The following sections provide further details.
6.2. A blation Study
Adaptability to Varied Rig Styles. To evaluate adaptability across diverse rig styles, exclude one motion from the training set per object and augment its rig create multiple test sets with consistent dynamics but varied configurations. Coverage is used as metric, measured as Area- Area-Under-ther Curve (AUC) by sweeping the similarity threshold from 0 to 1. We compare our method with two ablated variants: one without rig augmentation and another using GPT-4o- generated captions. Results are presented in Figure 4-(a).
We first examine the necessity of the introduced positional encodings (PEs, i.e. TreePE and RestPE) and the rig aug- mentation technique in enhancing the adaptability of models to diverse object skeletons to improve overall performance. These experiments focus on the first-stage, lose-level diffu- sion models, with results presented in Table 1.
The results clearly demonstrate that models without PEs suffer from significantly degraded performance across all evaluation metrics and datasets. In particular, these mod- els struggle to estimate rotations that produce poses well- aligned with the conditioned inputs, as reflected in low R-Precision (R@) 1) and Alignment scores. When combin- ing both PEs and rig augmentation, the model achieves the best generalization performance. While PEs alone deliver substantial improvements by encoding essential rig-specific information, rig augmentation further enhances the model's robustness to diverse skeleton templates during training.
As shown in the figure, the model without the rig augmen- tation technique exhibits poor adaptability to diverse rig styles, indicated by the lowest AUC. Among others, the model trained with high-quality human-annotated descrip- tions achieves the largest AUC, validating the effectiveness of both the rig augmentation technique and the use of high- quality,
captions for enhancing adaptabil- ity and performance.
Ability to Synthesize Novel Motions. This evaluation tests the model's 's capacity to synthesize motion dynamics not in- cluded in the dataset for particular object by leveraging textual descriptions from other objects. We introduce two additional methods: Single-Object Motion Diffusion Mod- els (SO-MDMs) (Tevet et al., 2023), trained individually for
6.3. Text-to-Motion Synthesis
Next, we evaluate the models' ability to synthesize motions from text descriptions, focusing on two key generalization aspects: (1) adaptability to diverse rigs and (2) ability to syn-

---

How to Move Your Dragon: Text-to-Motion Synthesis for Objects
Reference
Pig
Pegasus
Winged Monster
Alamosaurus
Figure Motion synthesis on novel objects and skeletons. Surprisingly, our model generalizes well to objects with unseen skeletal structures and motion dynamics. Notably, while our dataset includes quadrupedal horses, bipedal reptiles, and dragons, it lacks examples combine four legs with wings or feature winged reptiles. Additionally, no objects in dataset possess necks as long as that of the Alamosaurus. All evaluation data were sourced from the web. For video demonstrations, please visit the project page in this link.
wakes up observes something - bites something inside cave
walks slowly forward - sprinting with its head facing forward
dashes forward bites into something chews and swallows it
hit by something
walks awkwardly - falls to the ground.
Figure Long motion synthesis. Our model generates temporally coherent extended motions for story-level synthesis by conditioning on sequence of descriptions and sampling overlapping short sequences, such as T-rex's 's journey. Please watch the video in this link.
T-rex Skeleton Template
walking forward (47) with its body low (41)
head, back, tail aligned (31) balanced as it strides (25)
Figure 8. Level of detail description and variability in synthesized motions. The value in parentheses represents the variability of motions synthesized from the same caption, measured using the multimodality metric with higher values indicate greater variability. The video can be viewed at this link.
specific objects, and retargetting approach that bypasses learning entirely. Due to computational constraints, SO- MDMs are trained on five representative objects (Dragon, Tyrannosaurus, Cat, Cricket, and White Shark) chosen for their structural and dynamic diversity. We narrow the scope of this evaluation to these five objects accordingly.
rior alignment with queries while introducing novel motions. The analysis reveals two key insights: (1) The drop in perfor- mance for models using GPT-generated captions trained without rig augmentation underscores the importance of high-quality human-annotated captions and rig augmenta- tion, which help improving the model's understanding of motion dynamics and facilitating generalization across di- verse rig styles. (2) The poor performance SO-MDMs and retargetting approaches highlights the importance of amortized learning, which leverages shared structures across objects, enabling better generalization.
Due to the limited scale of motion data, we do not exclude novel motion data from the training set for evaluation. In- stead, we curate 10 key high-level actions that encompass the range of actions represented in the entire dataset. For each action, we randomly select two motion descriptions that align with the action from training objects not included in the five selected. These motions are treated as the target reference dynamics that the model should replicate when queried with the corresponding text descriptions.
Table 2. User Preference and Smoothness (MSI) Scores.
Method
Preference (%, 1)
MSI (x103, 1)
Ours
65.60
8.78
We coverage metric to assess how well generated mo- tions align with reference dynamics. Additionally, we mea- sure novelty by evaluating the inverse coverage of original motions of the chosen objects the training set by their syn- thesized motions. Lower novelty scores indicate replication of training data, while higher scores reflect new, adaptive motions tailored the queried descriptions.
w/ GPT-Caption
2.27
7.30
w/o Rig Aug.
12.40
7.39
SO-MDMs
10.67
7.21
Retargeting
9.07
9.93
Finally, we report comparison results from a user study and a motion smoothness evaluation (measured by MSI). summarized in Table 2. Our method consistently preferred by participants in terms of alignment with text prompts and overall motion quality, receiving the highest preference
Figure 4-(b) shows that our method achieves the highest AUC for both coverage and novelty, demonstrating supe-
8

---

How to Move Your Dragon: Text-to-Motion Synthesis for Large-Vocabulary Objects
score by a substantial margin. Regarding smoothness, our approach achieves the highest score among all data-driven methods, indicating more temporally stable motions, while slightly trailing the retargeting method, which benefits from direct motion transfer but lacks flexibility to handle novel prompts. The results further highlight the effectiveness of our method producing both perceptually preferable and plausible motions.
8. Discussion
We introduced a novel framework for text-driven motion synthesis across diverse object categories with heteroge- neous skeletal structures. By augmenting the Truebones Zoo dataset with rich textual descriptions, introducing rig augmentation techniques, and extending motion diffusion models to dynamically adapt to arbitrary skeletal templates, our approach enables realistic, coherent motion generation for diverse objects. Experiments further demonstrate the potential of our method in generalizing to unseen objects, generating long motion sequences, and enabling controlled motion synthesis. We hope our work lays strong founda- tion for motion synthesis across diverse object categories, ad advancing many applications in 3D content creation.
7. Additional Analysis
7.1. Generalization To Unseen Objects and Skeletons
We evaluate our model's ability to generalize to novel ob- jects with entirely different species and rig configurations.
To this end, we obtain additional object meshes and arma- ture data from external repositories such as SketchFab and use the curated text descriptions from Section 6.3 gener- ate motions. As shown in Figure 6, our model synthesizes coherent motions that align with reference dynamics, even for objects with previously unseen skeletal structures. This demonstrates its potential as a robust foundation for mo- tion synthesis in apen-vocabulary settings, expanding its applicability to diverse and novel object categories.
Despite its strengths, our method has a few limitations. First, for simplicity, our current implementation uses joint-wise Euler angle rotations only and ignores global root transla- tion. However, this is not a fundamental limitation. Our model operates on motion tensors of shape T x J X D, where D denotes joint features-in our case, rotations. This formulation can naturally be extended to include additional features such as global translation, relative translations for coft-constrained joints, or physically relevant signals like joint velocities or foot contact indicators. Including these signals may further improve temporal coherence, realism, and physical plausibility of synthesized motions.
7.2. Long Motion Sequence Generation
Though trained on sequences up to F = 90 frames, our model extends to longer motions by conditioning on multi- ple sequential textual descriptions. To ensure smooth tran- sitions, we apply weighted blending at the boundaries of consecutive motion chunks during sampling. As shown in Figure 7, our method enables the generation of extended, coherent motion sequences. This ability suggests promising applications in story-level motion synthesis (Zhou et al., 2024), where objects dynamically transition through com- plex, extended narratives, opening new possibilities for ani- mation, virtual storytelling, and interactive experiences.
Second, while our rig augmentation strategy enables gen- eralization across diverse skeletal structures, the physical plausibility of augmented skeletons is limited. They may thus violate physical constraints (e.g. symmetry, balance, or anatomical consistency), limiting direct use in production- grade animation or simulation. Future work could improve plausibility via automated validation (e.g. foot foot-ground con- tact, joint velocity bounds, or end-effector stability), com- bined with rejection sampling or two-stage training. For example, pretraining on diverse augmented rigs for general- ization, then tneettunig on physically grounded data, may better balance flexibility and realism.
7.3. Generating Motions with Multi-Level Descriptions
We examine how textual detail impacts motion synthesis. Using T-rex skeleton, we generate motions with progres- sively detailed descriptions. For each level, we sample 50 motions, measure variability using the Multimodality metric (higher values indicate greater diversity), and visualize their mid-frame framef in Figure 8. As shown, increased detail guides motion synthesis toward more structured and nuanced mo- tion patterns, reducing ambiguity while preserving diversity. Thanks to the multi-level description training, our model effectively learns to generate both broad and fine- -in-igdd motions, demonstrating the importance of detailed and well- structured textual annotations in motion synthesis.
Third, while the method advances the field by offering a unified framework for motion synthesis across highly het- erogeneous skeletal structures, it does not generalize well to human or open-vocabulary objects in zero-shot settings, This stems from the limited diversity, scale, and anatomical coverage of the training data: the Trucbones Zoo dataset, though rich in non-human motions, lacks human-like skele- tal structures and dynamics. Expanding to more balanced, large-scale motion corpora, such Objaverse-XI -XL (Deitke et al., 2024), could improve generalization to both human and novel object categories. This is a promising direction for future work, in line with recent advances in open-vocabulary dynamic mesh synthesis (Ren et al., 2023; 2024).
9

---

How to Move Your Dragon: Text-to-Motion Synthesis for Large-Vocabular Objects
Acknowledgement
Guo, C., Zou, S., Zuo, X., Wang, S., Ji, W., Li, X., and Cheng. L.. Generating diverse and natural 3d human motions from text. In CVPR, 2022.
Wonkwang Lee was supported by Institute of Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government (MSIT) (No. RS- 2022-11220156 Fundamental research continual meta- learning for quality enhancement of casual videos and their 3D metaverse transformation). Wonkwang Lee thanks Chac- won Kim, Hakyoung Lee, Howon Lee, Hyeongjin Nam, JungSeok Cho, Kangwook Lee, Moonwon Yu, Suekyeong Nam, and Seongjoon Yang for their insightful feedback dur- ing the early stages of this work. He also thanks Dongmin Park, Gihyun Kwon, Hyuck Lee. Hyunjin Kim, Hyunse- ung Kim, Joo Young Choi, Juhyeong Scon, Jinseo Jeong, Minkyu Kim, and Sehun Lee for their helpful discussions during the development of this work.
Guo, C., Mu, Y., Javed, M. G., Wang, S., and Cheng, Momask: Generative masked modeling of 3d human motions. In CVPR, 2024.
Harris, N.
Pig, 2021. //sketcchaa..com pig- 24g-9335844000447666e66 9d6b0860468
URL https:
Ho, J. and Salimans, T. Classifier-free diffusion guidance. NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. In NeurlPS, 2020.
Impact Statement
Kapon, R., Tevet, G., Cohen-Or, D., and Bermano, A. H. Mas: Multi-view ancestral sampling for 3d motion gener- ation using 2d diffusion. In CVPR, 2024.
This work advances the field of machine learning by intro- ducing novel approach for synthesizing motions across broad spectrum of objects with diverse and arbitrary skeletal structures, enabling more flexible and generalizable motion generation. Potential applications include gaming, anima- tion, virtual reality, and interactive experiences, broadening accessibility motion synthesis tools.
Kim, Y., Lee, H., Kim, S.-B., Nam, S., Ju, J., and Oh, T.- H. A large-scale 3d face mesh video dataset via neural re-pan parameterized optimization. TMLR, 2024.
Lee, H.-Y., Yang. X., Liu, M.-Y., Wang, T.-C., Lu, Y.-D., Yang, M.-H., and Kautz, Dancing to music. In NeurlPS, 2019.
While method enhances creative applications, it also raises considerations regarding the ethical use of synthetic motion data, such as potential misuse in deceptive media or unintended biases in generated animations. We encour- age responsible deployment and further research into bias mitigation and transparency generative models.
Li, P., Aberman, K., Zhang, Z., Hanocka, R., and Sorkine- Horuung O. Ganimator: Neural motion synthesis from a single sequence. SIGGRAPH, 2022.
Liang, H., Bao, J., Zhang, R., Ren, S., Xu, Y., Yang. S., Chen, X., Yu, J., and Xu, L. Omg: Towards open- vocabulary motion generation via mixture of controllers. In CVPR, 2024.
References
Baran, and Popović, J. Automatic rigging and animation of 3d characters. SIGGRAPH, 2007.
Ma, N., Goldstein, M., Albergo, M. S., Boffi, N. M., Vanden- Eijnden, E., and Xie, S. Sit: Exploring flow and diffusion- based generative models with scalable interpolant trans- formers. In ECCV, 2024.
Batuhan Pegasus, 2021.
URL https:
//sketchfab. //ssettchfaaa.ccmm3mmmmmoeaee com/3ddmommmsls.
pegasus- 02Feelc9CCC4499997722622288
Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S. W., Fidler, S., and Kreis, K. Align your latents: High- resolution video synthesis with latent diffusion models. In CVPR, 2023.
Mildenhall, B.. Srinivasan, P., Tancik, M., Barron, J. T., Ramamoorthi, R., and Ng, R. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
chengzijieczj.
icy dragon, 2020. ftttp::/sskecchnn htw.:/ttchhaaa.ccmmmddmeeeeeeee iaraaamcc2b222222211611113333333
URL
Nuvoli, S., Pictroni, N., Cignoni, P., Scateni, R., and Tarini, M. Skinmixer: Blending 3d animated models. SIG- Asia, 2022.
Deitke, M., Liu, R., Wallingford, M., Ngo, H., Michel, O., Kusupati, A., Fan, A., Laforte, C., Voleti, V., Gadre, S. Y., VanderBilt, E., Kembhavi, A., Vondrick, C., Gkioxari, G., Ehsani, K., Schmidt, L., and Farhadi. A. Objaverse-xl: a universe of 10m+ 3d objects. NeurlPS, 2024.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In ICCV, 2023.
Plappert. M., Mandery, C., and Asfour, T. The KIT motion- language dataset. Big Data, 2016.
10

---

How to Move Your Dragon: Text-to-Motion Synthesis for
Objects
Raab, S., Leibovitch, I,, Tevet, G., Arar, M., Bermano, A. H., and Cohen-Or, D. Single motion diffusion. In ICLR, 2024.
Zhao, Q. Li, P., Yifan, w., Olga, S.-H., and Wetzstein, G. Pose-to-motion: Cross-domain motion retarget- ing with pose prior. In Proceedings of the ACM SIG- GRAPH/Eurographis Symposium on Computer Anima- lion, 2024.
Ren, J., Pan, L.. Tang, J., Zhang, C., Cao, A., Zeng, G., and Liu, Z. Dreamgaussian4d: Generative 4d gaussian splatting. arXiv preprint ar arXiv:2312.17142 2023.
Zhou, Y., Zhou, D., Cheng, M.-M., Feng, J., and Hou, Q- Storydiffusion: Consistent self-attention for long-range image and video generation. NeurIPS, 2024.
Ren, J., Xie, K., Mirzaei, A., Liang, H., Zeng, X., Kreis, K., Liu, Z., Torralba, A., Fidler, S., Kim, S. W., and Ling, H. LAgm: Large 4d gaussian reconstruction model. NeurIPS, 2024.
Zhu, J.-Y., Park, T., Isola, P., and Efros, A. A. Unpaired image-to-image translation using cycle-consistent adver- sarial networkss. In ICCV, 2017.
robertfabiani.
Alamosaurus, 2018.
URL
https://sketchhn mowp
Shiv, V. and Quirk, C. Novel positional encodings to enable
tree-based transformers. In NeurlPS, 2019.
Song. Y. and Ermon, S. Improved techniques for training score-based generative models. In NeurIPS, 2020.
Tevet, G., Raab, S., Gordon, B., Shafir, Y., Cohen-or, D., and Bermano, A. H. Human motion diffusion model. ICLR, 2023.
Truehones. Free fbx/.bvh zoo. over 75 animals with anima- tions and textures. https::/// /ttp:pp///teeeee httreuuueeeee.eee gumroad. com/1/skzMC/ 2022. Accessed: 2024-07-01.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In NeurlPS, 2017.
Wang, J., Yuan, H., Chen, D., Zhang, Y., Wang. X., and Zhang, S. Modelscope text-to-video technical report. arXiv preprint wrXiv:2308.06571, 2023.
Yang, Z., Zhou, M., Shan, M., Wen, B., Xuan, Z., Hill, M., Bai, J., Qi G.-J,, and Wang, Omnimotiongpt: Animal motion generation with limited data. In CVPR, 2024.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training. In ICCV, 2023.
Zhan, X., Fu, R., and Ritchie, D. Charactermixer: Rig-aware interpolation of 3d characters. CGF, 2024.
Zhang, J., Huang, S., Tu, Z. Chen, X., Zhan, X., Yu, G., and Shan, Y. Tapmo: Shape-aware motion generation of skeleton-free characters. In ICLR, 2024.
Zhang, M., Li, H., Cai, Z., Ren, J., Yang, L., and Liu, motion
Z. Finemogen: Fine-grained generation and editing. In NewrIPS, 2023.

---

How to Move Your Dragon: Text-to-Motion Synthesis for Large-Vocabulary Objects
Appendix
This section presents additional details that are omitted from the main paper due to space constraints.
Front
Front Left
Left
Back Left
Back
Back Right
Right
Front Right
Top
Bottom
Figure 9. Rendered view images of a hamster's pose from ten predefined camera viewpoints. We use these rendering images to annotate descriptions for each motion and train the pose encoder.
A. Details on Data
Source In this paper, primarily utilize the Truebones Zoo (Truebones, 2022) dataset for training and evaluation. The types of objects and the number of 3D animations available for each object are summarized in Table 3. Each animation is provided in FBX format, consisting of an armature and its corresponding animation, with the armature rigged to mesh.
Preprocessing To prepare the data for training, preprocess each sample by standardizing its orientation, scale, and center. Specifically, during preprocessing, manually adjust the orientation of each motion to align the forward direction with the -Y-xiss and the upward direction with the Z-axis. The armature and mesh are then uniformly scaled that the longest dimension of their bounding box is normalized to 1. Finally, we reposition the object to be centered at the origin (0,0.0).
Extracting Training Data After preprocessing, extract the armature animation into BVH format, discarding joint positions and retaining only the bone hierarchy information (i.e. skeletal structures and rest poses) and joint rotations. These components are utilized as training features: the bone hierarchy information serves static features, while joint rotations are treated as dynamic features. The joint rotations are represented using Euler angles in a ZXY order. Additionally, individual poses in each motion are rendered from ten distinct camera perspectives: front, front left, left, back left, back, back right, right, front right, top, and bottom. Please find Figure for the example for single pose. These rendered images are used to create motion description annotations and train image-to-pose diffusion models.
Annotation Pipeline We manually annotate high-quality motion descriptions for each motion sample. Human annotators are initially provided with rendered videos captured from the front-left camera view but are given access to additional views when the front-left perspective alone is insufficient to accurately identify the animation. Annotators are instructed to describe the motions as much detail as possible, focusing on the following key aspects: (1) Initial Pose: clear description the starting pose of the object (e.g. initially standing), (2) High-Level Actions: the overarching actions being performed over time (e.g. standing, striking, etc.), and (3) Part-Level and Directional Dynamics: finer-grained details of motions, such as specific body part movements and directional actions (e.g. "Initially standing all fours, an object is standing on hind limbs by pushing off the ground with its two front limbs, followed by striking by lifting its right front paw over its head"",,
Subsequently, these annotated captions are refined using GPT-4o to create multi-level descriptions. Specifically, the annotated captions are first revised enhance clarity and ensure consistency across descriptions; these revisions are used as
12

---

How to Move Your Dragon: Text-to-Motion Synthesis for Large-Vocabullng Objects
Table The types of objects, the number of the 3D animations, and thier number joints in Truebones Zoo dataset (Truebones, 2022).
Object Name
# Animations
Average # Joints
Object Name
# Animations
Average # Joints
Alligator
21.00
Jaguar
14
40.00
Anaconda
18
27.00
Great White Shark
10
16.00
Ant
16
34.00
King Cobra
11
17.18
Bat
11
43.00
Leopard
14
43.00
Bear
30
62.00
Lion
15
26.00
Bird
19
53,00
Lynx
14
33.00
Brown Bear
22
32.00
Mammoth
12
36.00
Buffalo
18
36.00
Monkey
15
70.00
Buzzard
11
50.00
Ostrich
10
45.00
Camel
19
46.00
Parrot-1
14
56.00
Cat
4
38.00
Parrot-2
3
50.00
Centipede
64.00
Pigeon
12
7.08
Chicken
4
30.00
Piranha
11
22.00
Komodo Dragon
50.00
Polar Bear
36.00 9
Coyote
10
36.00
Baby Polar Bear
14
34.00
Crab
44.00
Pteranodon
34.00 13
Cricket
11
43.00
Puppy
34.00 4
Crocodile
12
34.00
Reindeer
9
32.00
Crow
10
23.00
Velociraptor-1
10
33.00
Deer
21
37.00
Velociraptor-2
39
49.00
Dog-1
42
46.00
Velociraptor-3
14
50.00
Dog-2
36
46.00
Rat
13,00
Dragon
111.00
Rhino
38.00
Eagle
12
39.00
Roach
11
35.00
Elephant
13
32.00
Sabre-Toothhr Tiger
54.00 45
Fire Ant
25
35.00
Sand Mouse
13
33,00
Flamingo
34.00
Scorpion-1
13
53.00
Fox
13
35.00
Scorpion-2
40
46.00
Gazelle
18
35.00
Skunk
8
29.00
Giant Bee
39.00
Spider-1 1
33
57.00
Goat
11
27.00
Spider-2
18
51.00
Hamster
5
38.00
Stegosaurus
10
35.00
Hermit Crab
12
55,00
Tyrannosaurus Rex
70
49,00
Hippopotamus
10
36.00
Triceratops
9
27.00
Horse
29
63,00
Toucan
9
15.00
Hound
12
40.00
Turtle
10
39.00
Termite
52.00
Tyrannosaurus
10
54.00
the long descriptions. Next, GPT-4o is tasked with producing coarser-level descriptions by progressively omitting part-level or directional details from the long descriptions, resulting mid-level and short descriptions.
For visualization of word count distributions across description levels, see Figure 10. Statistics on the most frequent verbs and nouns are presented in Figure 11 and Figure 12, respectively.
External Test Data We downloaded and used four armatured object meshes from Sketchfab: Pig (Harris, 2021), Icy Dragon (chengzijieczj. 2020), Pegasus (Batuhan 13, 2021), and Alamosaurus (rohertfabiani, 2018) to evaluate our model's ability to generate motions for novel objects.
13

---

How to Move Your Dragon: Text-to-Motion Motion Synthesis for Objects
160
Short (Mean: 12.2)
Mid (Mean: 29.5)
140
Long (Mean: 56.2)
120
100
Frequency
80
60
40
20
20
40
60
80
100
120
Word Count
Figure 10. Word count histogram by description type.
14

---

How to Move Your Dragon: Mext-to-Motion Synthesis for Vargblloyyyy Objects
Top 50 Verbs in Short Descriptions (# Unique Verbs: 303)
145
100
99
Count
80
61
40
30
35
31
20
22
19
18
16
14
Stand
walk
ralse
lower
strike
turn
move
run
flap
Tall
attack
lie
tite
collapse
side
lift
shake
prepare
sway jump
use extend
appear
die
spread
leave
step
return perform
leap
breathe
rise
wag shook
bend
face
open lose cur!
flip twist
Swim
charge
fly nüsh maintain rest
Top 50 Verbs in Mid Descriptions (# Unique Verbs: 496)
253
250
200
195
Count
130
114
107
99
85
76
68
49
50
40
stand
lower
move
lift
raise
side
extend
turn
maintain
onen
appear
cause
strike
use face flap
run bend
sway begin
push
equip
attack ck
lose
prepara
Spread
step
bite
collapse
fall
shake
curl
take
Rull
return
leap
position
breathe
keep rotate
fold
follow
Bit propel
rica direct
fly
Top 50 Verbs in Long Descriptions (# Unique Verbs: 708)
400
368
300
Count
250
225
197
167
150
150
125
107
100
80
62
maintain stand
move
begin
Tower
lift
propel
Use
extend
create
prepare
side
cause
raise
ensure
open
indicate follow sway
push serise strike
shift
alm ssition remain
berd
walk
flap
allow
gather
suggest
focus
rise take stabilize e
face collapse
start lose execute
run elevate
for
adjust Curl
Figure 1I. Top 50 verbs and their counts in annotated motion descriptions.
15

---

How to Move Your Dragon: Mext-to-Motion Synthesis for Varge-oocabulrry Objects
Top 50 Nouns in Short Descriptions (# Unique Nouns: 402)
223
165
150
Count
100
75
57
50
32
28
19
10
head body
tail
ground
leg
wing
side
attack
dog
bear
rex
raptor
scorpion
Spider
position
left tiger right mouth
air motion balance
posture degree
horse
four
sabre
ant
foot
neck
hind
fire
deer
crab
alligator
bite
stance
claw
water
prey
camel
crouch
anacordd
sky
gazelle
buffalo
bird
arm
Top 50 Nouns in Mid Descriptions (# Unique Nouns: 560)
600
SS7
500
450
Count
346
300
277
255
200
158
148
90
84
71
67
50
46
36
31
leg body head
tail
ground
side
wing
moutt
mation
hind
paw
position
right
neck
air
bear attack
Toot knee
dog
raptor
belance
four
posture
scorpion
Spider
Fant
strength
back
tiger
step
claw
stance
arm opponent
horse
degree
ant surraundin niinn pinteer stinger
jaw
inpaat
belly
Top 50 Nouns in Long Descriptions (# Unique Nouns: 1000)
881
Count
530
496
400
374
271
16605
200
18474,56
135
113
103
97
78
body
leg head ground
motion
tail
sidle
balance
moverent
posture
position
wing hind air mouth
farget
bear
force stance attack
dog momentum
right
heck
aurrounding
left
raptor
step
threat
weight
faut scorpion
knee
impact
stabllity precisian
jaw eye spider front speed environmet state
action
four
sky
Figure 12. Top 50 and their counts in annotated motion descriptions.
16

---

How to Move Your Dragon: Text-to-Motion Synthesis for Large-Vocabulary Objects
B. Details on Implementation & Training
B.1. Generalized Motion Diffusion Models (G-MDMs)
Architecture We adapt and modify the SiT-S (Ma et al., 2024) model to align with our setup, configuring the depth, hidden dimension, and the number of self-attention heads to 12, 384, and 6, respectively. On top of this, we introduce additional encoding layers to enhance the model's functionality: the condition encoding layer, tree position encoding (TreePE) layer, rest pose offset encoding (RestPE) layer, and frame position encoding layer. Each layer utilizes Linear-Sil.U-Linees block to project input features into D-dimensional hidden space. Specifically, the condition encoding layer maps features from S-SS00000---eeee 4--38 (Zhai et al., 2023) model into the diffusion model's
either the image or text encoder of the hidden dimension. TreePE and RestPE encode spatial positional information, while the frame position encoding layer captures the temporal sequence of motion.
To handle the variable number of joints and frames in motion data, we set the maximum number of joints and frames to 140 and 90, respectively. The maximum joint count reflects the largest number of joints observed in the dataset, while the maximum frame count was manually set to 90 after observing the motion sequences inputs with fewer joints or frames than these maximum values are padded with zeros, and the padded elements are masked within the self-attention modules to ensure they do not influence the diffusion process. For motions containing more than 90 frames, we randomly sample a chunk of frames during training. To condition the model on rendering images or text, we adopt the adaLN-Zerg conditioning approach (Peebles & Xie, 2023), which has demonstrated strong effectiveness in integrating conditional embeddings into the model.
reasonable cutoff. During training,
To address the challenges posed by limited computational resources and potential overfitting due the small size of the training motion dataset (approximately IK samples), draw inspiration from text-to-video diffusion models (Blattmann et al., 2023; Wang et al., 2023) and adopt a two-stage training approach alongside factorized attention mechanisms. This approach decouples the modeling of poses from motion dynamics, effectively reducing computational complexity while maintaining adaptability and generalization.
Two-Stage Training
In the first stage, we train a pose diffusion model to capture dependencies among joint rotations within each frame using spatial attention blocks. Instead of relying on textual descriptions, the pose synthesis model is conditioned on average embeddings of poses image renderings viewed from ten predefined camera angles (e.g. front, front left, left, back left, back, back right, right, front right, top, bottom). Once the pose diffusion model is trained, it is frozen.
In the second stage, temporal attention blocks are introduced after each spatial attention block. These temporal attention blocks enable the model to capture motion dynamics across time, leveraging the pose-level knowledge acquired during the first stage. This staged training approach reduces computational overhead while ensuring the synthesis of coherent and realistic motion sequences.
To further manage computational complexity, we adopt a factorized attention mechanism inspired by text-to-video diffusion models. In this setup, spatial attention operates within each frame, allowing joints to attend to one another and effectively capturing structural dependencies. Temporal attention, on the other hand, operates across frames, enabling each joint to focus on its motion trajectory over time. By separating spatial and temporal attention, the model reduces the computational cost of full spatiotemporal modeling while maintaining the capacity to generate realistic and temporally coherent motions.
B.2. Pose Encoder
We train an additional pose-level encoder network to facilitate the evaluation of the pose and motion diffusion models. The encoder adopts the same architecture as our pose-level generalized diffusion model, as detailed in Section B.I.
The forward pass of the pose encoder identical to that of the pose diffusion model, except that the pose encoder's output is derived by average pooling the pose representations along the joint dimension. Specifically, given two types of inputs-44 (1) static feature information, represented by TreePE and RestPE, and (2) dynamic feature information, either real or synthesized, represented as rotation values with a shape of (B x J x the pose encoder pools the output along the joint dimension to produce a final representation of shape (B x D). This representation is then projected into the feature space of
1-33444 serving as pose embeddings.
The pose encoder is trained to align its embeddings with those of the image encoder of SigLIP S0LLP--S400MM-pach1144404 using contrastive learning. To improve this alignment, we introduce learned view image embedding layer. Specifically,
17

---

How to Move Your Dragon: Text-to-Motion Synthesis for Large-Vocabular Objects
Query
Retrieved Top-1 Samples
Idle
Idle
Idle
IdleNest
Attack3
Attack
Attack2
Rearing
Strikel
Attack3
Die3
Die
Dying
Shot
die
Figure 13. Qualitative evaluation of the learned pose embedding space. Given query pose of sandmouse, present rendered images of the top-1 retrieved poses from four distinct objects: coyote, horse, eagle, and crab. We display the file name of the motion file containing each retrieved pose. These results highlight the ability of the learned pose encoder effectively embed similar poses from different animal objects into the same embedding space, capturing meaningful semantic relationships within the pose space.
for each pose, a paired of camera views, as described in Section A, embedded into image embeddings using the pretrained image encoder. Camera position embeddings then added these view embeddings, followed by layers of self-attention blocks. The outputs average pooled produce unified image embedding. which serves as the target for alignment with the pose embeddings.
To evaluate the quality of the learned pose embedding space, we perform both quantitative and qualitative analyses. For the quantitative evaluation, which measures the alignment performance of the trained pose encoder, please refer to Table The qualitative results can be found in Figure 13.
As evident from both evaluations, pose encoder demonstrates strong alignment with the aggregated image embeddings of the pretrained image encoder (e.g., high retrieval and alignment scores in the table). Furthermore, it has learned meaningful semantic structures within the pose embedding space, effectively identifying and grouping similar poses of different objects within this space.
C. Details on Coverage and Novelty Metrics
In human motion synthesis, it is common practice to train and utilize motion encoder that projects motion sequences into embedding space, where evaluation metrics subsequently computed. However, due to the limited availability of motion data in our setup, we instead leverage a learned pose encoder to assess the quality of the synthesized motions. We adapt the original coverage metric (Li et al., 2022) to our setup, making slight modifications to better suit our motion synthesis evaluation.
Given reference motion x and a synthesized motion è of lengths L, and Lg, respectively, we extract all possible windows of size F.. from each motion:
18

---

How to Move Your Dragon:
Text-to-Motion Synthesis for Large-Vocabulie Vocabulary Objects
1.0
Coverage (1)
0.8
0.6
0.4
Ours
Ours w/ GPT-Caption
0.2
Ours w/o Rig Aug.
0.0
0.45
0.56
0.67
0.78
0.89
1.00
Threshold
Figure 14. Quantitative comparisons of the models on the training set, evaluated using the coverage metric. While our model achieves comparable performance fitting the training data compared the version without rig augmentation, our full approach demonstrates significant gains in generalization (i.e. Figure 4).
W(x, Fw) Fo Ma Fw+t W(E,F.) Fohiit Fullt---ll siml
(8)
Here, the window size Fw determined F. min(L L2, F), where F = 90 is the predefined maximum frame length for our generalized motion diffusion model.
Coverage To assess how well the synthesized motion å covers the reference motion we compute the coverage metric as:
Cov(,, () [W(r, Fw)]
E
1
max CosineSimilarity((r..... kw)) > 6) 6) B.E.EE
(9)
This metric quantifies proportion reference motion windows zy that have at least one synthesized counterpart â., with cosine similarity above threshold O. To obtain a comprehensive evaluation, we sweep from and compute the area under the curve (AUC) of the coverage function, providing aggregate measure of how well the synthesized motion spans the reference motion across different similarity thresholds.
Novelty We also introduce the novelty metric, which measures how distinct the synthesized motion å is from the reference motion x. Higher novelty values indicate that the generated motions introduce new patterns rather than replicating the reference.
Ni
x; 0) [W(s, Fw.)l
Σ
1/a-
1
max FL)
CosineSimilarityyr.. Zww)) > θ (10)
FL(.FF)
This metric captures the proportion of synthesized motion windows a. that do not closely resemble any reference window Tar, ensuring that the generated motions exhibit novelty. As with coverage, we sweep θ from and compute the AUC of the novelty function to obtain overall measure of the diversity of the synthesized motion.
19


Reviews:
[{"review_id": "rPnfQH3KuU", "reviewer": "Reviewer_4SvT", "summary": "This work presents a major advancement in text-driven motion synthesis for large-vocabulary objects with heterogeneous skeletal structures. By augmenting datasets, introducing novel rig adaptation techniques, and extending motion diffusion models, the authors enable realistic motion synthesis for both seen and unseen objects. This framework lays a strong foundation for animation, gaming, virtual reality, and robotics applications.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s valuable time and feedback.\n\n### Q1: Comparisons with SOTA models like OmniMotion-GPT or SinMDM.\n\nWe agree that comparisons to SOTA models are valuable. However, OmniMotion-GPT and SinMDM address different goals and operate under different assumptions, making direct, meaningful comparisons less feasible.\n\nOmniMotion-GPT leverages human motion priors for text-driven motion synthesis of SMAL-based quadrupeds. It relies on:\n- Predefined joint correspondences between humans and animals, supporting only fixed, four-limbed skeletons (e.g., quadrupeds), and failing on animals with non-analogous structures (e.g., snakes, fish, insects).\n- Text input alone is insufficient: relevant human motion is required at inference (e.g., animating “a lion pushing a box” needs a human pushing motion), limiting text-only synthesis.\n- Since our dataset (Truebones Zoo) contains a wide variety of non-human skeletons for which such paired human motion does not exist, applying OmniMotion-GPT in this context is infeasible.\n\nSinMDM, while based on MDM like ours, differs fundamentally:\n- It aims to discover and recombine submotions (i.e., motion motifs) from a single clip using local attention to increase the diversity within a single motion, rather than focusing on text-driven synthesis or spanning across object categories.\n- It assumes a fixed skeleton and rest pose, making it incompatible with varied or unseen structures, and is not text-conditioned.\n- To adapt SinMDM for our setting, it would require substantial changes: introducing text conditioning and training a separate model per object type. In that case, the closest equivalent in our experiments is Single-Object MDM, which we already include as a baseline.\n\nThat said, we view SinMDM as orthogonal and potentially complementary to our work. While SinMDM enhances diversity within a fixed structure and motion, our model focuses on text-driven synthesis and cross-categories modeling. Combining these directions would be a promising avenue for future research.\n\n### Q2: Evaluation on synthesized long motions.\n\nWe thank the reviewer for the suggestion. We evaluated both semantic consistency and temporal smoothness in long-form motion with respect to the original short-form motion.\n\nTo assess semantic fidelity, we conducted a retrieval test using the pretrained pose encoder:\n- Using 10 captions from Sec. 7.2, we generated 100 long motions (900 frames each = 10 × 90-frame segments), with shuffled caption order. These were split into 1000 90-frame segments.\n- Separately, we generated 100 90-frame reference clips per caption (1000 total).\n- For each segment, we retrieved the top match from the reference set using cosine similarity. A correct match retrieved a clip from the same caption.\n\nThis yielded 95.6% top-1 accuracy, indicating that segments remain semantically aligned after blending.\n\nTo assess temporal smoothness, we measured joint velocity norm (lower = smoother) and Motion Stability Index (MSI [1,2]; higher = smoother) over 5-frame clips sampled from three conditions, using the same generated and reference sequences as above:\n- Upper Bound: from random mid-segments within reference clips — velocity = 0.094, MSI = 79,602\n- Ours: from overlapping segments with blending in long-form generated motion — velocity = 0.188, MSI = 26,716\n- Baseline: from hard concatenation of independently generated clips — velocity = 0.444, MSI = 3.11\n\nThese results show that our method significantly reduces boundary discontinuity compared to naive concatenation, while preserving motion dynamics close to intra-clip smoothness. We will include these results in the revised manuscript.\n\nReferences:\n\n[1] Ling et al., StableFace: Analyzing and Improving Motion Stability for Talking Face Generation, ECCV 2022.\n\n[2] Kim et al., Audio-driven Talking Face Generation with Stabilized Synchronization Loss, CVPR 2023.\n\n### Q3: Generalization to unseen skeletons\n\nTreePE and RestPE are introduced to make the model skeleton-aware, aiding learning, while rig augmentation is designed to drive generalization.\n\nOur intuition behind rig augmentation is simple: First, by constructing and showing the model a distribution of skeletons that share the same motion and caption, we encourage it to focus on the motion’s semantic meaning rather than its specific structural form. In addition, by displaying a variety of skeleton topologies and rest poses for each motion, we aim to teach the model how to manipulate bones to generate the motion. Moreover, as these distributions expand through augmentation, they may begin to overlap across different object classes, which we believe acts as a bridge and helps the model transfer motion patterns more effectively to novel skeletons.\n\nWe’ll clarify this in the revision—our intent was for PE to support learning via structural awareness, and for rig augmentation to promote generalization by encouraging structural invariance.", "claims_and_evidence": "1. Claim: The proposed method significantly outperforms all existing approaches in text-to-motion generation.\nPartially supported, but lacks direct baseline comparisons.\nWhile the model is tested against ablated versions (e.g., without rig augmentation or using GPT-generated captions), there is no direct comparison with existing SOTA models like OmniMotion-GPT or SinMDM. The paper would benefit from side-by-side quantitative evaluations against prior text-to-motion models using the same datasets.\n\n2. Claim: The framework enables high-fidelity, temporally coherent long-motion synthesis.\nLimited empirical validation.\nFigure 7 shows generated longer sequences, but there are no explicit temporal consistency metrics (e.g., FID over extended motion sequences, smoothness scores).\nThe paper could strengthen this claim by adding quantitative evaluations of motion continuity and stability over time.", "methods_and_evaluation_criteria": "N.A", "theoretical_claims": "No Theoretical Bound for Generalization to Novel Skeletons:\nThe claim that the model generalizes well to unseen skeletons (Section 7.1) is based on empirical evidence, but there is no theoretical proof explaining why TreePE and RestPE should generalize across arbitrary skeletal configurations.", "experimental_designs_or_analyses": "N. A", "relation_to_broader_scientific_literature": "N. A", "essential_references_not_discussed": "N. A", "other_strengths_and_weaknesses": "N.A", "other_comments_or_suggestions": "N.A", "questions_for_authors": "N.A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N.A"}}, {"review_id": "JT20V7M0l8", "reviewer": "Reviewer_cRx3", "summary": "This work presents a unified framework for motion synthesis across a diverse range of objects with varying skeletal structures and rest poses. To generate training data, the authors augment the Truebones Zoo dataset by modifying skeletal structures and rest poses , and providing textual descriptions at multiple levels of detail.\n\nThe diffusion model is built upon DiT, where the technical contributions lie in the new TreePE and RestPE layers introduced to accommodate arbitrary skeletal topologies and rest poses. Additionally, a two-stage training strategy is implemented to address computational resource constraints and data limitations.\n\nBased on the reported quantitative and qualitative results, the proposed method demonstrates strong performance compared to baselines and exhibits generalizability to novel skeletons and motion descriptions.\n\n## update after rebuttal\n\nAfter reading the rebuttal, I will maintain my score as weak accept. I do feel the solution proposed by the authors is interesting, and with the release of the augmented dataset, it should have border impact for subsequent works in related areas such as artificial agents and robotics.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for raising important points and providing constructive feedback.\n\n### Q1: Why do the simple retargeting method perform better than other baselines?\n\nData-driven learning-based methods each have limitations in generalization or controllability:\n- GPT-Caption is trained on captions automatically extracted from rendering images of motions, which tend to be noisy or ambiguous. This weakens the model’s ability to learn reliable text-motion alignment, resulting in limited text-driven controllability.\n- Single-Object MDM is trained on motion from a single object category and tends to overfit, making it difficult to generalize beyond the training distribution.\n- w/o Rig Aug is trained on high-quality captions and diverse objects, but without rig augmentation, the model lacks sufficient exposure to structural variation, limiting its cross-embodiment transfer capability, as our response to Q3 of Reviewer 4SvT.\n\nIn contrast, retargeting directly copies bone transformations from source motion. This naturally boosts novelty and, when source and target skeletons are similar, leads to higher coverage. That said, retargeting can't generate motion from text and depends on existing motions and structural similarity.\n\n### Q2: More description and analysis for long-motion synthesis.\n\nWe fully agree that naively blending joint rotations, regardless of the representation used (e.g., euler angles, quaternions, rotation matrices, or even continuous 6D representations), does not always guarantee a smooth or physically plausible transition, due to the discontinuous and nonlinear nature of rotational spaces [1]. We also acknowledge that this is closely related to motion in-betweening, which typically requires dedicated interpolation strategies or learned dynamics.\n\nTo generate long motions, we prepare $B$ consecutive text descriptions, each guiding a motion segment of length $F$, resulting in a long sequence of $B \\times F$ frames. Before denoising, we reorganize the sequence into $B$ overlapping segments. Specifically, the $b$-th segment spans frames from $(b - 1)F - (b-1)W + 1$ to $bF - (b - 1)W$ in the original sequence, introducing overlaps of $W$ frames with both the $(b - 1)$-th and $(b + 1)$-th segments. The first segment ($b = 1$) is an exception and simply takes the first $F$ frames, i.e., frames $1$ to $F$. This reorganization results in duplicated content within the overlapping regions. It’s worth noting that, as a result, the final segment ends at frame $BF - (B - 1)W$, and any frames beyond that (i.e., from $BF - (B - 1)W + 1$ onward) are ignored during sampling. Each divided segment is then denoised independently, conditioned on its respective caption. After denoising, the segments are concatenated to reconstruct the full sequence, while the overlapping regions are blended by simple averaging to resolve duplication.\n\nWhile our implementation uses uniform blending (i.e., averaging), more sophisticated strategies—such as linear ramps or cosine-weighted curves—can also be applied. Despite its simplicity, we empirically found that uniform averaging produces smooth, coherent transitions, especially when the overlap size $W$ is moderate (e.g., $W = 5$) and the text prompts transition naturally between segments.\n\nAs also noted by Reviewer 4SvT, we evaluated long motion quality via semantic fidelity and smoothness, comparing it to short sequences ($F=90$) used during training, to assess whether quality degrades over time. Please find our response to Q2 of Reviewer 4SvT for details.\n\nReferences:\n\n[1] Zhou et al., On the Continuity of Rotation Representations in Neural Networks, CVPR 2019.\n\n### Q3: About global translation.\n\nIn our current setup, we model only joint-wise euler angle rotations and omit global translation of the root joint. However, this was a design choice made for simplicity, not a fundamental limitation of the model.\n\nThe model operates on input-output tensors of shape $F \\times J \\times D$, where $D = 3$ corresponds to joint rotations. This can be naturally extended to include additional joint-level features—such as global root translation or relative translations for soft-constrained joints—by increasing $D$ (e.g., $D = D_\\text{rot} + D_\\text{trans}$), or further expanded to incorporate physically informative signals like joint velocities, foot contact indicators, or positions to enhance temporal coherence, realism, and physical plausibility.\n\nWe will include a discussion of this direction in the revised manuscript.\n\n### Q4: About computational resources.\n\nAll models were trained on a Linux system with either an NVIDIA RTX 48GB A6000 or 40GB A100 GPU.\n- Pose Diffusion Model: \\~29GB VRAM, batch size 512, 400K iterations (\\~30 hours)\n- Motion Diffusion Model: \\~38GB VRAM, batch size 4, sequence length 90, 1M iterations (\\~4 days)\n\nWe will include this information in the revised manuscript.", "claims_and_evidence": "- The necessity of the proposed positional encoding layers and data augmentation techniques is validated through ablation studies.\n- The method's generalizability to diverse rigs and motion descriptions is demonstrated through both quantitative and qualitative experiments.", "methods_and_evaluation_criteria": "The proposed method and evaluation are reasonable.", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "- The ablation study examines the necessity of positional encodings and rig augmentation. It looks good to me.\n- The quantitative comparisons in Figure 4 compares the proposed model design with other baseline architectures on text-to-motion synthesis. It's sound to me, with only one small question: why does the simple retargeting method perform better than other baselines? Any insights here?\n- Motion synthesis on novel objects and skeletons are qualitatively evaluated on different species, which are reasonable to me.\n- Long motion sequence generation is achieved by conditioning on sequential descriptions with a sliding window. For this experiment, I would like to see more detailed descriptions like how the weighted blending is performed. I can understand the idea in general, but I'm a bit skeptical since I don't think naively blending the joint rotations can always ensure a natural and smooth transition. It's more like a motion in-betweening problem, so more detailed description and analysis are welcome.\n- Generating motions with multi-level descriptions is sound to me.", "relation_to_broader_scientific_literature": "This paper proposes a motion synthesis framework for a broad range of objects, extending previous motion diffusion models beyond humanoid applications. In general, this approach has potential applications in areas such as artificial agents and robotics, enabling motion synthesis for non-humanoid robots.", "essential_references_not_discussed": "References are fairly enough for me.", "other_strengths_and_weaknesses": "[Strengths]\n\n- The paper proposes a motion synthesis framework capable of handling skeletal topologies and rest poses, covering a wide range of objects.\n- The authors augment the Truebones Zoo dataset to include more skeletal variations and high-quality text descriptions. The release of this dataset should faciliate research in related areas.\n- Both quantitative results and qualitative evaluations demonstrate that the proposed method surpasses baselines in terms of generation quality.\n- The paper is well-written, technically correct and good quality.\n\n[Weaknesses]\n\n- No global translation. Due to the model design, all the generated motions are pinned at the origin without global translation of the root joint. It would be beneficial to include discussions on how to generate the global translation, expecially for those non-humanoid skeletons.", "other_comments_or_suggestions": "- As the two-stage training stategy is specially designed for computational resource constraints and data limitations, it would be helpful to include details on the hardware specs used for model training as well as the overall training time.", "questions_for_authors": "n/a", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1kkKet0HSR", "reviewer": "Reviewer_2SeY", "summary": "This paper proposes a novel problem, text-driven motion synthesis of different skeletal structures, constructs a dataset, and develops a new model structure. The key innovation is the explicit incorporation of skeletal configuration information through Tree Positional Encoding (TreePE) and Rest Pose Encoding (RestPE). Experiments show the  method generates realistic, coherent motions from textual descriptions for diverse and even unseen objects, setting a strong foundation for motion generation across diverse object categories with heterogeneous skeletal templates", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the valuable feedback and questions.\n\n### Q1: Typo & Link for the demo.\nWe will correct the typo. For the demo, please visit: t2m4lvo.github.io\n\n### Q2: Human study.\nPlease refer to our response to Q2 of Reviewer qPq2, where we address a similar point.\n\n### Q3: Why choose to use PE to encode the rest pose.\nWe opted not to explore alternative methods, as our primary goal was to preserve the scalability and architectural simplicity of the backbone, Diffusion Transformers (DiTs/SiTs), which are known for their scalable and generalizable design. Positional Encoding (PE) offered a straightforward and effective way to incorporate rest pose information while maintaining full compatibility with our transformer-based framework. That said, we acknowledge that exploring alternative conditioning strategies could be a fruitful direction for future work.\n\n### Q4: Zero-shot inference on human\nWhile our method is designed to accommodate diverse and heterogeneous skeletal templates, we find that it does not extend to human skeletons. This limitation stems from the characteristics of our training data: the Truebones Zoo dataset, even with rig augmentation, does not contain human-like skeletal structures. Additionally, the motion dynamics and textual descriptions in our dataset differ significantly from those in human motion domains.\n\nA natural question is why human motion data was omited in our training set. Our primary objective is to expand motion synthesis to underrepresented non-human species within a unified framework. To this end, we used the Truebones Zoo dataset, which consists of ~1K motion clips over 70 non-human object categories. In contrast, existing human motion datasets often comprise tens of thousands to millions of samples. Incorporating human motion would introduce a significant imbalance, likely skewing the model’s capacity toward human motion and limiting its ability to learn meaningful representations for underrepresented yet a lot more diverse categories.\n\nWe will include a qualitative example and further discussion of this limitation in the revision. In future work, we plan to address this by scaling to larger datasets, such as Objaverse-XL, to construct a more balanced corpus of human and non-human motions. This would enable training a unified model capable of generalizing across a broader spectrum of skeletal structures and motion styles.\n\n### Q5: How to keep the motion quality of the augmented rigs.\nYes, we retarget the original motion to the augmented skeletons, using Blender’s retargeting pipeline.\n\nTo ensure motion quality, we carefully designed the augmentation process to minimize disruption to motion dynamics. Changes like rest pose adjustments and bone subdivision have limited impact, while more sensitive bone length adjustment and bone erasing were applied conservatively.\n- Length adjustments were restricted to 0.8×–1.2× of the original and applied symmetrically when symmetry existed.\n- Bone erasing was limited to distal appendages (toes, head tips, tail ends) and redundant spine/root bones.\n\nUnder these constraints, we visually verified ~10K augmented results and found them suitable for training, contributing to improved generalization across diverse skeletons. \n\nWe acknowledge that these details were not fully described in the original submission, and we will include a more detailed explanation in the revision to clarify the design choices and safeguards used in the rig augmentation process. To further ensure plausibility, exploring quantitative checks (e.g., joint velocity, momentum shifts, foot contact, end-effector deviations), followed by rejection sampling could be a promising direction. Another promising direction is two-stage training: pretraining on augmented rigs to improve generalization and bone manipulation, then fine-tuning on physically grounded data for higher realism.\n\n### Q6: Further ablation study on TreePE vs RestPE\nWe conducted an ablation study to isolate the effects of PEs. This experiment was performed under the same setting as Table 1 in the paper.\n\n| RestPE | TreePE | Rig Aug. | Train |  |  | Test |  |  | Test+ |  |  |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|  |  |  | FID (↓) | R@1 (↑) | Align. (↑) | FID (↓) | R@1 (↑) | Align. (↑) | FID (↓) | R@1 (↑) | Align. (↑) |\n| X | X | O | 0.98 | 0.33 | 0.66 | 2.27 | 0.26 | 0.68 | 0.86 | 0.39 | 0.74 |\n| X | O | O | 0.15 | 0.85 | 0.90 | 1.17 | 0.53 | 0.83 | 0.48 | 0.58 | 0.84 |\n| O | X | O | 0.02 | 0.95 | 0.95 | 0.69 | 0.59 | 0.87 | 0.42 | 0.66 | 0.89 |\n| O | O | O | 0.01 | 0.97 | 0.97 | 0.68 | 0.60 | 0.89 | 0.26 | 0.67 | 0.93 |\n\nThe results confirm that both are beneficial: RestPE has a stronger impact, but the best performance is achieved when both are used. We believe the strong performance of RestPE alone is partly due to the model's ability to implicitly infer parent-child relationships from relative offsets and bone lengths.", "claims_and_evidence": "Overall, the claims are almost supported. However, I still suspect the generalizations. Could the author provide some human motion in a zero-shot way?", "methods_and_evaluation_criteria": "The methods are good. And some human user studies may needed for evaluation.", "theoretical_claims": "No problem here.", "experimental_designs_or_analyses": "Why did the author choose PE to encode the rest pose? Does the author try other ways, like attention?", "relation_to_broader_scientific_literature": "The previous method makes it hard to process different skeletons in one model. This paper proposes to using PE to solve it.", "essential_references_not_discussed": "No problem here.", "other_strengths_and_weaknesses": "Strengths: This paper is interesting and meaningful. The dataset and the architecture design make sense.\nWeaknesses: See the questions.", "other_comments_or_suggestions": "Typo: 1. Page 2, Line 93, 'Similarly' should not be a reference.", "questions_for_authors": "1. Could the author provide the video demos link again? I could not open the link in the main paper PDF file.\n2. Can this model generate human skeleton motions in a zero-shot way?\n3. I did not get the process of rig augmentation. Did the author do retargetting after the augmentation? If yes, how do you keep the motion quality here? If not, could you explain further how to adjust the original motion to the new skeleton?\n4. Could the author separate the TreePE and TestPE ablations to see what actually works?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HMwndlimHg", "reviewer": "Reviewer_qPq2", "summary": "The paper aims to generate motions of different rigs from text inputs. The paper witness the problem in the current 3D content creation community, that it lacks high-quality motion dataset with annotations, and it lacks methods handling hetergeneous skeleton templates. Therefore, the paper present a high-quality text-annotated text-to-object motion dataset, and a motion diffusion model that supports different skeleton templates. Specifically, the paper repsents three rig augmentation methods, and present incorporating Tree Position Encoding to extend MDM incoporating different skeleton structures. Experiment comparing with several component-removed baselines and motion retargetting, proving the effectiveness of the rig augmentation strategy, and generalized motion diffusion models. Experiment with partial data also presents the adapbility to various rigs, motions, and unseen objects.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewer’s detailed review and thoughtful insights.\n\n### Q1: Physical plausibility of the augmented rigs.\nAs noted in our response to Q5 of Reviewer 2SeY, we carefully designed the augmentation pipeline and visually inspected ~10K augmented motions, which we found sufficient at this scale. The resulting skeletons appeared visually plausible and were successfully used for training.\n\nThat said, we acknowledge that these rigs were not derived from physically simulated or animation-ready meshes, and may not be suitable for production use. As discussed in response to Q3 of Reviewer 4SvT, their primary goal was to increase skeletal diversity and improve model generalization—e.g., robustness to unseen rigs or cross-object transfer.\n\n### Q2: Does using the same verb for semantically different motions across objects hurt cross-embodiment generalizability?\nRather than harming generalization, we believe this diversity encourages the model to learn embodiment-aware interpretations of similar textual prompts.\n\nThe key lies in context-awareness. While both model design and data quality are important, we view this primarily as a data-centric challenge—facilitated by the adoption of a proven architecture (e.g., DiTs)—and best addressed through exposure to diverse, high-quality, and context-rich examples.\n\nTo this end, we provide multi-level captions that describe actions, part-level dynamics, and initial poses. As shown in Section 7.3, these annotations help the model learn contextualized verb semantics and produce more structured motions.\n\nAs a promising direction, scaling up with diverse resources like Objaverse-XL (~100K animated 3D meshes), along with automated pipelines for multi-level captioning, could greatly enhance cross-embodiment generalization through fine-grained contextual understanding. We opt to explore this as our future work.\n\n### Q3: Additional evaluation in terms of the quality of motion.\nAs suggested, we conducted additional evaluations on the test sets from Figure 4-(b), where all baselines are available.\n\nFor motion smoothness, we used the Motion Stability Index (MSI) [1]; higher values indicate smoother, more stable motion. GT scored 12.01e3. Results:\n- Retargeting: 9.93e3, Ours: 8.78e3, NoAug: 7.39e3, GPT-Caption: 7.30e3, SO-MDMs: 7.21e3.\n\nOur method achieves the highest MSI among data-driven baselines, closest to GT. Retargeting scores highest overall due to direct motion transfer but lacks flexibility for novel prompts.\n\nWe also conducted a user study with 30 participants, who selected the most caption-aligned motion. Results:\n- Ours: 65.6%, NoAug: 12.4%, SO-MDMs: 10.667%, Retargeting: 9.067%, GPT-Caption: 2.267%. \n\nThese results show that our method produces smooth and semantically accurate motions, both quantitatively and perceptually. We will include them in the revision.\n\nReferences:\n\n[1] Kim et al., Audio-driven Talking Face Generation with Stabilized Synchronization Loss, CVPR 2023.\n\n### Q4: Essential references not discussed & details on RestPE / Figure 1.\nWe will include the relevant references in the revised manuscript. Details on RestPE will be added to the supplementary material. Also, the examples in Figure 1 are from the dataset (not generated); we will clarify this to avoid confusion.\n\n### Q5: Word stats to support “large-vocab” claim.\nIn our work, “large-vocab” refers to the diversity of object categories and skeletal structures, 70+ distinct categories with unique topologies, rather than textual or action diversity within a fixed skeleton, as in prior human motion literature.\n\nThat said, we appreciate the suggestion and will include text-level stats in the revision. As a preview:\n- Avg. words per caption: short (12.2), mid (29.5), long (56.2)\n- Verb counts: short (302), mid (495), long (707)\n- Noun counts: short (402), mid (560), long (1000)\n- Top verbs (short): stand (145), walk (99), raise (95), lower (75), strike (73)\n- Top nouns (short): head (223), body (177), tail (165), leg (150), ground (107)\n\nFor reference, the full list of object categories is also provided in Appendix Table 2, which may help clarify the intended meaning of “large-vocab” in our current draft.\n\n### Q6: About normalizing scales of different creatures.\nWe apply size normalization, motivated by early observations that it led to faster loss convergence during training.\n\nNonetheless, we believe normalization does not hinder the model’s ability to learn size-specific motion patterns when such cues are present in the data. For example, even if a small and large cat are normalized to the same scale, their motions may still differ in agility, stride, or posture—captured through joint rotations, or temporal dynamics and timing. Although our annotated captions do not specify size explicitly, skeletal topology and relative proportions are preserved, providing structural cues that help the model infer such differences.", "claims_and_evidence": "The claims are well supported.", "methods_and_evaluation_criteria": "+ My major concern here is on the physical playsibility of the augmented rigs. As these rigs are augmented by language models, it could potentially breaks the physical plausibility of the original rigs (i.e. uneven lengths feet leads to unstable standing, bones might intersect). Does the paper provide evaluation for these properties? I acknowledge that even though without physical playsibility, a MDM can still benefit from the augmented skeletons. However, are these augmented rigs really animation-usable?", "theoretical_claims": "+ Since different objects might behaves drastically different with the same verb, (i.e. how scorpion and horse attack looks drastically different as shown in Figure 10), will these actually harms the cross embodiement generalizability of the MDM?", "experimental_designs_or_analyses": "+ The paper compares the baseline with other methods using 'pose-level' encoders but not 'motion-levels'. Though I acknowledge that it's hard to train such a motion-level encoder, but the paper needs to provide evaluation in terms of the quality of motion, such as motion smoothness, or even perceptual study.", "relation_to_broader_scientific_literature": "+ The dataset would be the major contribution, which might interesting both animation and robotics community. \n+ The cross-skeleton MDM model is also interesting.", "essential_references_not_discussed": "Please include these works that has the ability to generate character motion with different rigs, or overlap with the rigging augmentation techniques proposed in the paper. \n(1) CharacterMixer: Rig-Aware Interpolation of 3D Characters\n(2) SkinMixer: Blending 3D Animated Models\n\nPlease also include some paper using surface-representation or multi-view image representation for text to motion synthesis. Here I name some as an example:\n(1) L4GM: Large 4D Gaussian Reconstruction Model\n(2) DreamGaussian4D: Generative 4D Gaussian Splatting", "other_strengths_and_weaknesses": "+ I can see the paper, especially the rigging augmentation part, might also have broader impact in articulated object augmentation, that might benefit 3D design or robotics community.\n+ The paper would also be interesting in studying skill transfer across embodiments, which could might interested robotics community.", "other_comments_or_suggestions": "+ The paper using the phrase \"large-vocab\" to describe the text-to-motion dataset. However, detailed stats of word frequency and word count are not provided. \n+ Please specify the RestPE in the supplementary material, if it is not provided.", "questions_for_authors": "+ Are examples in figure 1 generated results or examples from the dataset? Please specify. \n+ The dataset uniformed all skeletons as mentioned in section 4.1. However, since different creatures has different sizes, will uniforming object sizes confuses GPT or any text-related components, i.e., text-to-motion model?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The work received very supportive reviews from the reviewers even prior to the rebuttal. They acknowledged that the manuscript presents a significant advancement in motion synthesis. All four reviewers agreed that the paper is a valuable contribution. The authors provided a strong rebuttal and successfully addressed the remaining concerns. Therefore, the AC recommends acceptance. Congrats!
Decision: Accept (poster)