DartControl: A Diffusion-Based Autoregressive Motion Model for Real-Time Text-Driven Motion Control
Authors: Kaifeng Zhao
Gen Li
Siyu Tang
Venue: ICLR 2025 Spotlight (2025)
Keywords: Human Motion Generation
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We present DartControl, a method for high-quality, real-time motion generation from streaming text inputs. By incorporating latent space control, our method further enables diverse motion generation tasks requiring spatial control.

Abstract: Text-conditioned human motion generation, which allows for user interaction through natural language, has become increasingly popular. Existing methods typically generate short, isolated motions based on a single input sentence. However, human motions are continuous and can extend over long periods, carrying rich semantics. Creating long, complex motions that precisely respond to streams of text descriptions, particularly in an online and real-time setting, remains a significant challenge. Furthermore, incorporating spatial constraints into text-conditioned motion generation presents additional challenges, as it requires aligning the motion semantics specified by text descriptions with geometric information, such as goal locations and 3D scene geometry. To address these limitations, we propose **DartC**ontrol, in short **DART**, a **D**iffusion-based **A**utoregressive motion primitive model for **R**eal-time **T**ext-driven motion **C**ontrol. Our model, DART, effectively learns a compact motion primitive space jointly conditioned on motion history and text inputs using latent diffusion models. By autoregressively generating motion primitives based on the preceding history and current text input, DART enables real-time, sequential motion generation driven by natural language descriptions. Additionally,  the learned motion primitive space allows for precise spatial motion control, which we formulate either as a latent noise optimization problem or as a Markov decision process addressed through reinforcement learning. We present effective algorithms for both approaches, demonstrating our model’s versatility and superior performance in various motion synthesis tasks. Experiments show our method outperforms existing baselines in motion realism, efficiency, and controllability. Video results and code are available at https://zkf1997.github.io/DART/.

Published as a conference paper at ICLR 2025
DARTCONTROL: A DIFFUSION-BASED AUTOREGRES- SIVE MOTION MODEL FOR REAL-TIME TEXT-DRIVEN MOTION CONTROL
Kaifeng Zhao, Gen Li, Siyu Tang
ETH Zürich
(kai feng. zhao, gen. 1i, siyu.tang) wwinffttz..c.m
ABSTRACT
rext-conditioned human motion generation, which allows for user interaction through natural language, has become increasingly popular. Existing methods typically generate short, isolated motions based on single input sentence. How- ever, human motions are continuous and extend over long periods, carrying rich semanties. Creating long, complex motions that preciscly respond to streams of text descriptions, particularly in an online and real-time setting, remains a significant challenge. Furthermore, incorporating spatial constraints into text-conditioned motion generation presents additional challenges, as requires aligning the mo- tion semanties specified by text descriptions with geometric information, such goal locations and 3D scene geometry. To address these limitations, we propose DartControl, in short DART, a Diffusion- model for Real-time Text-driven motion Control. Our model effectively learns a compact motion primitive space jointly conditioned on motion history and text inputs using latent diffusion models. By autoregressively generating motion primi- tives based on the preceding motion history and current text input, DART enables real-time, sequential motion generation driven by natural language descriptions. Additionally, the learned motion primitive space allows for precise spatial motion control, which formulate either a latent noise optimization problem as Markov decision process addressed through reinforcement learning. We present effective algorithms for both approaches, demonstrating our model's versatility and superior performance in various motion synthesis tasks. Experiments show our method outperforms existing baselines in motion realism, efficiency, and controlla- bility. Video results and code are available at bttps://2kf19997.itubbbb git.b...tbbbbb.AARR
Autoregressive motion primitive
INTRODUCTION
Text-conditioned human motion generation has gained increasing popularity in recent years for flexible user interaction via natural languages. Existing text-conditioned motion models (Tevet et al. 2023] Guo et al., 2024) /Zhang et al. 2023a Guo et al., [022 Jiang et al.] 2024a) primarily focus on generating standalone short motions from a single descriptive sentence. These methods f'al to accurately generate long and complex motions composed of multiple action segments, where each segment conditioned on distinct action descriptions. FlowMDM (Barquero al.) 2024) is the stute-of-the-as temporal motion composition method, capable of generating complex, continuous motions by composing desired actions with precise adherence to their specified durations. However, FlowMDM is an offline method that requires prior knowledge of the entire action timeline and has slow generation speed, making it unsuitable for online and real-time applications.
In addition to text-based semantic control, generating human motion within spatial constraints and achieving specific goals, such as reaching a key keyframe body pose, following a joint trajectory, or interacting with objects, has broad applications but introduces additional complex challenges. Recent works (Shafir et al. 2024 Karunratanakul et al.) 2024b Xie et al. 2024) have sought to integrate text-conditioned motion models with spatial control capabilities. However, they still face challenges in effectively balancing spatial control, motion quality, and semantic alignment with text. Moreover, these approaches are typically restricted to controlling isolated short motions in offline setting.

---

Published as a conference paper at ICLR 2025
Meanwhile, interactive character control (Kovar et al. 2008 Holden al. 2013 Ling al. 2020 Peng et al. 2022) has been a longstanding focus in computer graphics, with a primary emphasis on achieving motion realism and responsiveness to interactive control signals. However, most of these works lack support for text-conditioned semantic control and are limited by being trained on small, curated datasets. Incorporating eext-conditioned motion generation could provide novel intuitive language interface for animators and everyday users to control the characters, reducing the effort required when specifying detailed spatial control signals is challenging or tedious.
To address these limitations, we propose DART, a diffusion-based autoregressive motion primitive model for real-time text-driven motion control. Moreover, the compact and expressive motion space of DART provides a foundation for integrating precise spatial control through latent space optimization or reinforcement learning (RI.)-ased policies. DART features three key components.
First, DART represents human motion as collection of motion primitives (Zhang & Tang (022). which are autoregressive representations consisting of overlapping short motion segments tailored for online generation and control. These short primitives also provide more precise alignment with atomic action semantics compared to longer sequences, enabling effective learning of a text- conditioned motion space. By focusing on shorter primitives, DART avoids the complexities and extensive data demands of modeling entire motion sequences, allowing for high-quality motion generation with only few diffusion steps.
Next, DART learns a text-conditioned autoregressive motion generation model from large-scale data using latent diffusion architecture, which contains a variational autoencoder for learning compact latent motion primitive space, and a denoiser network for generating motion primitives conditioned on texts and history. Leveraging the trained denoiser and decoder models, DART employs an autoregressive rollout to synthesize motion sequences from real-time text inputs, enabling the efficient generation of motions of arbitrary length. Compared with the offline temporal motion composition method FlowMDM, DART provides real-time response and 10x generation speed.
Lastly, we introduce latent space control framework based on DART for spatially controllable motion synthesis, leveraging its learned space of realistic human motions to ensure high-quality generation. We present effective optimization and learning algorithms that explore the latent diffusion noise space to synthesize motion sequences that precisely follow both textual and spatial constraints. We evaluate DART across various motion synthesis tasks, including generating long, continuous sequences from sequential text prompts, in-between motion generation, scene-conditioned motion, and reaching synthesis. The experimental results show that DART is a simple, unified and highly effective motion model, consistently outperforming or matching the performance of baselines.
2 RELATED WORKS
Conditional Motion Generation. Generating realistic and diverse human motions is long-standing challenge in computer vision and graphics. Apart from generating highly realistic human motions
(Kovar et al. 2008 Holden et al. 2020 Clavet et al., 2016 Zinno 2019). conditional generation is
another important factor that aligns motion generation with human intentions and various application
constraints. Text-conditioned motion generation (Tevet et al.) 2023] ZZhag et al. 2022a Petrovich
al.. 2022] Guo et al., 2022] Jiang et al., [224a]] Zhang et al. (202a)) has become increasingly
popular since it allows users to modulate motion generation with flexible natural languages. Audio
and speech-driven motion synthesis methods (Alexanderson et al., 2023 Tseng et al.) 2023 Siyao
et al. 2022, Ao et al., 2022 2023) have also made significant progress recently. Moreover, there exist many applications that require spatial awareness and precise control in motion generation, such
as interactive character control (Ling et al. 2020 |eng et al. [022,] Starke et al. [022] [uo et al.
2024
Starke et al.] 2024), human-scene interactions(Haassn et al. |021] | [ttarke et al. 20 Zhao
ct al.
2022 2023 Li al. 2024a] Xu et al. [2023] Jiang al. [222bb] Wang al. [2024] Liu et al.
2024
Li al. 2024b Zhang et al. 2022b: 2025), and human- human noidd) interactions (Liang et al.
2024
Zhang et | 2023c Christen et al.) 2023] Cheng et al. 2024 Shan et al. 2024). Synthesizing
high-quality motions with precise spatial control remains challenging, and DART is a step toward a
general and efficient motion model that supports precise control tasks.
Diffusion Generative Models. Denoising Diffusion Models (Ho et al., 2020 Song et al. 202 10 b) generative models that learn to predict clean data samples by iteratively annealing the noise from
2

---

Published as a conference paper at ICLR 2025
standard Gaussian sample. Diffusion models have achieved unprecedented performances in many generation tasks including images. videos, and 3D human motions (Tevet et al.) 2023 Rombach
ctt al. 2022 Ho et al. 2022). Diffusion models
modulate the generation, such as text prompts, images, audio, and 3D objects (Rombach et al., 2022 Tevet et al.
accept flexible conditions
2023
Alexanderson et al. 2023 [Teng et al. 2023 Zhang et al.) 2023b Xu al. 2023 et al.
2023).
motion generation methods focus on offline generations of short. isolated motion sequences while neglecting the autoregressive nature of human motions (Teeet et al.
Most existing diffusion-ber
2023;
Barquero et al., 2024 Chen et al., 2023b Karunratanakul et al. 2024b) Cohan et al. 2024
Dai et 2025 Chen al. 2023a). Among these methods, DNO (Karunratanakul et al. 2024b)
closely related to our optimization-based control approach, as both use diffusion noises as the latent space for editing and control. However, the key distinction lies in our latent motion primitive- based diffusion model, which, unlike DNO's diffusion model trained with full motion sequences in explicit motion representations, achieves superior performance in harmonizing spatial control with text semantic alignment during experiments. There are also works that incorporate history
conditions in diffusion- baaed motion generation and capture (Xu et al., 2023 Jiang et al.) 2024b
Van Wouwe et al.] 2024 Han et al. 2024). [Shi et al. (2024) and et al. (2024) adapt diffusion
models for real-time character motion generation and control. While et al. (2024) learns character
control policies using diffusion noises the action space, akin to reinforcement- leaeeett-eeaann-baaaaaae
control, their method focuses on single-frame autoregressive generation and lacks support for text conditions, which offers a compact and intuitive interface for users control character behaviors. In contrast, DART is an efficient and general motion model that scales effectively to large motion-text datasets. DART supports natural language interfacing and provides a versatile foundation for various motion generation tasks with spatial control. Concurrent work, CloSD (Tevet et al. (0255). trains autoregressive motion diffusion models conditioned on target joint locations to guide the human motion towards these goals. This approach relies paired training data of control signals and human motions. In contrast, DART learns a latent motion space from motion only data and introduces latent space control methods to achieve flexible control goals without the need for paired training data.
3 METHOD
3.1 PRELIMINARIES
Problem Definition. We focus on the task of text-conditioned online motion generation with spatial control. Given an H frame seed motion H [h ...., h",, sequence of N text prompts C C= [cl "..,,...)))) c..,,""),, and spatial goals g. the objective to autoregressively generate continuous and realistic human motion sequences M = Heedd X ..... X, where each motion segment X matches the semantics of the corresponding text prompt c' and satisfies the spatial goal constraints g. This task imposes challenges in high-level action semantic control, precise spatial control, and smooth temporal transition in motion generation.
Autoregressive Motion Primitive Representation. We model long-term human motions as the sequential composition of motion primitives (Zhang & Tang 2022) with overlaps for efficient generative learning and online inference. Each motion primitive p H X*) is a short motion clip containing H frames of history motion H [h h that overlap with the previous motion primitive, and F' frames of future motion X' = [x ..., x.. The history motion of the i-th motion primitive H' consists of the last H frames of the previous motion primitive - 1- -H-1: 1:F Therefore, infinitely long motions can be represented as the rollout of such overlapping motion primitives as M = Haeed: X ..... X To represent human bodies in each motion frame, we use an overparameterized representation based on the SMPL-X (Pavlakos et al. 2019) parametric human body model. Each frame is represented as D = 276 dimensional vector including the body root translation t, root orientation R. local joint rotations 0, joint locations J, and the temporal difference features of locations and rotations. Each motion primitive is canonicalized in local coordinate frame centered at the first-frame body pelvis. We use history length H = 2 and future length F' = 8 in our experiments. Further details of the primitive representation are attached in the Appendix A For brevity, we omit the primitive index superscript i when discussing in the context of a single primitive.
In contrast to directly modeling long motion sequences, the primitive representation decomposes globally complex sequences into short and simple primitives, resulting in a more tractable data distribution for generative learning. Moreover, the autoregressive and simple nature of the prim-
3

---

Figure 1: Architecture illustration of DART. The encoder network compresses the future frames X [x....,,x' x.., xf into a latent variable, conditioned on the history frames = [h ..... h The decoder network reconstructs the future frames conditioned on the history frames and the latent sample. The denoiser network predicts the clean latent sample 2n conditioned on the noising step, Published as a conference paper at ICLR 2025
Positional encoding Addition Z
Fix weights
Linor
Transherrer Encader
PE
PE
Loce
Naata
Dianbeton
Zaro vkkes
liiuary
Thrara Tranas
Hissey fuano
Tex Teerp
Encoder 3
Decoder D
Latent Denoiser
text prompt, history frames, and noised latent sample z,. During the denoiser training, the encoder and decoder network weights remain fixed.
itive representation makes it inherently suitable for fast online generations. Furthermore, motion primitives convey more interpretable semantics than individual frames, enhancing the learning of text-conditioned motion space.
3.2 DART: A DIFFUSION- BASED AUTOREGRESSIVE MOTION PRIMITIVE MODEL
We propose latent diffusion model (Rombach et al. 2022 Chen et al. 2023b designed for seamless autoregressive motion generation, conditioned on text prompts and motion history. The proposed model contains variational autoencoder (VAE) (Kingma & Welling 2014) that compresses the motion primitives into a compact latent space and latent denoising diffusion model that predicts clean latent variables from noise, conditioned on text prompts and motion history.
Learning the Latent Motion Primitive Space. We introduce motion primitive VAE that com- presses motion primitives into compact latent space, upon which we train our latent diffusion models, rather than using the raw motion space (Rombach et al. 2022 Chen et al. 2023b). The design of learning a compressed latent space of motion primitives is inspired by the observations that raw motion data from the used motion capture dataset AMASS (Mahmood et al. 2019 often contain various levels of artifacts including glitches and jitters, and training diffusion models raw motion space leads to results inheriting such artifacts. This is evidenced by the significantly higher jittering in generated motions of the ablative model without VAE in Appendix. [.3] The compression achieved through the motion primitive VAE significantly mitigates the impacts of these outlier artifacts in motion data. The resulting latent representation is not only more compact but also more computationally efficient than the raw motion data, thereby enhancing the efficiency of our generative model and improving the control capabilities within the latent space.
Our motion primitive VAE employs transformer-basee architecture based on MLD Chen et al. 20236), comprising an encoder & and decoder D, as shown in Fig. [ The encoder takes as input the history motion frames H and future motion frames X as well the learnable distribution tokens T and T., which are responsible for predicting the distribution mean and variance. The latent sample is obtained from the predicted distribution via reparameterization (Kingma & Welling 2014). The decoder then predicts the future frames X from zero tokens conditioned on the latent sample z and the history frames H. The motion primitive VAE is trained with the future frame reconstruction loss Lrec. auxiliary losses Laue that penalize unnatural motion reconstruction, small Kullback-Leibe (KL) regularization term LKL, and additional SMPL-based reconstruction and regularization terms. We refer Appendix C for further details about the motion primitive VAE.
4

---

Published as a conference paper at ICLR 2025
Algorithm 1 Autoregressive rollout generation using latent motion primitive model
Input: primitive decoder D, latent variable denoiser G, history motion seed Heeeds text prompis C [c ,..., e total diffusion steps T, classifier-free guidance scale w, diffusion sampler S.
Optional Input: Latent noises Zr = [2] (....... 2
Output: motion sequence M
H <- Haeed
M + Heed
for i <- to N do
number of rollouts
sample noise z from (0, 1) if not inputted
ž f -S(g, z7. T, H. c', w))
diffusion sample loop with classifier free guidance
X < DHH, 20)
M CONCAT(M, X)
concatenate future frames to generated sequence update the rollout history with last H generated frames
H < CA
end for return M
Latent Motion Primitive Diffusion Model. Building on the compact latent motion primitive space, we formulate text-conditioned autoregressive motion generation via a probabilistic distribution
(2 c), and train latent diffusion model 9 to approximate it. Diffusion models (Ho al. 2020) are generative models that learn data distributions by progressively reversing a forward diftusion process, which iteratively adds Gaussian noise data samples until they approach pure noise (O I). Given a motion primitive sampled from the dataset and its latent representation Zo obtained using the encoder E, the forward diffusion produces a sequence of increasingly noisy samples Zi...., , by iteratively adding noises as: g(z, ( = N(V1 3,2,-1, 3,I), where 33 are noise schedule hyper parameters. The denoiser model learns the reverse process po ( 1 t, H, c) N E,) for generating motion primitives conditioned the motion history H and text label of the motion primitive. The vairance E, of the reverse process distribution is scheduled using hyper parameters, while the mean μu is modeled using denoiser neural network. We design the denoiser model g to predict the clean latent variable 20 = G(z,, t, H, c), from which μe, can be derived as follows:
Var- 3 g(z,, H.C) Var(l (a1(1- -a-1)) at
[1
Ze
1 -,t
where 04 1 3, and ā₁ the denoiser model to predict the clean variable 2u, which is subsequently diffused to a lower noise level This denoising process iterates until a clean sample zu is obtained.
= - Og,, During generation, initialize with Zr (O, I) and use
The denoiser model architecture is shown in Fig. The diffusion step t is embedded using a small MLP, while the text prompt is encoded using the CLIP (Radford et al. (02T) text encoder. The text prompt is randomly masked out by probability of 0.1 during training to enable classifier- free guidance (Ho & Salimans, 202T) during generation. The cleaned latent variable 2n can be converted back to the future frames using the frozen decoder D: X = DHH, (2) We train the latent denoiser G using the simple objective (Ho et al. 2020). Additionally, we apply the reconstruction loss Lreec and auxiliary losses Lauz the decoded future frames X. Notably,
use only 10 diffusion steps for both training and inference. This small number suffices for realistic sample generation due the simplicity of motion primitive representation, enabling highly efficient online generation. Moreover, we use the scheduled training (Ling et al.) 2020 Bengio et al. 2015
Rempe et al. 202I) to progressively introduce the test test-time distribution of the history motion H. which improves the stability of long sequence online generation and the text prompt controllability for unseen poses. We refer to Appendix for the details. With the trained motion primitive decoder D, latent denoiser G and diffusion sampler S such as DDPM and DDIM (Ho et al.) 2020 [oog cet al. 2021a), we can autoregressively generate motion sequences given the history motion seed Hoeed and the online sequence of text prompts C, as shown in Alg. [ During sampling, we use classifier-fres guidance (Ho & Salimans) 2021) on the text condition with a guidance scale w: (z, c) G(z,, t. H. 9) + (g(z,, H. c) ((z,, t, H. 0)). Using the rollout algorithm, DART generates over 300 frames per second using a single RTX 4090 GPU, enabling real- sal-timee applications and online reinforcementleamming control as in Sec. 3.3
5

---

Published as a conference paper at ICLR 2025
Algorithm 2 Latent noises optimization
Input: Latent noises ZT [z], ...; x Optimizer O learning rate 1), and goal g. (For brevity,
we do not reiterate the inputs of the rollout function defined in Alg. and criterion terms in Eq. Output: a motion sequence M. for i f- to optimization steps do
M f ZZr, Heeed C)
V Vz, (F), 9) cons cons(M)
ZT t O(Zr, V lI:
D update using normalized gradient
end for
return M + ROLLOUT(ZT: H Hseed; C)
3.3 SPATIALLY CONTROLLABLE MOTION SYNTHESIS VIA DART
Text-conditioned motion generation offers user- user-friendly interface for controlling motions through natural language. However, relying solely on text limits precise spatial control, such as walking to specific location or sitting in a designated spot. Therefore, it is necessary to incorporate motion control mechanisms to achieve precise spatial goals, including reaching a keyframe body pose, following joint trajectories, and interacting with scene objects. We formulate the motion control task as generating the motion sequence M that minimizes its distance to a given spatial goal g under a criterion function -(),)) and the regularization from the scene and physical constraints cons(:):
M argmin .) + cons(M),
where g is the task dusk-dependent spatial goal (e.g., a keyframe body for motion in-between tasks or a target location for navigation tasks), [I.) is the projection function that extracts goal-relevant features from motion sequences and maps them into the task-aligned observation space, and cons(.) denotes physical and scene constraints, such as preventing scene collisions and floating bodies.
Directly solving the motion control task in the raw motion space often results in unrealistic motions since most samples in the raw motion space do not represent plausible motions. To improve the generated motion quality, many previous methods tackle such motion control tasks in a latent motion space, where samples can be mapped to plausible motions (Karunratanakul et al., 2024b Ling et al.. PPng et al. 2022 Holden et al. 2015). DART offers powerful coxt-conditioned latent motion space for such latent space control, as it learns generative model capable of producing diverse and realistic motions from standard Gaussian samples. Using the deterministic DDIM (Song et al.) 2021a) sampler, we adapt DART sampling to function as a deterministic mapping from latent noises Zr to plausible motions. This allows to reformulate the motion control task in Eq, 2 as latent space control problem as follows: Given the initial motion history H Heedd a sequence of text prompts C. the pretrained DART models, and deterministic diffusion sampler S, the rollout function in Alg. 1 can deterministically map a list of motion primitive latent noises Z = [z/ ... 2 sequence conditioned on the history motion seed and text prompts: M ROLLOUT(ZT, (Z,, H Haceds C)). The minimization objective is converted as:
to a motion
Zr argminz,, (ROLLLOLU Hseed, C)), + cons(ROLLOUT((Z, Hoeed, C")
(3)
Note that we do not use DDIM to skip diffusion steps at sampling, which we observe to cause artifacts in generated motion. We then propose two solutions to this latent space motion control problem. one to directly optimize the latent noises using gradient descent, and the other is to model the control task as Markov process and use reinforcement learning to learn control policies.
Motion Control via Latent Diffusion Noise Optimization. One straightforward solution to this minimization problem (Eq. 3) is to directly optimize the latent noises Zr given the criterion function using gradient descent methods (Kingma & Ba 2015; Karunratanakul et al., (024a)). The latent noise optimization is illustrated in Alg. 2 This optimization-aase
control framework is general and applicable for various spatial control tasks. We instantiate the latent noise optimization method in two example control scenarios: in be-between motion generation and human-scene interaction generation.
First, we address the motion in-between task that aims to generate the motion frames transition between given history and goal keyframes, g that is frames away conditioned on the text prompt c.

---

Published as a conference paper at ICLR 2025
Citc
DART
Futzo Modse
Lioce
Looce
Lioce
Liwa
CLIP
Hiiaay Notknn
Goal 5"
Stere
pooege
Figure 2: Architecture of the reinforcement learning- based control policy. The pretrained DART diffusion denoiser and decoder models transform the latent actions into motion frames. The last predicted frames are canonicalized and provided to the policy model as the next step history condition.
We use the distance between the f-th frame of the generated motion and the goal keyframe as the optimization objective. Second, we show that physical and scene constraints cons(-) can be incorporated to synthesize human motions in contextual environment. Given input 3D scenes, text prompts C, and spatial goals of the interaction anchor joint locations, e.g., locations of the pelvis when sitting, the objective is to generate motions that not only perform the desired interaction but also achieve the goal joint positions while adhering to scene constraints. During the optimization, the 3D scenes are represented as signed distance fields (SDF) to compute body-scene distances, which serve as the basis for deriving comman-scens contact and collision metrics that encourage foot-floor contact and scene collision avoidance, as detailed in Appendix. [F]
Motion Control via Reinforcement Learning. Although the proposed latent noise optimization is effective for general control tasks, the optimization can be computationally expensive. To address this, the autoregressive primitive-based motion representation of DART allows for another efficient control mechanism using reinforcement learning (RL) (Sutton Barto 1998). We model the latent motion control Markov decision process with latent action space and use RL to learn policy models to achieve the goals. We model digital humans as agents to interact with an environment according to a policy maximize the expected discounted return. At each time step the agent observes the state s' of the system, samples an action a' from the learned policy, with the system transitioning to the next state s++1 due to the performed action at, and receives reward pf r(s', at, 1+1)).
Our latent motion primitive model naturally fits in the Markov decision process due to its autore- gressive nature. We use the latent noises ZT as the latent action space and train goal-conditioned policy models as controllers. The policy architecture is shown in Fig. 2 Our policy model employs the actor-critic (Sutton & Barto 1998) architecture and is trained with the PPO (Schulman al. 2017) algorithm. The state s' includes the history motion observation H', goal observation g', scene observation s², and the CLIP embedding of the text prompt c². The policy model takes in [H', g², s", c'] to predict the latent noise 2i as the action. The latent noise 2/7 is mapped to the future motion frames X using the frozen latent denoiser G and motion primitive decoder D. The new history motion extracted from the last // predicted frames and fed the policy network in the next step. We reformulate the minimization problem in Eq. as reward maximization to train the policy.
We instantiate the reinforcement learning control with the text-conditioned goal-reaching task. Given a text prompt and 2D goal location we aim to control the human to reach the goal location using the action specified by the text. The goal location transformed into local observation, which includes its distance to the body pelvis at the last history frame and its local direction within the human-centin is the relative floor height to the body pelvis at the first history frame. The policy is trained with distance rewards encouraging the human pelvis to reach the goal location and scene constraint rewards penalizing foot skating and floor penetration. Further details can be found in appendix G With the trained control policies, we can efficiently control a human to reach dynamic goals using specified skills like walking or hopping.
coordinate frame. We consider a simple flat scene and the scene observation
7

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTS
We provide extensive experiments showing how DART can serve as general model for text- conditioned temporal motion composition (4.1) and various motion generation tasks requiring precise
spatial control via latent noise optimization (4.2) and reinforcement learning policy (4.3). Qualitative
results and comparisons are available in the supplementary videos on the project page
Our DART is trained on motion-text data from the BABEL (Punnakkal et al. 2021] dataset in our experiments if not otherwise stated. BABEL contains motion capture sequences with frame-aligned text labels that annotate the iin-grained semantics of actions. Fine- -ene-graindd text labels in BABEL allow models learn precise human action controls and natural transitions among actions. However, DART can also learn using motions with coarse sequence-level labels such as the HML3D (Goo
al. 2022) dataset, in the optimization-based motion in-between experiments in Sec. 4.2
4.1 TEXT-CONDITIONED TEMPORAL MOTION COMPOSITION
Text-conditioned temporal motion composition aims to generate realistic motion sequences that faithfully align with list of action segments, each defined by a specific text prompt and duration. We evaluate the motion composition task on the DABEL/Punnakkee et al. (202]) dataset consisting of motion capture sequences with human-annotated per-fram frame action descriptions, which facilitate the evaluation of precise action controls and natural transitions in motion composition. Since BABEL does not release the test set, we compare our DART with baseline methods on the BABEL validation set. We extract the list of action segments described by tuples of text prompts and durations from each data sequence and feed the action lists as conditions for motion composition.
We evaluate the generation results using metries proposed in Guo et al. (2022) and Barquero et al. (2024). For each action segment, we evaluate the similarity between generation and dataset (FID), motion-text semantic alignment (R-prec, MM-DIST), and generation diversity (DIV). To evaluate smooth transitions between two segments, we measure the jerk (the derivative of acceleration) of the 30-frame transition clip centered at the splitting point of two action segments, reporting the peak jerk (PJ) and Area Under the Jerk (AUJ). Moreover, we profile all methods in a benchmark of generating one 5000-frame-long sequence and report the generation speed, the latency of getting the first generated frames, and memory usage. We also conduct human preference studies to evaluate motion realism and motion-text semantic alignment of generation results, during which the participants are given generation results from two different methods and are asked to select the generation that is perceptually more realistic or better aligns with the action text stream in subtitles. We compare DART with baselines inlcuding TEACH (Athanasiou et al.] 2022). Double'Take (Shafir et al.) (024]). history-conditioned modification of T2M-GPT (Zhang et al. [ as T2M-GPT*) and the state-of-the-art offline motion composition method FDowMDM (Barquero et al., 2024).
We present the quantitative results in Tab. and Tab. 2 DART achieves the best FID in both the segment and transition evaluation, indicating the highest similarity the dataset and best motion realism. DART also displays second-best jerk metrics indicating smooth action transitions. We observe that DART performs slightly worse than FlowMDM in motion-text semantic alignment (R-prec and MM-Dist) because of the online generation nature of DART. Natural action transitions require time transit the new action after receiving the new action prompt, leading to delay the emergence of the new action semantics. For instance, a human cannot immediately transition from kicking leg in the air stepping backward without first recovering a standing pose. This inherent delay in transitions results in a motion embedding shift that impacts the R-prec metric of DART. In contrast, the offline baseline FlowMDMM generates the entire sequence as a whole and requires oracle information of the full timeline of action segments to modulate compatibility between subsequent segments. The slight but natural action transition delay of DART is perceived as natural by humans, as shown in the preference study results in Tab. 2 FlowMDM, for both motion realism and motion- text semantic alignment in human evaluations.
DART is preferred over all the baselines, including
DART requires significantly less memory than the offline baseline MDM and achieves approx- imately 10x faster generation, with a frame rate exceeding 300 and a latency of 0.02s, enabling real-time text-conditioned motion composition (see supplementary video). We refer to Appendix [ for details of experiments, user studies, and ablation studies about model architecture and hyperpa- rameters, and refer to the supplementary video for qualitative comparisons.
8

---

Published as a conference paper at ICLR 2025
Table 1: Quantitative evaluation results on text-conditioned temporal motion composition. The first row includes the metrics of the dataset for reference. Symbol '->" denotes that closer to the dataset reference better and 't' indicates the 95% confidence interval. Bold and blue texts indicate the best and second best results excluding the dataset, respectively.
Segment
Transition
Profliigg Lotencyts)
FI24
R-prect
DIV
MM-Disr
PIDL
DIV
Pl
AUJ 4
Specdiframosst
Mcm.(MiB
Datanel
0.00
0.72
84424015
3.36
00000000
6,20
0.0200000
0.00:00000
TEACH
0..5
0.66::0000
100220006
5.8
3.89:0055
5,4500.77
1.994:001
58
3800:
0.05=00.00
DoubleTake
79220.13
0.502:001
16
5.59
3.56::003
6.06=0:66
0.301:0000
1:23:0011
1
59. 00776
1474
TIM-GPT
0.75
0.49:00011
8.89
6.69
2.5140044
6.61-6022
14440000
4:10:000
885
0.23 =0000
2172
RowMDM
5 581#0.10
0,7700000
8,90=0006
5.08
2 2.354001
6.63=0:88 0,04:0.00
0.11+0000
51
161.2940.24
11892
Ours
F.00
F305
0.00-0.33 0,06+-00000 0:2000000
334
0:00
Table 2: Human preference study results comparing our method against baselines in generation realism and motion-text semantic alignment on lext-conditioned temporal motion composition. We report the percentage of each method being voted better than the other (Ours vs. Baselines).
Realism (%8
Semantic (%)
Ours TEACH
66.7 33.3
66.0 34,0
Ours vs. DoubleTake
66.4 vs. 33.6
66.1 vs. 33.9
Ours vs. T2M-GPT
61.3 38.7
66.7 33.3
Ours vs. How
53.3 vs. 46.7
51.3 vs. 48.7
4.2 LATENT DIFFUSION NOISE OPTIMIZATION-BASEI CONTROL USING DART
Text-conditioned motion in-between. Motion in- ne-beeweennng aims to generate realistic motion frames that smoothly transition between a pair of history and goal keyframes. We consider text- conditioned variant where an additional text prompt is inputted to specify the action semantics of the frames in between. We compare our method with DNO (Karunratanakul et al. (024b) and OmniControl (Xie et al., (2024). For a fair comparison, we train DART on the HML3D dataset same as the baselines. We evaluate using test sequences covering diverse actions, with the sequence lengths ranging from 2 to 4 seconds. The quantitative evaluations are shown in Tab. 3 We report the L norm errors between the generated motion and the history motion and goal keyframe. We also evaluate the motion realism with the skate and jerk metrics. The skate metric (Ling et al. 2020: Zhhan et al.) 2018) calculates a scaled foot skating when in contact with the floor: s = disp- (2 24/0.03). where disp is the foot displacement in two consecutive frames, h denotes the higher foot height in consecutive frames and 0.03m is the threshold value for contact. We do not calculate skate metric for sequences where the feet are not on flat floor, such as crawling and climbing down stairs. Our method can generate the motions closest to the keyframe and show fewer skating and jerk artifacts. Our method effectively preserves the semantics specified by the text prompts, while the baseline DNO occasionally ignores the text prompts reach the goal keyframe, as illustrated in the examples of pacing in circles and dancing in the supplementary video. This highlights the superior capability of our latent motion fimitivee-based DART in harmonizing spatial control and text semantic alignment.
Table 3: Quantitative evaluation of text-conditioned motion in-between. The best results excluding the dataset are in bold and '+' indicates the 95% confidence interval.
History [am)
Goal emor (cm).
Skate (cm/s))
Jerk
Datusel
0.00 1. 0.00
0.00
0.00
2,27 1 0.00
0.74 1. 0.00
OmniControl
21.22 2.86
7.79
1.91
4.97 1: 1.31
1.11 1 0.08
DNO
1.20 - 0.20
4.24
1.34
5.38 0.70
0.65 + 0.06
Ours
0.00 = 0.00
0.59
0.01
2.98
+ 0.32
0.61 + 0.01
Human-scene interaction. We qualitatively show that our latent noise optimization control can be applied to human-scene interaction synthesis, where the goal is to control the human to interact naturally with the surrounding environment. Given an input 3D scene and the text prompts specifying the actions and durations, we use latent noise optimization to control the human to reach the goal joint location while adhering to the scene contact and collision constraints. The input scenes are represented signed distance fields for evaluating human scenne collision and contact constraints as detailed in Appendix F We present generated interactions of climbing stairs and walking to sit on a chair in Fig. 3 and the supplementary video.

---

Published as a conference paper at ICLR 2025
(a) Walk, turn left, sit on the chair
(b) Walk upstaris
(c) Walk downstaris
Figure 3: Illustrations of human-scene interaction generation given text prompts and goal pelvis joint location (visualized as a red sphere). Best viewed in the supplementary video.
4.3 REINFORCENT BHARNING-BASER CONTROL USING DART
By integrating DART with reinforcement learning-based control, we train text-conditioned goal- reaching policy models capable of three locomotion styles: 'walk', 'run', and 'hop on the left leg'. We evaluate goal-reaching on paths consisting of sequences of waypoints. We compare our method to a baseline GAMMA (Zhang & Tang 2022), which also trains goal-reaching policies with a learned motion action space. Unlike DART, GAMMA lacks text conditioning and is limited generating walking motions. The evaluation metrics include the reach time, the success rate of reaching the final goal waypoint, foot skating, and foot-floor distance. The evaluation results shown in Tab. 4 Our policy consistently reaches all goals within reasonable timeframe, while GAMMA occasionally fails to meet the final goal and may float off the floor beyond the contact threshold. Moreover, our text-conditioned goal-reaching policy achieves generation speed of 240 frames per second. These results demonstrate the potential of DART foundational human motion model, upon which versatile control models for various tasks can be learned through reinforcement learning.
Table 4: Quantitative evaluation of text-conditioned goal-reaching controller. The best results are in bold and 't' indicates the 95% confidence interval.
Time (4)
Success rate?
Skate (emis)
Floor distance (em))
GAMMA walk
31.44 2.58
0.95 1 0.03
5.14 1 1.58
5.35 0.84
Ours 'walk"
17.08 0.05
1.0 1: 0.0
2.67 0.12
2,24 +: 0.02
Ours 'run"
10.55 + 0.06
1.0 0.0
3.23 2 0.24
3.86 4 0.05
Ours 'hop on left log"
20.50 0.24
1. 0.0
2.22 - 0.12
4.11 0.07
LIMITATIONS AND CONCLUSIONS
DART relies on motion sequences with frame-level aligned text annotations, in BABEL, to achieve precise text-motion alignment and natural transition between actions. When trained on the coarse sentence-level motion labels from HML33D, the text-motion alignment degenerates for texts describing multiple actions, resulting in motions randomly switching between the described actions in random order. This occurs because each short motion primitive inherently matches only a portion of the sequence's semantics. Using a coarse sentence-level description the text label for primitive causes semantic misalignment and ambiguity. We aim to explore hierarchical latent spaces to effectively tackle both fine-grat fine-grained and global sequence-leve semanties (Stoffl al. 2024) in the future.
Our method DART effectively learns a text-conditioned motion primitive space that enables real-time online motion generation driven by natural languages. Additionally, the learned powerful motion primitive space allows for precise spatial motion control via latent noise optimization or reinforcement learning policies. Experiments demonstrate the superiority of DART in harmonizing spatial control with motion text-semantic alignment in the generated motions.
10

---

Published as a conference paper at ICLR 2025
Acknowledgements. We sincerely acknowledge the anonymous reviewers for their insightful feedback. We thank Korrawe Karunratanakul for helpful discussion and suggestions, and Siwei Zhang for meticulous proofreading. This work is supported by the SNSF project grant 200021 204840 and SDSC PhD fellowship.
REFERENCES
Simon Alexanderson, Rajmund Nagy, Jonas Beskow, and Gustav Eje Henter. Listen, denoise, action! audio-dríven motion synthesis with diffusion models. ACM Transactions on Graphics (TOG), 42 (4):1-20, 2023.
Marcin Andrychowicz, Anton Raichuk, Piotr Stañczyk. Manu Orsini, Sertan Girgin, Raphaël Marinier, Leonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem. What matters for on-policy deep actor-critic methods? a large-scale study. In International Conference on Learning Representations, 2021.
Tenglong Ao, Qingzhe Gao, Yuke Lou, Baoquan Chen, and Libin Liu. Rhythmic gesticulator: Rhythm-awaad aware co-speech gesture synthesis with hierarchical neural embeddings. ACM Transac- tions on Graphics (TOG), 41(6);1-1, 2022.
Tenglong Ao, Zeyi Zhang, and Libin Liu. Gesturediffuclip: Gesture diffusion model with clip latents. ACM Transactions on Graphics (TOG), 12(4))1-18, 2023.
Nikos Athanasiou, Mathis Petrovich, Michael Black, and Güi Varol. TEACH: Temporal action composition for 3d humans. In International Conference on 3D Vision (3DV), pp. 414-423, 2022.
German Barquero, Sergio Escalera, and Cristina Palmero. Seamless human motion composition with blended positional encodings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 457-469, 2024.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. Advances in neural information processing systems, 28, 2015.
Ling-Hap Hao Chen, Jiawei Zhang, Yewen Li, Yiren Pang, Xiaobo Xia, and Tongliang Liu. HumanMAC: Masked motion completion for human motion prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9544-9555, 2023a.
Rui Chen, Mingyi Shi, Shaoli Huang, Ping Tan, Taku Komura, and Xuelin Chen. Taming diffusion probabilistic models for character control. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-10, 2024.
Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu. Executing your commands via motion diffusion in latent space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 180000180101 2023b.
Xuxin Cheng, Yandong Ji, Junming Chen, Ruihan Yang, Ge Yang, and Xiaolong Wang. Expressive Whole-Body Control for Humanoid Robots. Proceedings of Robotics: Science and Systems, 2024.
Sammy Christen, Wei Yang, Claudia Pérez-D' Arpino, Otmar Hilliges. Dieter Fox, and Yu- Wei Chao. Learning human-to-robott handovers from point clouds. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pp. 9664-9664, 2023.
Simon Clavet et al. Motion matching and the road to next-gen animation. In Proc. of GDC, volume 2, pp. 4, 2016.
Setareh Cohan, Guy Tevet, Daniele Reda, Xue Bin Peng, and Michiel van de Panne. Flexible motion in-betweening with diffusion models. ACM SIGGRAPH 2024 Conference Papers, pp. 1-9, 2024.
Wenliang Dai, Junnan Li. Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general- lasion-language models with instruction tuning. In Conference Neural Information Processing Systems, 2023.

---

Published as a conference paper at ICLR 2025
Wenxun Dai, Ling-Hao Chen, Jingbo Wang, Jinpeng Liu, Bo Dai, and Yansong Tang. MotionL.M, Real-time controllable motion generation via latent consistency model. In European Conference Computer Vision, pp. 390-408, 2025.
Ross Girshick. Fast R-CNN. In International Conference on Computer Vision (ICCV), 2015.
Shubham Goel, Georgios Pavlakos, Jathushan Rajasegaran, Angjoo Kanazawa*, and Jitendra Malik*. Humans in 4D: Reconstructing and tracking humans with transformers. In International Conference Computer Vision (ICCV), 2023.
Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang. Wei Ji, Xingyu Li, and Li Cheng. Generating Diverse and Natural 3D Human Motions from Text. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5142-5151, 2022.
Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. MoMask: Generative masked modeling of human motions. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pullern Recognition, pp. 1900-1910, 2024.
Bo Han, Hao Peng, Minjing Dong, Yi Ren, Yixuan Shen, and Chang Xu. AMD: Autoregressive motion diffusion. AAAI, 2024.
Mohamed Hassan, Duygu Ceylan, Ruben Villegas, Jun Saito, Jimei Yang, Yi Zhou, and Michael Black. Stochastic scene-aware motion prediction. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 11374-113 1174-11344, 2021.
Jonathan Ho and Tim Salimans. Classifier- free diffusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-666 6851, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633-8644 2022.
Daniel Holden, Jun Saito, Taku Komura, and Thomas Joyce. Learning motion manifolds with convolutional autoencoders. In SIGGRAPH Asia 2015 technical briefs, pp. 1-4, 2015.
Daniel Holden, Taku Komura, and Jun Saito. Phase-functiones neural networks for character control. ACM Transactions on Graphics (TOG), 36(4):1-13, 2017.
Daniel Holden, Oussama Kanoun, Maksym Perepichka, and Tiberiu Popa. Learned motion matching. ACM Transactions on Graphics (TOG), 39 2020.
Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu. and Tao Chen. MotionGPT: Human motion as a foreign language. Advances in Neural Information Processing Systems, 36, 2024a.
Nan Jiang, Zhiyuan Zhang, Hongjie Li, Xiaoxuan Ma, Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, and Siyuan Huang. Scaling up dynamic human- interaction modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1737-1747, 2024b.
Roy Kapon, Guy Tevet, Daniel Cohen-Or, and Amit H Bermano. MAS: Multi-view ancestral sampling for 3d motion generation using 2d diffusion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1965-1974, 2024.
Korrawe Karunratanakul, Konpat Preechakul, Emre Aksan, Thabo Beeler, Supasorn Suwajanakorn, and Siyu Tang. GMD: Controllable human motion synthesis via guided diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024a.
Korrawe Karunratanakul, Konpat Precchakul, Emre Aksan, Thabo Beeler, Supasorn Suwajanakorn, and Siyu Tang. Optimizing diffusion noise can serve universal motion priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1334-1345, 2024b.
Diederik Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In International Conference Learning Representations, 2015.
12

---

Published as a conference paper at ICLR 2025
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In International Conference on Learning Representations, 2014.
Lucas Kovar, Michael Gleicher, and Frédéric Pighin. Motion graphs. In ACM SIGGRAPH 2008 classes, pp. 1-10, 2008.
Gen Li. Kaifeng Zhao, Siwei Zhang, Xiaozhong Lyu, Mihai Dusmanu, Yan Zhang, Mare Pollefeys. and Siyu Tang. EgoGen: An egocentric synthetic data generator. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024a.
Jiaman Li, Jiajun Wu, and C Karen Liu. Object motion guided human motion synthesis. ACM Transactions Graphics (TOG), 42(6):1-11, 2023.
Jiaman Li, Alexander Clegg, Roozbeh Mottaghi, Jiajun Wu, Xavier Puig, and C Karen Liu. Control- lable human-object interaction synthesis. In European Conference on Computer Vision, 2024b.
Han Liang, Wenqian Zhang, Wenxuan Li, Jingyi Yu, and Lan Xu. Intergen: Diffusion-based multi- human motion generation under complex interactions. International Journal of Computer Vision, pp. 1 1-21, 2024.
Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang. Motion-X: A lcrge-scale 3d expressive whole-body human motion dataset. Advances in Neural Information Processing Systems, 2023.
Hung Yu Ling, Fabio Zinno, George Cheng, and Michiel van de Panne. Character Controllers Using Motion VAEs. ACM Transactions on Graphics, 39(4), 2020.
Xinpeng Liu, Haowen Hou, Yanchao Yang, Yong-Lu Li, and Cewu Lu. Revisit human-scene interaction via space occupancy. In European Conference on Computer Vision, 2024.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia), 34(6): 248:1-248. 16, 2015.
Zhengyi Luo, Jinkun Cao, Alexander W. Winkler, Kris Kitani, and Weipeng Xu. Perpetual humanoid control for real-time simulated avatars. In International Conference Computer Vision (ICCV), 2023.
Zhengyi Luo, Jinkun Cao, Josh Merel, Alexander Winkler, Jing Huang, Kris M Kitani, and Weipeng Xu. Universal humanoid motion representations for physies-based control. In International Conference on Learning Representations, 2024.
Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Black. AMASS: Archive of motion capture as surface shapes. In International Conference on Computer Vision, pp. 5442-5452 2019.
Julieta Martinez, Michael J Black, and Javier Romero. On human motion prediction using recurrent neural networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2891-2900, 2017.
Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3D hands, face, and body from a single image. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 10975-10985, 2019.
Xue Bin Peng, Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. ASE: large-scale reusable adversarial skill embeddings for physically simulated characters. ACM Trans. Graph., 41(4), 2022.
Mathis Petrovich, Michael Black, and Gül Varol. TEMOS: Generating diverse human motions from textual descriptions, In European Conference Computer Vision, pp. 480-497, 2022.
Abhinanda R Punnakkal, Arjun Chandrasekaran, Nikos Athanasiou, Alejandra Ramros- and Michael Black. BABEL: Bodies, action and behavior with english labels. In Proceedings of the IEFF/VFF Conference Computer Vision and Pattern Recognition, pp. 722-731, 2021.
13

---

Published as a conference paper at ICLR 2025
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 1748-8763, 2021.
Davis Rempe, Tolga Birdal, Aaron Hertzmann, Jimei Yang, Srinath Sridhar, and Leonidas Guibas. HuMoR: 3d human motion model for robust pose estimation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 11488-12 11488-11499,2 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
Yoni Shafir, Guy Tevet, Roy Kapon, and Amit Haim Bermano. Human motion diffusion as generative prior. In International Conference on Learning Representations, 2024.
Mengyi Shan, Lu Dong, Yutao Han, Yuan Yao, Tao Liu, Ifeoma Nwogu, Guo-Jun Qi, and Mitch Hill. Towards open domain text-driven synthesis of multi-person motions. In European Conference on Computer Vision, 2024.
Yi Shi, Jingbo Wang. Xuekun Jiang. Bingkun Lin, Bo Dai, and Xue Bin Peng, Interactive character control with auto-regressive motion diffusion models. ACM Trans. Graph., 43(4), 2024. ISSN 0730-0301.
Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang. Chen Qian, Chen Change Loy, and Ziwei Liu. Bailando: 3d dance generation by actor-critic gpt with choreographic memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11050-11059, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- fional Conference on Learning Representations, 2021a.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021b.
Sebastian Starke, He Zhang, Taku Komura, and Jun Saito. Neural state machine for character-scene interactions. ACM Trans. Graph., 2806) 2019.
Sebastian Starke, Ian Mason, and Taku Komura. DeepPhase: Periodic autoencoders for learning motion phase manifolds. ACM Transactions on Graphies (TOG), 41(4):1-13, 2022.
Sebastian Starke, Paul Starke, Nicky He, Taku Komura, and Yuting Ye. Categorical codebook matching for embodied character controllers. ACM Transactions on Graphics (TOG), -314):1-14, 2024.
Lucas Stoffl, Andy Bonnetto, Stéphane d' Ascoli, and Alexander Mathis. Elucidating the hierarchical nature of behavior with masked autoencoders. In European conference on computer vision, 2024.
Richard Sutton and Andrew Barto. Introduction to reinforcement learning, volume 135. MIT press Cambridge, 1998.
Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffusion model. In International Conference on Learning Representations, 2023.
Guy Tevet, Sigal Raab, Setareh Cohan, Daniele Reda, Zhengyi Luo, Xue Bin Peng, Amit H Bermano, and Michiel van de Panne. CLoSD: Closing the loop between simulation and diffusion for multi-task character control. In International Conference Learning Representations, 2025.
Jonathan Tseng, Rodrigo Castellon, and Karen Liu. Edge: Editable dance generation from music. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 448 458, 2023.
14

---

Published as a conference paper at ICLR 2025
Tom Van Wouwe, Seunghwan Lee. Antoine Falisse, Scott Delp, and C Karen Liu. DiffusionPoser Real-time human motion reconstruction from arbitrary sparse sensors using autoregressive diffusion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2513-2523, 2024.
Zan Wang, Yixin Chen, Baoxiong Jia, Puhao Li, Jinlu Zhang, Jingze Zhang, Tengyu Liu, Yixin Zhu, Wei Liang, and Siyuan Huang. Move as you say, interact as you can: Language-guinn human motion generation with scene affordance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2024.
Yiming Xie, Varun Jampani, Lei Zhong, Deqing Sun, and Huaizu Jiang. OmniControl: Control any joint any time for human motion generation. In International Conference Learning Representations, 2024.
Sirui Xu, Zhengyuan Li, Yu-Xiong Wang, and Liang-Yau Yan Gui. Interdiff: Generating 3d human-object interactions with physics-informed diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14928-149449 2023.
He Zhang, Sebastian Starke, Taku Komura, and Jun Saito. Mode-adaptive neural networks for quadruped motion control. ACM Transactions on Graphics, 37(4):1-11, 2018.
Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xi Shen, and Ying Shan. T2M-GPT: Generating human motion from textual descriptions with discrete representations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14730-14740, 2023a.
Lvmin Zhang, Anyi Rao, and Mancesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023b.
Mingyuan Zhang. Zhongang Cai, Liang Pan, Fangzhou Hong. Xinying Guo, Lei Yang, and Ziwei Liu. MotionDiffuse: Text-driven human motion generation with diffusion model. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46:4115 45-228, 2022a.
Xiaohan Zhang, Bharat Lal Bhatnagar, Sebastian Starke, Vladimir Guzov, and Gerard Pons-Moll. COUCH: Towards controllable human-chair interactions. In European Conference on Computer Vision (ECCV), 2022b.
Xiaohan Zhang, Bharat Lal Bhatnagar, Sebastian Starke, Ilya A. Petrov, Vladimir Guzov, Helisa Dhamo, Eduardo Pérez Pellitero, and Gerard Pons-Moll. FORCE: Dataset and method for intuitive physics guided human-object interaction. In International Conference on 3D Vision (3DV), 2025.
Yan Zhang and Siyu Tang. The wanderings of odysseus in 3d scenes. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 20481-20491, 2022.
Yunbo Zhang, Deepak Gopinath, Yuting Ye, Jessica Hodgins, Greg Turk, and Jungdam Won. Simula- tion and retargeting of complex multi-character interactions. In ACM SIGGRAPH 2023 Conference Proceedings, pp. 1-11, 2023c.
Kaifeng Zhao, Shaofei Wang, Yan Zhang, Thabo Beeler, and Siyu Tang. Compositional human-scene interaction synthesis with semantic control. European Conference on Computer Vision, pp. 311-327, 2022.
Kaifeng Zhao, Yan Zhang, Shaofei Wang. Thabo Beeler, and Siyu Tang. Synthesizing diverse human motions in 3d indoor scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14738-14749, 2023.
Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5745-5755 2019.
Fabio Zinno. MI tutorial day: From motion matching to motion synthesis, and all the hurdles in between. Proc. of GDC 2019, 2, 2019.
15

---

Published as a conference paper at ICLR 2025
A MOTION PRIMITIVE REPRESENTATION
Representation. We represent each frame of the motion primitive as a tuple of (t, R, J,J, dt, dJ), where t E IR denotes the global body translation, R E RR denotes the 6D rotation representation (hou et al. 2019) of the global body orientation, representation of 21 joint rotations, J _ denotes the 22 joints locations, dt e R3 denotes the temporal difference with previous frame's s translation, dR E R denotes the 6D representation the relative rotation between current and previous frame's 's body orientation, and dJ E R R22x3 denotes the temporal diffence between current and previous frame's joint locations.
221x6 is the 6D
Our motion representation is overparameterrized (Holden al., 2017, Ling et al., 2020: Rempe et al. 2021 Guo et al.) 2022), with multiple benefits. Firstly, the joint rotation components are compatible with animation pipelines, saving the time-consuming optimization-bann based skeleton- to-body conversion required by the commonly used HML3D (Guo et al. (2022) representation in text-to-motion methods. Morcover, including the joint location components J facilitates solving physical constraints like reducing foot skating and joint trajectory control. Our motion representation also models the first-order kinematics with the temporal difference features improve the motion naturalness.
Canonicalization. We represent motion primitives in a human-centric local coordinates frame to canonicalize the primitive features and facilitate model learning. Each motion primitive is canonical- ized in local coordinates system centered at the first frame body. The origin is located at the pelvis of the first frame body, the X-axis is the horizontal projection of the vector pointing from the left hip to the right hip, and the Z-axis pointing in the inverse gravity direction. Given the pelvis, left hip, and right hip joints, the origin is located at the pelvis joint, and the local axis system can be derived by:
Algorithm Motion primitive rotation canonicalization
Input: right_hing E TR3, left_ E R x_axis = right_hip left left_hip
__xxss[]] 0
b Project to the xy plane
normalize(x_nnn (x_xis)
z_axis [0, 0, 1]
Inverse gravity direction
y_axis = cross lleeccoosssssseee
cetssspooud_t_aaaae x_ axis)
normalizely _xxis_
return [x_axis, y y_axis, 2_axis]
We store the canonicalization transformations and apply their inverse transformation to the generated motion primitives recover global motions in the world coordinates.
B DATASETS
We train separate DART models on motion-text data from the BABEL (Punnakkal al. 2021 and HML3D (uoo et al. 2022) dataset. Both BABEL and HML3D use motion sources from the AMASS (Mahmood
2019) dataset. Their main difference that BABEL features fine-grained frame-level text annotation while HML3D uses coarse sequence-level annotation.
The BABEL dataset contains motion capture sequences with frame- alinned text labels that annotate the fine-grained semantics of actions. Fine-grained text labels in BABEL allow models to learn precise human action controls and natural transitions among actions. We use the motion data at a framerate of 30 frames per second the same as prior works (Barquero et al.] 2024 Athhnaaioou et al.) 2022). During training, motion primitives are randomly sampled from data sequences and the text label is randomly sampled from all the action segments that overlap with the primitive. To alleviate the action imbalance in the BABEL dataset, we use the provided action labels to perform importance sampling during training that the motion data of each action category has roughly equal sampling chances despite their varying frequency in the original dataset. To maintain compatibility with prior works, fix the human gender male and set the body shape parameter zero
16

---

Published as a conference paper at ICLR 2025
The HML3D dataset contains short motions with coarse sequence-level sentence descriptions. We only use subset of HML3D in training since its subset HumanAct12 and left-right mirroring motion sequences only provide joint locations instead of SMPL (Loper et al. 2015 Pavlakos et al.) 2019) body sequences that our motion representation requires. 'The joint rotations used in the original HML3D are calculated using naive inverse kinematics and can not be directly used animate human motions. We therefore only train using the subset where the SMPL body motion sequences is available. The HML3D motions are sampled at framerate of 20 frames per second. During training, we randomly sample primitives with uniform probability and the text label is randomly chosen from one of the multiple sentence captions of the overlapping sequence. To maintain compatibility with prior works, we fix the human gender to male and set the body shape parameter to zero.
C MOTION PRIMITIVE VAE
Architecture. Our Motion Primitive VAE employs the transformer- architecture. Both the encoder and decoder consist of 7 transformer encoder layers with skip connections (Chen et al. 2023b). The transformer layers the dropout rate of 0.1, ferdrforward dimension of 1024, hidden dimension of 256, attention heads, and the gelu activation function. The latent space dimension is 256. After finishing training, we follow (Rombach al. 2022) to calculate the variance of the latent variables using a data batch. When training the latent denoiser model, the raw latent output from the encoder scaled have a unit standard deviation.
Losses. The motion primitive VAE is trained with the following losses:
AV = Liee + WKL x LKL + Waur x Lourt X LSMPL-
The reconstruction loss Lyee aims to minimize the distance between the reconstructed future frames X and the ground truth future frames X as follows:
Treec F(X,))
(5)
where -(.-)) denotes the distance function and we use the smoothed L1 loss(Girshicei 2015) in implementation.
The Beilbackk-eibbee divergence (KL) term LKL penalizes the distribution difference between the predicted distribution and a standard Gaussian:
LKL = (L({(z)H) || I)),
(6)
where (T. denotes the Kullback-Leibler divergence (KL), and q(2) [ H) denotes the predicted distribution from the encoder E. We use a small KL term of le following Rombach et al. (2022) as we aim to keep the latent space expressive and only use the small KL loss to avoid arbitrarily high-variance latent spaces.
The auxiliary loss Louz regularizes the predicted temporal difference features d- of translations, global orientation, and joints to be close to the actual temporal differences d- calculated from the predicted motion features:
Lauz = F(dt, dt) + F(JJ, dJ) + (dRRRR)
For instance, the difference of the first two frames of the predicted root translation (dt." :- fl E
should be consistent with the predicted first frame root translation difference feature dit. We use Wanx 100 in our experiments.
The SMPL losses LSMPr, consist two components: the SMPL-based joint reconstruction loss Ljoint_re rens rens and the joint consistency loss Leonsistency- The SMPL losses LSMPL. are defined as follows:
LSMPL Lijoint_ree rec + Leconsstteneg-
The SMPL joint reconstruction loss Lijoint_rea penalizes the discrepancy between the joints regressed from the predicted body parameters and those regressed from the ground truth body parameters:
I Lej F(J(t, R. 0), J(. R)O)):
17

---

Published as a conference paper at ICLR 2025
where I denotes the SMPL body joint regressor that predicts joint locations given the body parameters. The SMPL body shape parameter is assumed to be zero and ignored for simplicity here.
The joint consistency loss Leonsistency regularizes the predicted joint locations and predicted SMPL body parameters consistently represent the same body joints:
Leonsistency FFJ, Jtt R. 0))
(10)
We use PSMPL 100 in our experiments. We train the motion primitive VAE with the AdamW (Kingma & Ba 2015) optimizer and the learning rate is set to le
linear annealing.
We conduct ablation studies on the impacts of the losses. We evaluate motion primitive VAEs trained with different loss weights on autoregressive motion sequence reconstruction error and motion jittering in reconstructed motions. Using large KI loss weight of wxg 1, reduces the model expressiveness (the reconstruction error increases from 0.08 to 0.44 compared to WKL = 1e on the test motions) and fails accurately reconstruct complex sequences like cartwheeling. Using a small KL loss weight of WKL = -e- can maintain the expressiveness of the learned VAE while allowing the latent distribution to deviate a bit more from a standard Gaussian. Applying the auxiliary losses with Wanz = 100 helps to reduce the jittering in the reconstructions and improve the motion quality compared to using Wour = as reflected by a smaller jerk metric of 2.45 when using Wonz 100
compared to a jerk of 3.67 when using Wour O.
D LATENT DENOISER MODEL
D.1 LOSSES
We train the latent denoiser model using DDPM (Ho et al. 2020) with 10 diffusion steps and use a cosine noise scheduler. The denoiser model is trained with the following losses:
Lieeoiser
Lismple + Wrec X Liee + Wouz X Lawxs
(11)
Lsimple [(xo,e) c. C), 20),
(12)
where F(::) -) is distance function and we use the smooth L. loss Girshick 2015) in our implemen- tation. We train the denoiser to predict the clean latent variable with the simple objective Liimples and apply the feature reconstruction loss Lyec and auxiliary losses Lauz on the decoded motion primitive X D(G(z,, t, c)) to ensure the decoded motion primitives are valid.
D.2 SCHEDULED TRAINING
We use scheduled training to improve the stability of long sequence generation and the text prompt controllability. Long-tern prediction stability is significant challenge in autoregressive generation since the sample distribution can drift and accumulate during autoregressive generation. When the sample drifts out of the distribution covered by the learned model, the generation results can go wild. Our latent motion primitive model also faces the long-term stability challenge as an autoregressive method.
To alleviate the al. 2013 Rempe al.) 2021: Martinez al. 2017) to progressively introduce the test-time distributions during training. Specifically, we train the latent denoiser model on sequences of N consecutive motion primitives and use the prediction result of the previous motion primitive instead of the ground truth dataset to extract the history motion input H. We name such history motion extracted from the predicted last primitive as rollout history. Using the rollout history instead of the ground truth history introduces the test-time distribution which can differ from the dataset distribution, e.g., unseen human poses or -uitt--diittibuttion combinations of human bodies and text labels. Exposing the model to such test-time distribution at training can improve the long-ten generation stability and increase the text controllability when facing novel combinations of history motion and text prompts at generation time.
problems, we use the scheduled training (Ling et al. 2020 Bengio
The scheduled training has three stages to progressively introduce the rollout history. The first stage is fully supervised training where only the ground truth history is used during training. The second
18

---

Published as a conference paper at ICLR 2025
scheduled learning stage randomly replaces the ground truth history motion with rollout history motion by probability linearly increasing from 0 to I. The third stage of rollout training always uses the rollout history instead of the ground truth history. The scheduled training algorithm for the latent denoising model is shown in Alg. 5
Algorithm 4 Calculate rollout probability
1: Input: current iteration number iter, number of train iterations in the first supervised stage 41, number of train iterations in the second scheduled stage Iz
2: Output: rollout probability p
3: function ROLLOUT_ ROLLOUT PRROBBBBILT(((lle,,, I,, I2)
4: if iler S 11 then
rollout in the first supervised stage
5:
+ 0
6: else if iter 11 + 1a then ++1
P the third rollout stage always use rollout
7:
8:
else
b linearly scheduled rollout probability in the second scheduled stage
9:
p
1ter-1 12
10:
end if
11: return p
end function
We Wyeec and Waxr 10000 in experiments. We do not the SMPL. losses L&M Pr in training the denoiser model because the SMPL body model inference process slows down the training. The denoiser model is trained using an AdamW optimizer. The learning rate is set 1e with linear annealing. Our denoiser model is trained with scheduled training (Ling et al. 2020 Bengio al. 2015), consisting of fully supervised stage of 100K iterations, scheduled stage of 100K iterations, and a rollout stage of 100K iterations. We set the maximum number of rollouts 4. With scheduled training, our latent motion primitive model can stably generate long motion sequences and better respond to the text prompt control even at poses that are not paired with the text prompt in the dataset.
E TEXT-CONDITIONED TEMPORAL MOTION COMPOSITION
E. EXPERIMENT DETAILS
We apply DART to conduct online motion generation using the rollout algorithm with default seed motion H seed of rest standing and using a classifier-free guidance weight 5. We use the released checkpoints of FlowMDM (Barquero et al.) 2024), TEACH (Athanasiou et al.) (022). and DoubleTake (Shafir et al. 2024) for baseline comparison. We adjust the handshake size and blending length of Double'Tat history- histtry-conditioned modification of T2M-GPT is retrained on the BABEL dataset using the original hyperparameters. At generation time, the last frames of the previous action segment encoded into tokens and provided as the first tokens when generating the next action segment to provide history conditioning of the previous action.
be compatible with the shortest segment length of 15 frames. The
We extract the timeline of action segments for evaluation from the BABEL (Punnakkal et al. 2021) valid set. For each sequence in the validation set, we sort the original frame labels provided by BABEL to obtain a list of tuples of text prompts and durations. We randomly sample one text label when segment annotated by multiple texts. We skip the 'transition' text labels due the ambiguous semantics and clamp the duration length with minimum of 15 frames.
E.2 HUMAN PREFERENCE STUDIES.
We conduct human preference studies to quantitatively compare our method DART with baselines to provide more comprehensive and convincing evaluation. We run human preference studies on Amazon Mechanical TArM(AMM) to evaluate generation realism and text-motion semantic alignment. Participants are given generation results from two different methods and asked to select the generation that is perceptually more realistic or better aligns with the action text stream in subtitles. as illustrated in Fig. During the realism evaluation, action descriptions were not displayed eliminate
19

---

b initialize motion history number rollouts Published as a conference paper at ICLR 2025
Algorithm 5 Scheduled training for the latent denoising model
1: Input: pretrained motion primitive decoder D and encoder &, latent variable denoiser Ge parameterized by 0, total diffusion steps T, optimizer O, loss criterion L, train dataset X. 2: Scheduled training parameters: the number of train iterations in the first supervised stage /1. the number of train iterations the second scheduled stage Iz. the number of train iterations in the third rollout stage I3. the maximum number of primitive rollouts N during training.
3:
4: Ttotal 11 12 13
5:
iter +0
6:
while iter Iootll do
7:
[Heed X ct 1 X , c -
1 ~
8:
sample N consecutive motion primitives with text labels from dataset X for to N do
9:
H Hseed
11:
2) E(H, )+)
compress motion primitive into latent space > sample diffusion step
12:
t ~ [0,
13:
z + FORWARD DIFFUSION (z).
14:
2 =
latent denoising model prediction decode predicted latent variable to future motion frames D model parameter gradient calculation model update using optimizer
15:
X D(H, 2
16:
V < VoL(zz) 2 H
17:
O V)
18:
19:
+ ROLLOUT PROBABILITY(iter, I2)
20:
update history motion using predicted GT motion by scheduled probability if Frand() then
21:
use predicted rollout history 2 FORWARD DIFFUSION T) maximum noising simulating inference time full DDPM denoising loop decode predicted latent variable and stop gradient
22:
23:
2/ DDPM_ DDPM_SAMPLE((C 2 T, H,)
X SG )
25:
H < CANONICALIZE X -
26:
else
P use ground truth history
27:
H + CANONICALIZE FF -H++1:P)
28:
end if
29:
30:
iter iter
31:
end for
end while
distractions. We sample 256 sequences of action texts and durations from the BABEL dataset and use each method to generate motions given the action timelines. Participants are shown random pairs of results from our method and a baseline, and they are asked choose the better generation. Each comparison is voted by 3 independent participants, with the video pairs randomly shuffled to ensure that participants not know the method source.
E.3 ABLATION STUDIES.
conduct ablative studies on architecture designs, diffusion steps, primitive representation, and scheduled training the text-conditioned motion composition task.
Removing the variational autoencoder (DART-VAF) and training diffusion model in the raw motion space results in significantly higher jittering reflected by the peak jerk (PJ) and Area Under Jerk (AUJ) metrics. This validates the effectiveness of integrating the variational encoder to compress the high-frequency noise in motion data and improve motion generation quality.
Without the scheduled training -DART-sceddle), the model can not effectively respond to text control and have significantly worse R-Prec and FID metrics. This is because, without scheduled training, the model can easily encounter diitt---iittiuution combinations history motion and text prompts during autoregressive generation.
20

---

Published as a conference paper at ICLR 2025
Select an option
step back
step book
023/8.00
Select option
Right
DO2 / DIS
Figure 4: Illustration of the human preference study interface for evaluating motion-tex semantic alignment (top) and perceptual realism(bottom). Participants are requested select the generation that is perceptually more realistic or better aligns with the action descriptions in subtitles (only visible in semantic preference study).
We also include the ablation study of training model to predict the single next frame conditioned on the current frame (per frame) as used in Shi et al. (2024). This is a special case of primitive with history length H 1 and future length F 1. The ablative model has significantly worse R-Prec and FID metrics and cannot respond to text prompts. This indicates using frame-by-frame prediction is less effective than primitives with a reasonable horizon (Il 2, 8) in learning text-conditioned motion space.
DART learn high-quality text-conditioned motion primitive models using very few diffusion steps because of the simplicity of the primitive representation. We conduct ablative studies of training DART using different numbers of DDPM diffusion steps. Reducing the diffusion steps from 100 to fewer than 10 does not significantly affect performance. However, an extremely low diffusion step number of leads much higher FID, indicating poorer motion quality.
Table 5: Ablation studies results on text-conditioned temporal motion composition. The first row includes the metrics of the dataset for reference. Symbol "- denotes that closer to the dataset reference better and '+' indicates the 95% confidence interval.
Segment
Transition
FIDL
R.prect
MM-Dist
Dataset
0.00:1::00
0.724:.00
8.424.0.5
3.3610.00
0.00-00.00
6,20:10.06
0.02--0.00
0.00
Ours
3.79110.06
0.621.0.1
8.0510.100
5.271.0.1
1.86.1000
6.70110.33
0.06.10.00
0.211:0.00
4.23:10.02
6.73-10.2
0
DART-schodole
8.08+10.09
0,39::0.01
8.06-0012
6.96-00008
7.1100.10
6,58-0.06
0.03-00.00
0.18=0.00
per Fra))
10.3140.09
0.29::0.01
6.8240.13
7.4120.01
7.82=0.09
6.03-20.88
0.02-00.00
0.08-00.00
H=2,F=16
4,0410.100
0.661:0.00
8.201:0.06
4.96110.01
2.22:10.10
6.60:10.20
0.06:10.00
0,18::0.00
steps
4.4410..4
0.60:1..00
8.2010.15
5.38100.11
2.24 0.02
6.77..0.07
0.05
0,20100.0
steps
3.49100099
0.63-00000
8.25+0.15
5.1810.01
2.11
6.7440.11
0.20-10.00
steps
3,70:10.03
0.62#00011
8.04
5.25-0.03
2.15=0.08
6.72=0.15
0.06+0.00
0,20=0.00
sleps 10 (Ours)
3.79:10.06
0.62::0.01
805500100
5.27:0.01
1.86:2005
6.70::005
0.06:0000
0.21-:0.00
sleps 50
3,8210005
0.60
7.74
5.3010.01
2.1110.10
6.58110.10
0.06
0,22110.00
steps 100
4.16:10.06
0.61:1000
7.820..5
5.32
2.20
6,43:10.10
0.06-10.00
0.21-10.00
21

---

Published as a conference paper at ICLR 2025
F LATENT DIFFUSION NOISE OPTIMIZATION--ASEE CONTROL
Optimization details. We use the Adam (Kingma & Ba 2015) optimizer with learning rate of 0.05 to optimize latent noises Zr for 100 to 300 steps. The learning rate is linearly annealed to 0 and the gradient is normalized to stabilize optimization. The latent noises are initialized by sampling from Gaussian distributions. Empirically, variance, such as 0.01, slightly improves the jerk and skate metrics at the cost of reduced motion diversity in the in-between experiment, compared to unit variance initialization. The optimization running time is dependent on both the motion sequence length and the number of optimization steps. An example experiment of target motion sequence of 60 frames and 100 optimization steps costs around 74 seconds.
observe that initializing these latent noises with a small
Human-scene interaction. In human-scene interaction synthesis, we use scene constraints to avoid human-scene -cenn collision and foot-floating artifacts. To reduce human-scent seennn-ceeneeeee interpenetration, cons_ cons_coll(.) penalizes joints that collide into scenes and exhibit negative SDF values in each frame:
22
cons_coll([) - ((V(J -
(13)
k=1
where J denotes the location of the k-th joint in the scene coordinates, (-) denotes the signed distance function returning the signed distance from the query joint location to the scene, T denotes the joint-dependent contact threshold values, which are determined by the joint-skin distance in the rest pose, and (.)_ denotes clipping positive values. To reduce the occurrence of foot-floating artifacts, cons cons_cont(() encourages the foot to in contact with the scene in every frame and defined as:
cons_cont (J) = min
(r() -T)+
(14)
Fot
where Foot denotes the set of foot joint indices, t denotes the foot-floor contact threshold distance, and (+)+ denotes clipping negative values.
G REINFORCEMENT LEARNING-BASER CONTROL
Architecture. Both the actor and critic networks are 4-layer MLPs with residual connections and a hidden dimension of 512. We apply the tanh scaling: output to clip the action prediction in the range of [-4,4] 4). avoiding unbounded action predictions. The actor networks are initialized with close to zero weights boost policy training following (Andrychowicz et al. 2021). The observation input contains a 512D text embedding, a 552D of history motion, ID observation of floor height relative the human pelvis, ID observation of the floor plane distance from the pelvis to the goal location, and 3D unit vector of the goal direction in the human- centric coordinates frames. We clamp the goal distance observation with a maximum value of 5m and the goal direction in the range of a 120-degree field of view to simulate egocentric human perception.
4 tanh(x:) (Ling al. 2020) the actor
Rewards. The text-conditioned goal-reaching control policies are trained to maximize the discounted expectation of the following rewards (Zhang & Tang 2022 Li et al., 2024a Zhao et al. 2023):
We use three distance-related rewards to encourage the human agent to minimize its distance to the goal location.
Ediet = df d'
(15)
The distance reward rdist encourages the human to get closer to the goal location, where d' is the 2D distance between the human pelvis and the goal location at step i.
{
1
< 0.3
I Tsuce
(16)
otherwise
The success reward suce gives a sparse but strong reward when the human arrives at the goal, where 0.3m is the success threshold value.
(p'
9 p pp-1)
Tors
(17)
2
22

---

Published as a conference paper at ICLR 2025
The moving orientation reward encourages the moving orientation to be aligned with the goal orientation, where p' is the human pelvis location at step i, and is the goal location.
Moreover, apply scene constraints-rclate rewards to discourage unnatural behaviors such as foot skating and collision with the floor.
Tskate disp (2 22//0.03,,
(18)
The skate reward rsktate penalizes foot displacements when contact with the floor, where disp is the foot displacement in consecutive frames, h denotes the higher foot height in consecutive frames and 0,03m is the threshold value for contact.
f'llorr - (11/ 0.03)++
(19)
The floor contact reward floor penalizes when the lower foot distance to the floor is above the threshold of 0.03m, where i.f denotes the height of lower foot, | denotes the absolute value operator, and (-)_ is the clipping operator with minimum of 0.
The rewards are weighted with Wdist = 1, Wsuce 1, Worl = 0.1, Wskatt 100, Wloor 10 for hopping and running and floor 100 for walking.
H COMBINATION WITH PHYSICS-BASED MOTION TRACKING
Our method, DART, enables real-time motion generation in response to online text prompts. However, as a kinematic-based approach, DART may produce physically inaccurate motions with artifacts such as skating and floating. address this, we demonstrate that DART can be integrated with physically simulated motion tracking methods, specifically PHC (Luo al. 2023), to generate more physically plausible motions. In Fig. 5 we present an example sequence of a person crawling. The generation results from DART exhibit artifacts such hand-floor penetration. Applying physics-based tracking to refine the raw motion successfully produces more physically plausible results, improving joint-floor contact and climinating penetration artifacts. This integration combines the versatile text-driven motion generation of DART with the physical accuracy provided by the bhase simulation. Given the real-time capabilities of both DART and PHC, it is possible to leverage physics to correct the kinematic motion generated by DART on the fly and then use the corrected motions for subsequent online generation.
(a) Crawling sequence generated by DART
(b) Physics-based motion tracking result
Figure 5: We demonstrate an example of integrating DART with the physics-based motion tracking method PHC at. 2023) achieve more physically plausible motions. The left image illustrates crawling sequence generated by DART, exhibiting artifacts such as hand-floor penetration. The right image displays the physics-basee based motion tracking outcome applied to the raw generated sequence, which enhances foint-floor contact and resolves the hand-floor penetration issue.
23

---

Published as a conference paper at ICLR 2025
1 DISCUSSION OF LONG ROLLOUT RESULTS GIVEN A SINGLE TEXT PROMPT
Our DART can autoregressively generate perpetual rollouts of actions that are inherently repeatable and extendable. For example, DART can produce minutes-long sequences of continuous human motion, such as jogging in circles, performing cartwheels, or dancing. These actions are inherently repeatable, and such extensions are also represented in the AMASS (Mahmood al. 2019) dataset. DART can stably generate minutes-long rollouts of the same action, enabled by its autoregressive motion primitive modeling and scheduled training scheme.
Some other actions, however, have inherent boundary states that mark the completion of the action. For instance, "kneel down" reaches boundary state where the knees achieve contact with the floor. Further extrapolation of "kneel down" beyond this boundary state is not represented in the dataset and is not intuitively anticipatable by humans, as no further motion logically extends within the action semantics. Continuing rollout using the "kneel down" text prompt results in motions exhibiting fluctuations around the boundary state.
In summary, long rollout results given single text prompt will repeat naturally fluctuate around boundary state, depending on the inherent nature of the action and its representation in the dataset. We provide video results of minutes-long rollout generation results our project website
J DISCUSSION ON OPEN-VOCABULAR MOTION GENERATION
Limited vocabulary is a critical limitation and challenge shared by existing text-conditioned motion generation methods. Existing methods, including our approach DART, struggle to generalize to open-vocabulary text prompts due to the scarcity of 3D human motion data with text annotations. The scale of motion data available is several orders of magnitude smaller than that for text-conditioned image and video generation, primarily due to the reliance on marker-based motion capture systems, which are challenging to scale.
To expand the dataset and enable open- vocabulcyy generation, extracting human motion data from in-the-wil internet videos and generative image/video models (Kapon et al. 2024 Goel et al. 2023
Lin et al. 2023] Shan et al. 2024), is promising direction. Additionally, the rapid advancement of vision-language models (VLMs) holds promise for automatically providing detailed, frame-aligned
motion text labels facilitate text-to-motion generation (Shan et al. 2024 Dai al. 2023
K
COMPUTING RESOURCES
Our experiments and performance profiling are conducted on workstation with single RTX 4090 GPU, intel 17-13700K CPU, 64GiB memory. The workstation runs with Ubuntu 22.04.4 LTS system.
24


Reviews:
[{"review_id": "fUbaxoMhqm", "reviewer": "Reviewer_tzns", "summary": "This paper proposes DART, a real-time, diffusion-based text-to-motion framework that enables interactive text-driven motion control. DART leverages a motion primitive as the action space and learns a diffusion model on top of the latent space to learn continuous human motion. The learned diffusion model conditions on both text and history motion input and can be autoregressively run in real-time. Results on a number of text-conditioned generative motion tasks show the superiority of the proposed method in motion generation quality, real-time interactive control, and alignment with text or scene.", "questions": "See weakness: how would DART perform if one text condition is given for a long time?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- I find DART a well-designed system that leverages existing techniques (motion primitives, autoregressive diffusion models) in an intuitive way. Using a pre-trained motion primitive during the diffusion motion generation process is well-thought-out and, as can be seen from the result, very performant.\n- The real-time **interactive** control part shown in the demo videos is very impressive and useful for downstream tasks. Asking the character to perform actions in real-time and the character has the ability to switch naturally to the next text prompt.\n- DART is versatile in terms of the control input and optimization techniques it can interface with (direct optimization, RL, etc.) and the resulting motion is of high quality while conforming to the text prompt.\n- Both quantitatively and qualitatively, I find the proposed method state-of-the-art.", "weaknesses": "- Since BABEL has a limited vocabulary, DART's vocabulary is limited.\n- Since motion generation in DART is autoregressive and real-time, I am wondering what the model would do if the text does not change but the model is continuously rolled out. Would the motion be repeated, or would the model be stuck in a weird space? Provide a few single-text prompt experiments for an extended period (e.g., several minutes) and analyze the resulting motion would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KBHqnYRV8K", "reviewer": "Reviewer_dw3E", "summary": "Authors of this propose a text-conditioned motion generation method that is capable of generating long motion sequences, is controlled by a stream of text prompts, and has short inference time and is thus suitable of real-time settings. The promposed method autoregressively generates motion latents using latent diffusion models, conditioned on motion history and the next text input. The proposed method also supports spatial constraints such as keyframe in-betweening and goal reaching. This is done via latent noise optimization or RL by treating the latent noise as actions.", "questions": "1. What is the advantage of using latent diffusion models in this work? Since the data dimensionality is low, perhaps there is not much performance or speed improvements. \n2. Related to the previous question, what is the advantages of incorporating spatial constraints in the latent space versus the real motion space?\n3. For the scene-interaction tasks, does the model have any awareness of the environment? For instance, in the walking upstairs example of Figure 3.b, is the motion first generated given the pelvis location, and the stairs are later added in the visualization?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Having text-to-motion generative models that are real-time and allow for spatial constraints are import components in computer graphics research. Therefore, I find the problem very relavent and interesting.\n* Paper is well presented, clearly written, and easy to read.\n* I appreciate the detailed overview of contrallable motion generation works in the related works section (Section 2).", "weaknesses": "* The primary weakness of the paper lies in its similarity to several concurrent works addressing the same idea. While this isn’t necessarily a flaw of this specific work, it does somewhat limit its novelty and overall significance in the broader research landscape.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W7X9XXEpcb", "reviewer": "Reviewer_WRyH", "summary": "This paper introduces DART, a novel method for real-time and text conditioned human motion. While previous methods are limited to short and offline motion generation, DART uses diffusion-based framework to produce continuous motion by learning a motion primitive space conditioned on the text and motion history. DART can also use latent space optimization or reinforcement learning to control the motion generation under different spatial constraints.", "questions": "Given that current short motion generation methods (such as MoMask) can achieve excellent generalization, and motion prediction methods (such as HumanMAC) are also quite effective, a straightforward idea is to concatenate two separately generated motions. This approach ensures that the generalization and details of individual motions are outstanding, as single motion generation is trained on large-scale sequence-level annotated data (like MotionX and HumanML3D), while motion prediction is trained on long-duration motion datasets (like Human3.6M). If DART only supports frame-level annotated data, it means that the quality of individual motions will fall significantly behind current state-of-the-art methods, indicating that the motivation of the paper is unclear. Therefore, I suggest the authors:\n\n- Provide results from training with sequence-level annotated data, even though the paper mentions that the performance is not good.\n- Present performance results on the MotionX dataset, which also has frame-level annotated data but is more challenging than BABEL.\n- Combine advanced motion generation methods (such as MoMask) with motion prediction methods (such as HumanMAC), and then compare DART’s performance in terms of individual motion generation quality and motion continuity. I anticipate that if DART does not address the issue of not being able to use sequence-level training data, the quality and generalization of the motions will be poor than other methods.\n\n[1] Chen, Ling-Hao, et al. \"Humanmac: Masked motion completion for human motion prediction.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[2] Guo, Chuan, et al. \"Generating diverse and natural 3d human motions from text.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[3] Guo, Chuan, et al. \"Momask: Generative masked modeling of 3d human motions.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[4] Lin, Jing, et al. \"Motion-x: A large-scale 3d expressive whole-body human motion dataset.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[5] Ionescu, Catalin, et al. \"Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments.\" IEEE transactions on pattern analysis and machine intelligence 36.7 (2013): 1325-1339.\n\n[6] Punnakkal, Abhinanda R., et al. \"BABEL: Bodies, action and behavior with english labels.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- DART has indeed achieved real-time continuous motion generation. Continuous motion generation is an important issue and deserves attention.\n- DART enables effective motion control, which is user-centric.\n- DART can achieve real-time motion generation, which is user-friendly.", "weaknesses": "- Due to the limitations of the methodology, training can only use frame-aligned text annotations, resulting in the model's performance on motion detail generation and generalization being significantly inferior to that of models trained with sequence-level annotated data, such as the HumanML3D dataset.\n\n- The experiments are insufficient, and the results are not solid enough. The MotionX dataset also provides frame-level data. Compared to the small-scale BABEL dataset, the authors lack demonstrations of training effectiveness on larger-scale datasets. \n\n- The motivation is unclear. Simply using a short motion generator combined with an motion prediction network can achieve high-quality motion generation while ensuring the quality of motion completion and spatial control over motion points. However, the authors did not conduct a comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UnaCDGcNy6", "reviewer": "Reviewer_J1Bn", "summary": "The paper presents a human motion diffusion model that allows for real-time generation with dynamic text conditioning, using an autoregressive latent-space model.  The technical contributions relate to the specifics of the latent + autoregressive architecture, and showing all that is capable of. Much like for VAEs, it is further shown that the latent can also be used an action, and therefore also used for (a) gradient-based motion sequence optimization, and (b) to develop RL policies for specific tasks. The time-precise text-conditioning is enabled by (trained on) the BABEL dataset, which has frame-aligned text annotation,  unlike the more generic motion clip captioning of other common human motion datasets. The results are demonstrated in a supplementary video. These demonstrations high-quality real-time motions, with dynamically changing text prompts, results for the various modes of generation, e.g., including RL, as well as various comparisons.", "questions": "The following is a mix of minor questions and feedback.\n\n- (a) the paper discusses number of \"frames\" and \"seconds\" at various points, but to the best of my knowledge \n  it never mentions the actual frame rates (fps or Hz) of the motions.  Please do mention these numbers;  they are vital to understanding\n  the speeds.  L429 mentions \"frame rate exceeding 300\", but this presumably refers to 300 generated frames per wall-clock second\n  for the synthesis, but the reader does not know whether the actual motion frames being generated should be played at 30 Hz, 60 Hz, etc. In fairness to the authors, other papers also fail to document animation frame rates, but perhaps as a community we can reverse this trend.\n- (b) work using RL, for either kinematic or physics-based motion generation, is by default\n  autoregressive, and so may be worth furthing noting.  The biggest difference is perhaps that these \n  are usually not developed with text conditioning in mind.\n- (c) \"DART represents human motion as a collection of motion primitives\"\n  There are a great many definitions of \"motion primitive\" that have been introduced over the past 30 years,\n  and this paper introduces another.  It might be more clear to simply refer to the \"motion representation\"\n  used for DART, e.g., a short motion segment.\n- (d) High frequencies in the raw motion capture are described as being problematic.\n  However, in practice, high frequencies are necessary for high-quality representations of motions, due to the rapid\n  accelerations that are part of many motions.\n- (e) Figure 3:  Can the character perceive the environment?\n- (f) The paper would be stronger if it documented examples of some of the limitations discussed in Section 5.\n- (g) Would the output be amenable to physics-based tracking, to produce higher-quality motions, e.g., effectively\n  a physics-based \"projection\"?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Strengths:\n- The work demonstrates impressive real-time motion generation, with responsive, on-the-fly text conditioning;\n  it provides the architectural design for these to work, as well as demonstrating the utility of the BABEL dataset\n  in this context.\n- the architecture is demonstrated to be effective across three different types of usage:  (1) autoregressive conditioning;\n  (2) motion sequence optimization; (3) reinforcement learning\n- The work provides what are, to the best of my knowledge, fairly thorough ablations and comparisons.\n  The Appendices and video provide a variety of thorough motion-quality evaluation and ablation studies.", "weaknesses": "Weaknesses:\n- video: foot-ground contact cannot be observed because of the faint ambient-occlusion-based ground shadows,\n  and thus the motion quality is difficult to judge qualitatively.  Caveat: other papers on the same topic\n  often suffer from a similar problem.\n- Relevant related work is missing, e.g., \"AMD: Autoregressive Motion Diffusion\" (AAAI 2024);\n   \"Interactive character control with auto-regressive motion diffusion models\" (SIGGRAPH 2024).\n- References are often missing publication venue & other details\n- It would be interesting to understand the impact of more \"motion primitive\" durations, i.e., H+F, on the results.\n  A wider window provides more context and therefore perhaps better quality, while a shorter window may allow\n  for more flexible transitions between different types of motions.\n  I acknowledge the discussion of the H=1,F=1 case in Appendix E.3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NUdUNDLD54", "reviewer": "Reviewer_emqN", "summary": "The authors present a model that can generate long motions controlled by sequences of text prompts, in an online and real-time setting.\nThis model can additionally follow spatial control such as goal reaching.\nThe model is based on learning a compact latent space that encapsulates motion primitives.\nA diffusion model is trained using this latent space, conditioned on motion-segment prefix and a text prompt.\nThe paper presents two ways to achieve spatial control: latent noise optimization and reinforcement learning.\n\nThe paper is well-written and achieves high-quality results. However, its novelty is weak, as it combines existing techniques. I tend to accept it as it combines these existing techniques in a novel and non-trivial way, to achieve SOTA results.", "questions": "Questions and Requests:\n- In your model description, please cite works relevant to your techniques (e.g., cite MLD when you describe your latent motion space).  Although you cited such works in the RW section, mentioning them again adjacent to each technique would give a better context.\n- L251: By diffusion sampler, do you mean DDPM/DDIM, etc.? This is not clearly stated until line 295, where it is only vaguely mentioned. It should be explained earlier and more explicitly.\n- Please explain why you selected the deterministic DDIM.\n- Can the two methods for achieving spatial goals be used for the same tasks? E.g., textual control is limited in the RL method and is more feasible in the noise optimization one. Please discuss. \n- Compare optimization time vs. baseline methods. In particular, I expect optimization using DART to be faster compared to DNO due to the usage of latent space. Is that correct? Please show relevant results.\n- RL-based control: \n    - Are you using the same policy for any prompt, or do you train three different policies, one for each action? (walk/run/hop). If the former is correct, do you support any prompt, or just walk/run/hop?\n    - scene observation (L368) - \"the relative floor height to the human pelvis\": relative floor height: are you considering a flat surface only? if not, are you holding the floor height of the whole scene or just at a specific temporal step? pelvis height: at which frame? \n- Appendix A:\n    - L761: Are the locations global or relative to the root?\n    - Cite more works that use representation redundancy, e.g., HML3D.\n- L808: On one hand you argue that SMPL format is a must for your representation. On the other hand, The representation learned by the network (L758) is agnostic to betas (body definition). Please explain.\n- L849: According to the Representation paragraph (L758), dt, dJ, and dR are already part of the representation, hence part of the reconstruction loss (L833). Then why use L_aux ?\n- Appendix G rewards: It seems as if for some rewards (e.g., r_dist) \"smaller is better\", where for others (e.g., r_succ) \"higher is better. Please explain.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Good exposition\n- Qualitative results outperform prior art\n- Quantitative results are either comparable (t2m) or outperform (spatial tasks) prior art\n- Novel composition of known methods", "weaknesses": "- Weak novelty: all the techniques mentioned are already known: autoregressive generation, latent motion space usage, RL predicting z as an action space, optimization-base control, ...\n- Qualitative results contain many foot sliding and floating artifacts.\n- RL-based control:\n    - Should be compared to more than one method. In particular, there should be a comparison to methods doing running and hopping on the left leg. It can be compared to data-driven methods such as the ones used for optimization-based control, GMD (Karunratanakul et al. 2023), or the recent CLoSD (Tevet et al. 2024).\n    - Skate and floor distance are used for both reward calculation and metric calculation. Testing a model on criteria that it was optimized for raises concerns.\n- Some points still need clarification. See the \"Questions\" section.\n- Several typos: \n   - L291: \"teacle\" \n   - L917: \"w=5\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)