هل يمكنك شرح الورقة التالية بالتفصيل مع تقديم أمثلة تشرح كل فكرة بالتفصيل     DART: ADiffusion-Based Autoregressive Motion Model
 for Real-Time Text-Driven Motion Control
 Kaifeng Zhao
 Gen Li
 ETHZürich
 Siyu Tang
 https://zkf1997.github.io/DART/
 arXiv:2410.05260v1  [cs.CV]  7 Oct 2024
 ABSTRACT
 Text-conditioned human motion generation, which allows for user interaction
 through natural language, has become increasingly popular. Existing methods typi
cally generate short, isolated motions based on a single input sentence. However,
 human motions are continuous and can extend over long periods, carrying rich
 semantics. Creating long, complex motions that precisely respond to streams of text
 descriptions, particularly in an online and real-time setting, remains a significant
 challenge. Furthermore, incorporating spatial constraints into text-conditioned
 motion generation presents additional challenges, as it requires aligning the mo
tion semantics specified by text descriptions with geometric information, such as
 goal locations and 3D scene geometry. To address these limitations, we propose
 DART, a Diffusion-based Autoregressive motion primitive model for Real-time
 Text-driven motion control. Our model, DART, effectively learns a compact motion
 primitive space jointly conditioned on motion history and text inputs using latent
 diffusion models. By autoregressively generating motion primitives based on the
 preceding history and current text input, DART enables real-time, sequential mo
tion generation driven by natural language descriptions. Additionally, the learned
 motion primitive space allows for precise spatial motion control, which we formu
late either as a latent noise optimization problem or as a Markov decision process
 addressed through reinforcement learning. We present effective algorithms for
 both approaches, demonstrating our model’s versatility and superior performance
 in various motion synthesis tasks. Experiments show our method outperforms
 existing baselines in motion realism, efficiency, and controllability.
 1 INTRODUCTION
 Text-conditioned human motion generation has gained increasing popularity in recent years for
 f
 lexible user interaction via natural languages. Existing text-conditioned motion models (Tevet et al.,
 2023; Guo et al., 2023; Zhang et al., 2023a; Guo et al., 2022; Jiang et al., 2024a) primarily focus on
 generating standalone short motions from a single descriptive sentence, thereby overlooking the inher
ently sequential nature of human motion. These methods fail to generate long and complex motions
 consisting of multiple segments with distinct conditions of action descriptions. FlowMDM (Barquero
 et al., 2024) is the state-of-the-art temporal motion composition method, capable of generating
 complex, continuous motions while providing control over the desired actions and their durations.
 However, FlowMDM is an offline method that requires prior knowledge of the entire action timeline
 and has a slow generation speed, making it unsuitable for online and real-time applications.
 In addition to text-based semantic control, generating human motion within spatial constraints and
 achieving specific goals—such as reaching a keyframe body pose, following the joint trajectory,
 or interacting with objects-offers broad applications but introduces additional complex challenges.
 Recent works (Shafir et al., 2024; Karunratanakul et al., 2024; Xie et al., 2024) have sought to
 integrate text-conditioned motion models with spatial control capabilities. However, they still face
 1
challenges in simultaneously harmonizing spatial control, motion quality, and semantic alignment
 with text. Moreover, these approaches are typically restricted to controlling isolated short motions
 in an offline setting. Conversely, interactive character control (Kovar et al., 2008; Holden et al.,
 2015; Ling et al., 2020; Peng et al., 2022) has been a longstanding focus in computer graphics, with
 a primary emphasis on achieving motion realism and responsiveness to interactive control signals.
 However, most of these works lack support for text-conditioned semantic control and are limited by
 being trained on small, curated datasets. Incorporating text-conditioned motion generation could
 provide a novel, efficient language interface for animators and everyday users to control the characters,
 reducing the effort required when specifying spatial control signals is challenging or tedious.
 To address these limitations, we propose DART, a diffusion-based autoregressive motion primitive
 model for real-time text-conditioned motion composition. Moreover, the compact and expressive
 motion space of DART provides a foundation for integrating precise spatial control through latent
 space optimization or reinforcement learning-based algorithms. DART features three key components.
 First, DART represents human motion as a collection of motion primitives (Zhang & Tang, 2022),
 which are autoregressive representations consisting of overlapping short motion segments tailored for
 online generation and control. These short primitives also provide a clearer alignment with atomic
 action semantics compared to longer sequences, enabling effective learning of a text-conditioned
 motion space. By focusing on shorter primitives, DART avoids the complexities and extensive data
 demands of modeling entire motion sequences, allowing for high-quality motion generation with
 only a few diffusion steps.
 Next, DART learns a text-conditioned motion primitive space from large-scale data using a latent
 diffusion architecture, which contains a denoiser network for generating motions in a compact latent
 motion space, and a decoder network for reconstructing the detailed motions. Leveraging this learned
 latent motion space, DART employs an autoregressive rollout to synthesize motion sequences from
 real-time text inputs, enabling the efficient generation of motions of arbitrary length. Compared with
 the offline temporal motion composition method FlowMDM (Barquero et al., 2024), DART provides
 real-time response and ten times faster generation speed.
 Lastly, the versatile and powerful motion space learned in DART enables precise spatial control
 through latent space optimization or RL-based policy learning. We present effective learning and
 optimization algorithms that explore the learned DART motion space to obtain motion sequences that
 precisely follow both textual and spatial constraints. We evaluate DART across various motion synthe
sis tasks, including generating long, continuous sequences from sequential text prompts, in-between
 motion generation, scene-conditioned motion, and goal-reaching synthesis. The experimental results
 show that DART is a simple, unified and highly effective motion model, consistently outperforming
 or matching the performance of baselines.
 2 RELATED WORKS
 Conditional Motion Generation. Generating realistic and diverse human motions is a long-standing
 challenge in computer vision and graphics. Apart from generating highly realistic human motions
 (Kovar et al., 2008; Holden et al., 2020; Büttner & Clavet, 2015; Zinno, 2019), conditional generation
 is another important factor that aligns motion generation with human intentions and various applica
tion constraints. Text-conditioned motion generation (Tevet et al., 2023; Zhang et al., 2022; Petrovich
 et al., 2022; Guo et al., 2022; Jiang et al., 2024a; Zhang et al., 2023a) has become increasingly
 popular since it allows users to modulate motion generation with flexible natural languages. Audio
 and speech-driven motion synthesis methods (Alexanderson et al., 2023; Tseng et al., 2022; Siyao
 et al., 2022; Ao et al., 2022; 2023) have also made significant progress recently. Moreover, there exist
 many applications that require spatial awareness and precise control in motion generation, such as
 interactive character control (Ling et al., 2020; Peng et al., 2022; Starke et al., 2022; Luo et al., 2023;
 Starke et al., 2024), human-scene interactions(Hassan et al., 2021; Starke et al., 2019; Zhao et al.,
 2022; 2023; Li et al., 2024; Xu et al., 2023; Jiang et al., 2024b; Wang et al., 2024; Liu et al., 2023; Li
 et al., 2023a), and human-human(noid) interactions (Liang et al., 2024; Zhang et al., 2023c; Christen
 et al., 2023; Cheng et al., 2024; Shan et al., 2024). Synthesizing high-quality motions with precise
 spatial control remains challenging, and DART is a step toward a general and efficient motion model
 that supports precise control tasks.
 2
Diffusion Generative Models. Denoising Diffusion Models (Ho et al., 2020; Song et al., 2020a;b)
 are generative models that learn to predict clean data samples by gradually annealing the noise from
 a standard Gaussian sample. Diffusion models have achieved unprecedented performances in many
 generation tasks including images, videos, and 3D human motions (Tevet et al., 2023; Rombach et al.,
 2022; Ho et al., 2022). Diffusion models can accept flexible conditions to modulate the generation,
 such as text prompts, images, audio, and 3D objects (Rombach et al., 2022; Tevet et al., 2023;
 Alexanderson et al., 2023; Tseng et al., 2022; Zhang et al., 2023b; Xu et al., 2023; Li et al., 2023b).
 Most existing diffusion-based motion generation methods focus on offline generations (Tevet et al.,
 2023; Barquero et al., 2024; Chen et al., 2023; Karunratanakul et al., 2024; Cohan et al., 2024; Dai
 et al., 2024) producing short, isolated motion sequences at once while neglecting the autoregressive
 nature of human motions. Among these methods, DNO (Karunratanakul et al., 2024) is closely related
 to our optimization-based control approach, as both use diffusion noises as the latent space for editing
 and control. However, the key distinction lies in our latent motion primitive-based diffusion model,
 which, unlike DNO’s diffusion model trained with full motion sequences in raw space, achieves
 superior performance in harmonizing spatial control with text semantic alignment during experiments.
 There are also works that incorporate history conditions in diffusion-based motion prediction. Recent
 works (Xu et al., 2023; Jiang et al., 2024b) use history-conditioned diffusion models to autoregressive
 generate human-scene interaction sequences. Van Wouwe et al. (2024) uses autoregressive diffusion
 for real-time human motion reconstruction from IMU sensors. Shi et al. (2024) and Chen et al.
 (2024) adapt diffusion models for real-time character motion generation and control. While Shi
 et al. (2024) learns character control policies using diffusion noises as the action space, akin to our
 reinforcement-learning-based control, their method focuses on single-frame autoregressive generation
 and lacks support for text conditions, which offer a compact and powerful interface for users to
 control character behaviors. In contrast, DART is an efficient and general motion model that scales
 effectively to large motion datasets with text annotations (Mahmood et al., 2019; Punnakkal et al.,
 2021). DART supports natural language interfacing and provides a versatile foundation for various
 motion generation tasks with spatial control.
 3 METHOD
 3.1 PRELIMINARIES
 Problem Definition. We focus on the task of text-conditioned online motion generation with spatial
 control. Given an H frame seed motion Hseed = [h1,...,hH], a sequence of N text prompts
 C =[c1,...,cN], and spatial goals g, the objective is to autoregressively generate continuous and
 realistic human motion sequences M = [Hseed,X1,...,XN], where each motion segment Xi
 matches the semantics of the corresponding text prompt ci and satisfies the spatial goal constraints
 g. This task imposes challenges in high-level action semantic control, precise spatial control, and
 smooth temporal transition in motion generation.
 Autoregressive Motion Primitive Representation. We model long-term human motions as the
 sequential composition of motion primitives (Zhang & Tang, 2022) with overlaps for efficient gen
erative learning and online inference. In contrast to directly modeling long motion sequences, the
 primitive representation decomposes globally complex sequences into short and simple primitives,
 resulting in a more tractable data distribution for generative learning. This enables DART to learn
 high-quality primitive generation using as few as 10 diffusion steps. Moreover, the autoregressive
 and simple nature of the primitive representation makes it inherently suitable for fast online genera
tions. Furthermore, motion primitives convey more interpretable semantics than individual frames,
 enhancing the learning of text-conditioned motion space.
 Each motion primitive Pi = [Hi,Xi] is a short motion clip containing H frames of history motion
 Hi =[hi,1,...,hi,H] that overlap with the previous motion primitive, and F frames of future motion
 Xi = [xi,1,...,xi,F]. The history motion of the i-th motion primitive Hi consists of the last H
 frames of the previous motion primitive Xi−1,F−H+1:F. Therefore, infinitely long motions can be
 represented as the rollout of such overlapping motion primitives as M = [Hseed,X1,...,XN].
 We represent each motion frame using an overparameterized representation based on the SMPL
X(Pavlakos et al., 2019) parametric human body model. The overparameterized representation is
 a D =276dimensional vector including the body root translation t, root orientation R, local joint
 rotations θ, joint locations J, and the temporal difference features of locations and rotations. All
 3
Positional encoding
 Addition 
Switch
 ❄ Freeze
 ~
 𝑧
 𝑥
 …… 𝑥
 𝑧
 Linear
 PE
 𝑇 𝑇
 Linear
 Transformer
 Linear
 …
 ℎ ℎ
 …
 Encoder
 Transformer
 ……
 𝑥 𝑥
 PE
 Linear
 …
 ℎ ℎ
 0
 …
 Decoder
 ……
 Linear
 Transformer
 0
 PE
 MLP
 Noising
 step t
 Linear
 Linear
 …
 ℎ ℎ 𝑧
 CLIP text
 ❄
 Text prompt
 “run in a circle”
 …
 Latent Denoiser
 Figure 1: Architecture illustration of DART. The encoder network compresses the future frames
 X =[x1,...,xF] into a latent variable, conditioned on the history frames H = [h1,...,hH]. The
 decoder network reconstructs the future frames conditioned on the history frames and latent sample
 z. The denoiser network predicts the clean latent sample ˆ z0 conditioned on the noising step, text
 prompt, history frames, and noised latent sample zt. During the denoiser training, both the encoder
 and decoder networks remain frozen.
 the frames within one motion primitive are canonicalized in a local coordinate frame centered at
 the first-frame body of the primitive. We use history length H = 2 and future length F = 8 in our
 experiments. Further details of the primitive representation are attached in the Appendix A.
 3.2 DART: A DIFFUSION-BASED AUTOREGRESSIVE MOTION PRIMITIVE MODEL
 Wepropose a novel latent diffusion model (Rombach et al., 2022; Chen et al., 2023) designed for
 motion primitives. This model can autoregressively generate motions with smooth transitions, condi
tioned on text prompts and motion history. The proposed model contains a variational autoencoder
 (VAE) (Kingma & Welling, 2022) that compresses the motion primitives into a compact latent space,
 and a latent denoising diffusion model that predicts clean latent variables from noises, conditioned on
 text prompts and motion history.
 Latent Motion Primitive Space. We introduce a motion primitive VAE that compresses motion
 primitives into a compact latent space, upon which we train our latent diffusion models, rather than
 using the raw motion space (Rombach et al., 2022; Chen et al., 2023). This design is inspired by
 the observations that raw motion data from motion capture datasets (Mahmood et al., 2019) often
 contain various levels of high-frequency noises, and training diffusion models on raw motion space
 leads to results inheriting such noise artifacts. This is evidenced by the significantly higher jittering in
 generated motions of the ablative model without VAE in Appendix. E.3. The compression achieved
 through the motion primitive VAE significantly mitigates the impact of high-frequency noise. The
 resulting latent representation is not only more compact but also more computationally efficient than
 the raw motion data, thereby enhancing the efficiency of our generative model and improving the
 control capabilities within the latent space.
 Our motion primitive VAE employs a transformer-based architecture based on Chen et al. (2023),
 comprising an encoder E and decoder D, as shown in Fig. 1. The encoder takes as input the history
 motion frames H and future motion frames X as well as the learnable distribution tokens Tµ and Tσ,
 which are responsible for predicting the latent distribution. The latent sample z is obtained from the
 predicted distribution via reparameterization (Kingma & Welling, 2022). The decoder then predicts
 the future frames ˆ
 Xfrom zero tokens conditioned on the latent sample z and the history frames H.
 The motion primitive VAE is trained with the future frame reconstruction loss Lrec, auxiliary losses
 Laux that penalize unnatural motion reconstruction, and a small Kullback-Leibler (KL) regularization
 term LKL. We refer to Appendix C for further details about the motion primitive VAE.
 4
Algorithm 1 Autoregressive rollout generation using latent motion primitive model
 Input: primitive decoder D, latent variable denoiser G, history motion seed Hseed, text prompts
 C =[c1,...,cN], total diffusion steps T, classifier-free guidance scale w, diffusion sampler S.
 Optional Input: Latent noises ZT = [z1
 T,...,zN
 T ]
 Output: motion sequence M
 H←Hseed
 M←Hseed
 for i ← 1to N do
 sample noise zi
 T from N(0,1) if not inputted
 ˆ
 zi
 0 ← S(G,zi
 T,T,H,ci,w)
 ▷ number of rollouts
 ▷ diffusion sample loop with classifier-free guidance
 ˆ
 X←D(H,ˆ zi
 0)
 M←CONCAT(M, ˆ X)
 ▷ concatenate future frames to generated sequence
 H←CANONICALIZE(ˆ XF−H+1:F) ▷ update the rollout history with last H generated frames
 end for
 return M
 Latent Motion Primitive Diffusion. Building on the motion representation provided by our trained
 motion primitive encoder E, we transform the motion primitives from the raw feature space to
 a more refined latent space that is more suitable for generative learning with diffusion models.
 This latent motion primitive space is then used to train a diffusion denoiser for generating motion
 primitives conditioned on text prompts. The denoiser architecture is shown in Fig. 1. Given the
 noised latent variable zt, the diffusion step t, the history motion H, and a text prompt c, we train a
 transformer-based latent denoiser model G to predict the cleaned motion primitive latent variable
 ˆ
 z0 = G(zt,t,H,c). The diffusion step is embedded using a small MLP, while the text prompt is
 encoded using the CLIP (Radford et al., 2021) text encoder. The text prompt is randomly masked
 out by a probability of 0.1 during training to enable classifier-free guidance (Ho & Salimans, 2022)
 during generation. During training, the motion primitive latent variable z0 = E(H,X) can be
 efficiently obtained with a single inference of the fixed encoder E. The cleaned latent variable ˆ z0 can
 be converted back to the future frames using the fixed decoder D: ˆ
 X=D(H,ˆ z0). Thelatent denoiser
 G is trained with the simple objective (Ho et al., 2020). Additionally, we use the reconstruction loss
 Lrec and auxiliary losses Laux on the decoded future frames ˆ X, the same as training the motion
 primitive VAE. Notably, we use DDPM (Ho et al., 2020) with only 10 diffusion steps during both
 the training and inference of the denoiser model. This is sufficient for generating realistic samples
 due to the simplicity of the short motion primitives and enables highly efficient online generation.
 Moreover, we use the scheduled training (Ling et al., 2020; Bengio et al., 2015; Rempe et al., 2021)
 to progressively introduce the test-time distribution of the history motion H, which improves the
 stability of long sequence online generation and the text prompt controllability for unseen poses. We
 refer to Appendix D for the details.
 With the trained motion primitive decoder D, latent denoiser G and a diffusion sampler S such as Ho
 et al. (2020) and Song et al. (2022), we can autoregressively generate motion sequences given the
 history motion seed Hseed and the online sequence of text prompts C, as shown in Alg. 1. During
 sampling, we use classifier-free guidance (Ho & Salimans, 2022) on the text condition with a guidance
 scale w:
 Gw(zt,t,H,c) = G(zt,t,H,∅) + w · (G(zt,t,H,c) − G(zt,t,H,∅))
 (1)
 Using the rollout algorithm, DART generates over 300 frames per second using a single RTX 4090
 GPU, enabling real-time applications and online reinforcement-learning control as in Sec. 3.3.
 3.3 CONTROLLABLE MOTION SYNTHESIS VIA DART
 Text-conditioned motion generation offers a user-friendly interface for controlling motions through
 natural language. However, relying solely on text limits precise spatial control, such as walking to
 a specific location or sitting in a designated spot. Therefore, it is necessary to incorporate motion
 control mechanisms to achieve precise spatial goals, including reaching a keyframe body pose,
 following joint trajectories, and interacting with scene objects.
 5
Algorithm 2 Latent noises optimization
 Input: Latent noises ZT = [z1
 T,...,zN
 T ], Optimizer O, learning rate η, criterion L, goal g (for
 brevity, we do not reiterate the inputs of the rollout function defined in Alg. 1)
 Output: motion sequence M
 for i ← 1to optimization steps do
 M←ROLLOUT(ZT,...)
 ∇←∇ZT
 L(M)
 ZT ←O(ZT,∇/∥∇∥,η)
 end for
 return M ← ROLLOUT(ZT)
 ▷ update using normalized gradient
 We formulate the motion control task as generating the motion sequence M that minimizes its
 distance to a given spatial goal g under a criterion function F(·,·) and the regularization from the
 scene and physical constraints cons(·):
 M∗ =argminMF(Π(M),g)+cons(M)
 (2)
 where g is the task-dependent spatial goal, e.g., g is a keyframe body for the motion in-between task
 or the goal location for the navigation task. Π(·) is the goal-dependent projection function that maps
 motion sequence to the goal observation space, cons(·) denotes physical constraints such as avoiding
 collisions into scenes.
 Directly solving the motion control task in the raw motion space often results in unrealistic motions
 since most samples in the raw motion space do not represent plausible motions. To improve the
 generated motion quality, many previous methods teacle such motion control tasks in a latent motion
 space, where samples can be mapped to plausible motions (Karunratanakul et al., 2024; Ling et al.,
 2020; Peng et al., 2022; Holden et al., 2015). DART offers a powerful text-conditioned latent motion
 space for such latent space control. DART learns a powerful generative model that can generate
 diverse and realistic motions from standard Gaussian samples ZT. Using the deterministic DDIM
 (Song et al., 2022) sampler, we adapt DART sampling to function as a deterministic mapping from
 latent noises ZT to plausible motions. Using the latent noises ZT as the latent space, we transform
 the motion control task in Eq. 2 to latent space control as follows: Given the initial motion history
 Hseed, a sequence of text prompts C, and a deterministic diffusion sampler S, the rollout function in
 Alg. 1 can deterministically map a list of motion primitive latent noises ZT = [z1
 T,..,zN
 T ] to a motion
 sequence M by :
 M=ROLLOUT(ZT,D,G,Hseed,C,T,w,S)
 (3)
 Weabbreviate it as ROLLOUT(ZT,...) for short since ZT is the only variable in latent control. The
 minimization objective is converted as :
 Z∗
 T = argminZT
 F(Π(ROLLOUT(ZT,...),g) + cons(ROLLOUT(ZT,...))
 (4)
 Note that we do not use DDIM to skip diffusion steps at sampling, which we observe to cause artifacts
 in generated motion. We then propose two solutions to this latent space motion control problem, one
 is to directly optimize the latent noises using gradient descent, and the other is to model the control
 task as a Markov process and use reinforcement learning to learn control policies.
 Motion Control via Latent Diffusion Noise Optimization. One straightforward solution to this
 minimization problem (Eq. 4) is to directly optimize the latent noises ZT given the criterion function
 using gradient descent methods. The latent noise optimization is illustrated in Alg. 2.
 This optimization-based control framework is general and applicable for various spatial control tasks.
 Weinstantiate the latent noise optimization method in two example control scenarios: in-between
 motion generation and human-scene interaction generation. First, we addresss the motion in-between
 task that aims to generate the motion frames transition between given history and goal keyframes g
 that is f frames away conditioned on the text prompt c. We use the distance between the f-th frame
 of the generated motion and the goal keyframe as the optimization objective.
 6
~
 𝑧
 Actor
 Critic
 Linear
 Linear
 Linear
 Text prompt 𝑐
 CLIP embedding
 History Motion
 𝑯
 Linear
 Linear
 Goal
 𝑔
 Scene
 𝑠
 DART
 ❄
 FutureMotion
 𝑿
 Canonicalization
 𝑯
 Figure 2: Architecture of the reinforcement learning-based control policy. The frozen DART diffusion
 denoiser and decoder models transform the latent actions into motion frames. The last predicted
 frames are canonicalized and provided to the policy model as the next step history condition.
 Second, we show that physical and scene constraints cons(·) can be incorporated to synthesize human
 motions in a contextual environment. Given the text prompts C and spatial goal g of the interaction
 anchor joint locations, e.g., locations of the pelvis when sitting, the objective is to generate motions
 that not only perform the desired interaction but also achieve the goal joint positions while adhering
 to scene constraints. We consider two scene constraints penalizing the human-scene collision and
 foot-floating artifacts, as detailed in Appendix. F.
 Motion Control via Reinforcement Learning. Although the proposed latent noise optimization is
 effective for general control tasks, the optimization can be computationally expensive. To address this,
 the autoregressive primitive-based motion representation of DART allows for another efficient control
 mechanism using reinforcement learning (RL) (Sutton & Barto, 1998). We model the latent motion
 control as a Markov decision process with a latent action space and use RL to learn policy models to
 achieve the goals. We model digital humans as agents to interact with an environment according to a
 policy to maximize the expected discounted return. At each time step i, the agent observes the state
 si of the system, samples an action ai from the learned policy, with the system transitioning to the
 next state si+1 due to the performed action ai, and receives a reward ri = r(si,ai,si+1).
 Our latent motion primitive model naturally fits in the Markov decision process due to its autore
gressive nature. We use the latent noises zT as the latent action space and train goal-conditioned
 policy models as controllers. The policy architecture is shown in Fig. 2. Our policy model employs
 the actor-critic (Sutton & Barto, 1998) architecture and is trained with the PPO (Schulman et al.,
 2017) algorithm. The state si includes the history motion observation Hi, goal observation gi, scene
 observation si, and the CLIP embedding of the text prompt ci. The policy model takes in [Hi, gi, si,
 ci] to predict the latent noise zi
 T as the action. The latent noise zi
 T is mapped to the future motion
 frames Xi using the frozen latent denoiser G and motion primitive decoder D. The new history
 motion is extracted from the last H predicted frames and fed to the policy network in the next step.
 Wereformulate the minimization problem in Eq. 4 as reward maximization to train the policy.
 Weinstantiate the reinforcement learning control with the text-conditioned goal-reaching task. Given
 a text prompt c and a 2D goal location g, we aim to control the human to reach the goal location
 using the action specified by the text. The goal location is converted to a local observation consisting
 of its distance to the human pelvis, and its local direction in the human-centric coordinates frame.
 Weconsider a simple flat scene and the scene observation is the relative floor height to the human
 pelvis. The policy is trained with distance rewards encouraging the human pelvis to reach the goal
 location and scene constraint rewards penalizing foot skating and floor penetration. Further details
 can be found in appendix G. With the trained control policies, we can efficiently control a human to
 reach dynamic goals using specified skills like walking or hopping.
 7
4 EXPERIMENTS
 We provide extensive experiments showing how DART can serve as a general model for text
conditioned temporal motion composition (4.1) and various motion generation tasks requiring precise
 spatial control via latent noise optimization (4.2) and reinforcement learning policy (4.3). The
 qualitative results and comparisons are attached in the supplementary video.
 Our DARTis trained on motion-text data from the BABEL (Punnakkal et al., 2021) dataset in our
 experiments if not otherwise stated. BABEL contains motion capture sequences with frame-aligned
 text labels that annotate the fine-grained semantics of actions. Fine-grained text labels in BABEL
 allow models to learn precise human action controls and natural transitions among actions. However,
 DART can also learn using motions with coarse sequence-level labels such as the HML3D (Guo
 et al., 2022) dataset, as in the optimization-based motion in-between experiments in Sec. 4.2.
 4.1 TEXT-CONDITIONED TEMPORAL MOTION COMPOSITION
 Text-conditioned temporal motion composition aims to generate realistic motion sequences that
 faithfully align with a list of action segments, each defined by a specific text prompt and duration.
 Weevaluate the motion composition task on the BABEL(Punnakkal et al., 2021) dataset consisting of
 motion capture sequences with human-annotated per-frame action descriptions, which facilitate the
 evaluation of precise action controls and natural transitions in motion composition. Since BABEL
 does not release the test set, we compare our DART with baseline methods on the BABEL validation
 set. We extract the list of action segments described by tuples of text prompts and durations from
 each data sequence and feed the action lists as conditions for motion composition.
 Weevaluate the generation results using metrics proposed in Guo et al. (2022) and Barquero et al.
 (2024). For each action segment, we evaluate the similarity between generation and dataset (FID),
 motion-text semantic alignment (R-prec, MM-DIST), and generation diversity (DIV). To evaluate
 smooth transitions between two segments, we measure the jerk (the derivative of acceleration) of
 the 30-frame transition clip centered at the splitting point of two action segments, reporting the
 peak jerk (PJ) and Area Under the Jerk (AUJ). Moreover, we profile all methods in a benchmark of
 generating one 5000-frame-long sequence and report the generation speed, the latency of getting the
 f
 irst generated frames, and memory usage. We also conduct binary user studies to evaluate the motion
 realism and motion-text semantic alignment of generation results, during which the participants are
 given generation results from two different methods and are asked to select the generation that is
 perceptually more realistic or better aligns with the action text stream in subtitles. We compare
 DARTwith baselines inlcuding TEACH (Athanasiou et al., 2022), DoubleTake (Shafir et al., 2024), a
 history-conditioned modification of T2M-GPT (Zhang et al., 2023a)(denoted as T2M-GPT*), and the
 state-of-the-art offline motion composition method FlowMDM (Barquero et al., 2024).
 We present the quantitative results in Tab. 1 and Tab. 2. DART achieves the best FID in both the
 segment and transition evaluation, indicating the highest similarity to the dataset and best motion
 realism. DART also displays second-best jerk metrics indicating smooth action transitions. We
 observe that DART performs slightly worse than FlowMDM in motion-text semantic alignment
 (R-prec and MM-Dist) because of the online generation nature of DART. Natural action transitions
 require time to transit to the new action after receiving the new action prompt, leading to a delay in the
 emergence of the new action semantics. For instance, a human cannot immediately transition from
 kicking a leg in the air to stepping backward without first recovering to a standing pose. This inherent
 delay in transitions results in a motion embedding shift that impacts the R-prec metric of DART. In
 contrast, the offline baseline FlowMDM generates the entire sequence as a whole and requires oracle
 information of the full timeline of action segments to modulate compatibility between subsequent
 segments. The slight but natural action transition delay of DART is perceived as natural by humans,
 as shown in the user study results in Tab. 2 where DART is preferred over all the baselines, including
 FlowMDM, for both motion realism and motion-text semantic alignment in human evaluations.
 DARTrequires significantly less memory than the offline baseline FlowMDM and achieves approx
imately 10x faster generation, with a frame rate exceeding 300 and a latency of 0.02s, enabling
 real-time text-conditioned motion composition (see supplementary video). We refer to Appendix E
 for details of experiments, user studies, and ablation studies about model architecture and hyperpa
rameters, and refer to the supplementary video for qualitative comparisons.
 8
Table 1: Quantitative evaluation results on text-conditioned temporal motion composition. The first
 row includes the metrics of the dataset for reference. Symbol ‘→’ denotes that closer to the dataset
 reference is better and ‘±’ indicates the 95% confidence interval. Bold and blue texts indicate the
 best and second best results excluding the dataset, respectively.
 Segment Transition Profiling
 FID↓
 R-prec↑
 DIV → MM-Dist↓ FID↓ DIV→ PJ→ AUJ↓ Speed(frame/s)↑ Latency(s)↓ Mem.(MiB)↓
 Dataset 0.00±0.00 0.72±0.00 8.42±0.15 3.36±0.00 0.00±0.00 6.20±0.06 0.02±0.00 0.00±0.00
 TEACH 17.58±0.04 0.66±0.00 10.02±0.06 5.86±0.00 3.89±0.05 5.44±0.07 1.39±0.01 5.86±0.02 3880±144
 0.05±0.00
 DoubleTake 7.92±0.13 0.60±0.01 8.29±0.16 5.59±0.01 3.56±0.05 6.08±0.06 0.32±0.00 1.23±0.01 85±1
 T2M-GPT* 7.71±0.55 0.49±0.01 8.89±0.21 6.69±0.08 2.53±0.04 6.61±0.02 1.44±0.03 4.10±0.09 885±12
 FlowMDM 5.81±0.10 0.67±0.00 8.90±0.06 5.08±0.02 2.39±0.01 6.63±0.08 0.04±0.00 0.11±0.00 31±0
 59.11±0.76
 0.23 ±0.00
 161.29±0.24
 2251
 1474
 2172
 11892
 Ours 3.79±0.06 0.62±0.01 8.05±0.10 5.27±0.01 1.86±0.05 6.70±0.03 0.06±0.00 0.21±0.00 334±2
 0.02±0.00
 2394
 Table 2: Binary user study results comparing our method against baselines in generation realism and
 motion-text semantic alignment on text-conditioned temporal motion composition. We report the
 percentage of each method being voted better than the other (Ours vs. Baselines).
 Realism (%)
 Semantic (%)
 Ours vs. TEACH
 Ours vs. DoubleTake
 Ours vs. T2M-GPT*
 Ours vs. FlowMDM
 66.7 vs. 33.3
 66.4 vs. 33.6
 61.3 vs. 38.7
 53.3 vs. 46.7
 66.0 vs. 34.0
 66.1 vs. 33.9
 66.7 vs. 33.3
 51.3 vs. 48.7
 4.2 LATENT DIFFUSION NOISE OPTIMIZATION-BASED CONTROL USING DART
 Text-conditioned motion in-between. Motion in-betweening aims to generate realistic motion
 frames that smoothly transition between a pair of history and goal keyframes. We consider a text
conditioned variant where an additional text prompt is inputted to specify the action semantics
 of the frames in between. We compare our method with DNO (Karunratanakul et al., 2024) and
 OmniControl (Xie et al., 2024). For a fair comparison, we train DART on the HML3D dataset same
 as the baselines. We evaluate using test sequences covering diverse actions, with the sequence lengths
 ranging from 2 to 4 seconds. The quantitative evaluations are shown in Tab. 3. We report the L2 norm
 errors between the generated motion and the history motion and goal keyframe. We also evaluate
 the motion realism with the skate and jerk metrics. The skate metric (Ling et al., 2020; Zhang et al.,
 2018) calculates a scaled foot skating when in contact with the floor: s = disp · (2 − 2h/0.03),
 where disp is the foot displacement in two consecutive frames, h denotes the higher foot height in
 consecutive frames and 0.03m is the threshold value for contact. We do not calculate skate metric
 for sequences where the feet are not on a flat floor, such as crawling and climbing down stairs. Our
 method can generate the motions closest to the keyframe and show fewer skating and jerk artifacts.
 Our method effectively preserves the semantics specified by the text prompts, while the baseline
 DNOoccasionally ignores the text prompts to reach the goal keyframe, as illustrated in the examples
 of pacing in circles and dancing in the supplementary video. This highlights the superior capability of
 our latent motion primitive-based DART in harmonizing spatial control and text semantic alignment.
 Table 3: Quantitative evaluation of noise optimization-based In-between. The best results except
 dataset are in bold and ‘±’ indicates the 95% confidence interval.
 History error (cm)↓ Goal error (cm)↓ Skate (cm/s)↓ Jerk↓
 Dataset-- 2.83±0.00 0.99±0.00
 OmniControl 17.22±0.80 5.88±0.57 5.48±0.90 1.26±0.10
 DNO 2.15±0.24 5.52±1.86 5.12±0.55 0.72±0.07
 DARTOpt.(ours) 0.00±0.00 0.54±0.03 3.97±0.96 0.71±0.03
 Human-scene interaction. We qualitatively show that our latent noise optimization control can
 be applied to human-scene interaction synthesis, where the goal is to control the human to interact
 naturally with the surrounding environment. Given the text prompts specifying the actions and
 durations, we use latent noise optimization to control the human to reach the goal joint location while
 adhering to the scene contact and collision constraints. The scenes are represented as signed distance
 f
 ields for evaluating human-scene collision and contact constraints. We present generated interactions
 of climbing stairs and walking to sit on a chair in Fig. 3 and the supplementary video.
 9
(a) Walk, turn left, sit on the chair
 (b) Walk upstaris
 (c) Walk downstaris
 Figure 3: Illustrations of human-scene interaction generation given text prompts and goal pelvis joint
 location (visualized as a red sphere). Best viewed in the supplementary video.
 4.3 REINFORCENT LEARNING-BASED CONTROL USING DART
 Combining DARTwith reinforcement learning-based control, we train text-conditioned goal-reaching
 policies for three actions including ‘walk’, ‘run’, and ‘hop on the left leg’. We evaluate the controllers
 on paths consisting of sequences of waypoints. We compare our method to a baseline GAMMA
 (Zhang & Tang, 2022), which learns a motion model using the same motion source of AMASS
 (Mahmood et al., 2019) dataset as ours and trains RL controllers to reach goal locations. However,
 GAMMAdoesnotsupport text conditioning and can only generate walking motion. The evaluation
 metrics include the reach time, the success rate of reaching the final goal waypoint, foot skating, and
 foot-floor distance. The evaluation results are shown in Tab. 4. Our controller consistently reaches
 all goals within a reasonable timeframe, while GAMMA occasionally fails to meet the final goal
 and may float off the floor beyond the contact threshold. Our text-conditioned goal-reaching policy
 achieves a generation speed of 240 frames per second. We believe our results on text-conditioned
 goal-reaching controllers demonstrate the potential of DART as a foundational human motion model,
 upon which control models for multiple tasks can be learned without collecting separate datasets for
 different tasks.
 Table 4: Quantitative evaluation of text-conditioned goal-reaching controller. The best results are in
 bold and ‘±’ indicates the 95% confidence interval.
 Time (s)↓ Successrate↑ Skate(cm/s) ↓ Floordistance (cm)↓
 GAMMAwalk 31.44±2.58 0.95±0.03 5.14±1.58 5.55±0.84
 Ours ‘walk’ 13.82±0.10 1.0±0.0 5.07±0.18 1.87±0.02
 Ours ‘run’ 12.16±0.06 1.0±0.0 4.70±0.13 2.02±0.02
 Ours ‘hop on left leg’ 13.89 ± 0.12 1.0±0.0 41.18±0.61 1.43±0.02
 5 LIMITATIONS AND CONCLUSIONS
 Our method relies on motion sequences with frame-level aligned text annotations, as provided in BA
BEL, to achieve precise text-motion alignment and natural transition between actions. When trained
 on the coarse sentence-level motion labels from HML3D, the text-motion alignment degenerates for
 texts describing multiple actions, resulting in motions randomly switching between the described
 actions in a random order. This occurs because each short motion primitive inherently matches only a
 portion of the sequence’s semantics. Using a coarse sentence-level description as the text label for a
 primitive introduces semantic misalignment and ambiguity. Inspired by Stoffl et al. (2024), we aim to
 explore hierarchical latent spaces to effectively tackle both fine-grained and global sequence-level
 semantics in future work.
 Our method DART effectively learns a text-conditioned motion primitive space that enables real-time
 online motion generation driven by natural languages. Additionally, the learned powerful motion
 primitive space allows for precise spatial motion control via latent noise optimization or reinforcement
 learning policies. Experiments demonstrate the superiority of DART in harmonizing spatial control
 with motion text-semantic alignment in the generated motions.
 10
REFERENCES
 Simon Alexanderson, Rajmund Nagy, Jonas Beskow, and Gustav Eje Henter. Listen, denoise, action!
 audio-driven motion synthesis with diffusion models. 2023.
 Marcin Andrychowicz, Anton Raichuk, Piotr Sta´ nczyk, Manu Orsini, Sertan Girgin, Raphael Marinier,
 Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, et al. What matters in
 on-policy reinforcement learning? a large-scale empirical study. arXiv preprint arXiv:2006.05990,
 2020.
 Tenglong Ao, Qingzhe Gao, Yuke Lou, Baoquan Chen, and Libin Liu. Rhythmic gesticulator:
 Rhythm-aware co-speech gesture synthesis with hierarchical neural embeddings. ACM Transac
tions on Graphics (TOG), 41(6):1–19, 2022.
 Tenglong Ao, Zeyi Zhang, and Libin Liu. Gesturediffuclip: Gesture diffusion model with clip latents.
 ACMTransactions on Graphics (TOG), 42(4):1–18, 2023.
 Nikos Athanasiou, Mathis Petrovich, Michael J. Black, and Gül Varol. TEACH: Temporal Action
 Composition for 3D Humans, September 2022.
 German Barquero, Sergio Escalera, and Cristina Palmero. Seamless human motion composition with
 blended positional encodings. 2024.
 Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled Sampling for Sequence
 Prediction with Recurrent Neural Networks, September 2015.
 Michael Büttner and Simon Clavet. Motion matching-the road to next gen animation. Proc. of Nucl.
 ai, 2015(1):2, 2015.
 Rui Chen, Mingyi Shi, Shaoli Huang, Ping Tan, Taku Komura, and Xuelin Chen. Taming Diffusion
 Probabilistic Models for Character Control, April 2024.
 Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu. Executing your
 commands via motion diffusion in latent space. In Proceedings of the IEEE/CVF Conference on
 Computer Vision and Pattern Recognition, pp. 18000–18010, 2023.
 Xuxin Cheng, Yandong Ji, Junming Chen, Ruihan Yang, Ge Yang, and Xiaolong Wang. Expressive
 whole-body control for humanoid robots. arXiv preprint arXiv:2402.16796, 2024.
 Sammy Christen, Wei Yang, Claudia Pérez-D’Arpino, Otmar Hilliges, Dieter Fox, and Yu-Wei
 Chao. Learning human-to-robot handovers from point clouds. In Proceedings of the IEEE/CVF
 Conference on Computer Vision and Pattern Recognition, pp. 9654–9664, 2023.
 Setareh Cohan, Guy Tevet, Daniele Reda, Xue Bin Peng, and Michiel van de Panne. Flexible motion
 in-betweening with diffusion models. arXiv preprint arXiv:2405.11126, 2024.
 Wenxun Dai, Ling-Hao Chen, Jingbo Wang, Jinpeng Liu, Bo Dai, and Yansong Tang. Mo
tionlcm: Real-time controllable motion generation via latent consistency model. arXiv preprint
 arXiv:2404.19759, 2024.
 Ross Girshick. Fast R-CNN, September 2015.
 Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating
 Diverse and Natural 3D Human Motions from Text. In 2022 IEEE/CVF Conference on Computer
 Vision and Pattern Recognition (CVPR), pp. 5142–5151, New Orleans, LA, USA, June 2022. IEEE.
 ISBN 978-1-66546-946-3. doi: 10.1109/CVPR52688.2022.00509.
 Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. Momask: Generative
 masked modeling of 3d human motions. 2023.
 Mohamed Hassan, Duygu Ceylan, Ruben Villegas, Jun Saito, Jimei Yang, Yi Zhou, and Michael J
 Black. Stochastic scene-aware motion prediction. pp. 11374–11384, 2021.
 Jonathan Ho and Tim Salimans. Classifier-Free Diffusion Guidance, July 2022.
 11
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in
 neural information processing systems, 33:6840–6851, 2020.
 Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J
 Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633–8646,
 2022.
 Daniel Holden, Jun Saito, Taku Komura, and Thomas Joyce. Learning motion manifolds with
 convolutional autoencoders. In SIGGRAPH Asia 2015 technical briefs, pp. 1–4. 2015.
 Daniel Holden, Oussama Kanoun, Maksym Perepichka, and Tiberiu Popa. Learned motion matching.
 ACMTransactions on Graphics (TOG), 39(4):53–1, 2020. ISSN 0730-0301. Publisher: ACM New
 York, NY, USA.
 Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as a
 foreign language. Advances in Neural Information Processing Systems, 36, 2024a.
 Nan Jiang, Zhiyuan Zhang, Hongjie Li, Xiaoxuan Ma, Zan Wang, Yixin Chen, Tengyu Liu, Yixin
 Zhu, and Siyuan Huang. Scaling up dynamic human-scene interaction modeling. arXiv preprint
 arXiv:2403.08629, 2024b.
 Korrawe Karunratanakul, Konpat Preechakul, Emre Aksan, Thabo Beeler, Supasorn Suwajanakorn,
 and Siyu Tang. Optimizing Diffusion Noise Can Serve As Universal Motion Priors, April 2024.
 Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization, January 2017.
 Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes, December 2022.
 Lucas Kovar, Michael Gleicher, and Frédéric Pighin. Motion graphs. In ACM SIGGRAPH 2008
 classes, pp. 1–10. 2008.
 Gen Li, Kaifeng Zhao, Siwei Zhang, Xiaozhong Lyu, Mihai Dusmanu, Yan Zhang, Marc Pollefeys,
 and Siyu Tang. EgoGen: An Egocentric Synthetic Data Generator. In IEEE Conference on
 Computer Vision and Pattern Recognition (CVPR), 2024.
 Jiaman Li, Alexander Clegg, Roozbeh Mottaghi, Jiajun Wu, Xavier Puig, and C Karen Liu. Control
lable human-object interaction synthesis. arXiv preprint arXiv:2312.03913, 2023a.
 Jiaman Li, Jiajun Wu, and C Karen Liu. Object motion guided human motion synthesis. ACM
 Transactions on Graphics (TOG), 42(6):1–11, 2023b.
 Han Liang, Wenqian Zhang, Wenxuan Li, Jingyi Yu, and Lan Xu. Intergen: Diffusion-based multi
human motion generation under complex interactions. International Journal of Computer Vision,
 pp. 1–21, 2024.
 Hung Yu Ling, Fabio Zinno, George Cheng, and Michiel van de Panne. Character Controllers Using
 Motion VAEs. ACM Transactions on Graphics, 39(4), August 2020. ISSN 0730-0301, 1557-7368.
 doi: 10.1145/3386569.3392422.
 Xinpeng Liu, Haowen Hou, Yanchao Yang, Yong-Lu Li, and Cewu Lu. Revisit human-scene
 interaction via space occupancy. arXiv preprint arXiv:2312.02700, 2023.
 Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL:
 A skinned multi-person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia), 34(6):
 248:1–248:16, October 2015.
 Zhengyi Luo, Jinkun Cao, Josh Merel, Alexander Winkler, Jing Huang, Kris Kitani, and Weipeng
 Xu. Universal humanoid motion representations for physics-based control. arXiv preprint
 arXiv:2310.04582, 2023.
 Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Black.
 AMASS: Archive of motion capture as surface shapes. In International Conference on Computer
 Vision, pp. 5442–5451, October 2019.
 12
Julieta Martinez, Michael J Black, and Javier Romero. On human motion prediction using recurrent
 neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition,
 pp. 2891–2900, 2017.
 Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios
 Tzionas, and Michael J. Black. Expressive body capture: 3D hands, face, and body from a single
 image. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp.
 10975–10985, 2019.
 Xue Bin Peng, Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. Ase: large-scale reusable
 adversarial skill embeddings for physically simulated characters. ACM Trans. Graph., 41(4),
 jul 2022. ISSN 0730-0301. doi: 10.1145/3528223.3530110. URL https://doi.org/10.
 1145/3528223.3530110.
 Mathis Petrovich, Michael J. Black, and Gül Varol. TEMOS: Generating diverse human motions
 from textual descriptions, July 2022.
 Abhinanda R. Punnakkal, Arjun Chandrasekaran, Nikos Athanasiou, Alejandra Quiros-Ramirez, and
 Michael J. Black. BABEL: Bodies, Action and Behavior with English Labels, June 2021.
 Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal,
 Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.
 Learning Transferable Visual Models From Natural Language Supervision, February 2021.
 Davis Rempe, Tolga Birdal, Aaron Hertzmann, Jimei Yang, Srinath Sridhar, and Leonidas J Guibas.
 Humor: 3d human motion model for robust pose estimation. In Proceedings of the IEEE/CVF
 international conference on computer vision, pp. 11488–11499, 2021.
 Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High
Resolution Image Synthesis with Latent Diffusion Models, April 2022.
 John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy
 optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
 Yoni Shafir, Guy Tevet, Roy Kapon, and Amit Haim Bermano. Human motion diffusion as a
 generative prior. In The Twelfth International Conference on Learning Representations, 2024.
 Mengyi Shan, Lu Dong, Yutao Han, Yuan Yao, Tao Liu, Ifeoma Nwogu, Guo-Jun Qi, and Mitch
 Hill. Towards open domain text-driven synthesis of multi-person motions. arXiv preprint
 arXiv:2405.18483, 2024.
 Yi Shi, Jingbo Wang, Xuekun Jiang, Bingkun Lin, Bo Dai, and Xue Bin Peng. Interactive character
 control with auto-regressive motion diffusion models. ACM Trans. Graph., 43(4), jul 2024. ISSN
 0730-0301.
 Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang, Chen Qian, Chen Change Loy, and
 Ziwei Liu. Bailando: 3d dance generation by actor-critic gpt with choreographic memory. In
 Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
 11050–11059, 2022.
 Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv
 preprint arXiv:2010.02502, 2020a.
 Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising Diffusion Implicit Models, October
 2022.
 Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben
 Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint
 arXiv:2011.13456, 2020b.
 Sebastian Starke, He Zhang, Taku Komura, and Jun Saito. Neural state machine for character-scene
 interactions. ACM Trans. Graph., 38(6):209–1, 2019.
 13
Sebastian Starke, Ian Mason, and Taku Komura. Deepphase: Periodic autoencoders for learning
 motion phase manifolds. ACM Transactions on Graphics (TOG), 41(4):1–13, 2022. ISSN 0730
0301. Publisher: ACM New York, NY, USA.
 Sebastian Starke, Paul Starke, Nicky He, Taku Komura, and Yuting Ye. Categorical codebook
 matching for embodied character controllers. ACM Transactions on Graphics (TOG), 43(4):1–14,
 2024.
 Lucas Stoffl, Andy Bonnetto, Stéphane d’Ascoli, and Alexander Mathis. Elucidating the hierarchical
 nature of behavior with masked autoencoders. In European conference on computer vision.
 Springer, 2024.
 Richard S Sutton and Andrew G Barto. Introduction to reinforcement learning, volume 135. MIT
 press Cambridge, 1998.
 Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano.
 Human motion diffusion model. In The Eleventh International Conference on Learning Represen
tations, 2023. URL https://openreview.net/forum?id=SJ1kSyO2jwu.
 Jonathan Tseng, Rodrigo Castellon, and C Karen Liu. Edge: Editable dance generation from music.
 arXiv preprint arXiv:2211.10658, 2022.
 Tom Van Wouwe, Seunghwan Lee, Antoine Falisse, Scott Delp, and C Karen Liu. Diffusionposer:
 Real-time humanmotionreconstruction from arbitrary sparse sensors using autoregressive diffusion.
 In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
 2513–2523, 2024.
 Zan Wang, Yixin Chen, Baoxiong Jia, Puhao Li, Jinlu Zhang, Jingze Zhang, Tengyu Liu, Yixin
 Zhu, Wei Liang, and Siyuan Huang. Move as you say, interact as you can: Language-guided
 human motion generation with scene affordance. In Proceedings of the IEEE/CVF Conference on
 Computer Vision and Pattern Recognition (CVPR), 2024.
 Yiming Xie, Varun Jampani, Lei Zhong, Deqing Sun, and Huaizu Jiang. Omnicontrol: Control
 any joint at any time for human motion generation. In The Twelfth International Conference on
 Learning Representations, 2024.
 Sirui Xu, Zhengyuan Li, Yu-Xiong Wang, and Liang-Yan Gui. Interdiff: Generating 3d human-object
 interactions with physics-informed diffusion. In Proceedings of the IEEE/CVF International
 Conference on Computer Vision, pp. 14928–14940, 2023.
 He Zhang, Sebastian Starke, Taku Komura, and Jun Saito. Mode-adaptive neural networks for
 quadruped motion control. ACM Transactions on Graphics, 37(4):1–11, August 2018. ISSN
 0730-0301, 1557-7368. doi: 10.1145/3197517.3201366.
 Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao,
 Hongtao Lu, and Xi Shen. T2m-gpt: Generating human motion from textual descriptions with
 discrete representations. arXiv preprint arXiv:2301.06052, 2023a.
 Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image
 diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision,
 pp. 3836–3847, 2023b.
 Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei
 Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint
 arXiv:2208.15001, 2022.
 YanZhangandSiyuTang. TheWanderingsofOdysseusin 3DScenes. In2022IEEE/CVFConference
 on Computer Vision and Pattern Recognition (CVPR), pp. 20449–20459, New Orleans, LA, USA,
 June 2022. IEEE. ISBN 978-1-66546-946-3. doi: 10.1109/CVPR52688.2022.01983. URL
 https://ieeexplore.ieee.org/document/9879049/.
 Yunbo Zhang, Deepak Gopinath, Yuting Ye, Jessica Hodgins, Greg Turk, and Jungdam Won. Simula
tion and retargeting of complex multi-character interactions. In ACM SIGGRAPH 2023 Conference
 Proceedings, pp. 1–11, 2023c.
 14
Kaifeng Zhao, Shaofei Wang, Yan Zhang, Thabo Beeler, and Siyu Tang. Compositional Human-Scene
 Interaction Synthesis with Semantic Control, July 2022.
 Kaifeng Zhao, Yan Zhang, Shaofei Wang, Thabo Beeler, and Siyu Tang. Synthesizing Diverse Human
 Motions in 3D Indoor Scenes, August 2023.
 Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the Continuity of Rotation
 Representations in Neural Networks, June 2020.
 Fabio Zinno. Ml tutorial day: From motion matching to motion synthesis, and all the hurdles in
 between. Proc. of GDC 2019, 2, 2019.
 15
A MOTIONPRIMITIVE REPRESENTATION
 Representation.
 We represent each frame of the motion primitive as a tuple of
 (t, R,θ,J,dt,dR,dJ), where t ∈ R3 denotes the global body translation, R ∈ R6 denotes the
 6D rotation representation (Zhou et al., 2020) of the global body orientation, θ ∈ R21×6 is the 6D
 representation of 21 joint rotations, J ∈ R22×3 denotes the 22 joints locations, dt ∈ R3 denotes the
 temporal difference with previous frame’s translation, dR ∈ R6 denotes the 6D representation of the
 relative rotation between current and previous frame’s body orientation, and dJ ∈ R22×3 denotes the
 temporal diffence between current and previous frame’s joint locations.
 Such overparameterized motion representation has multiple benefits. Firstly, the joint rotation
 components θ are compatible with animation pipelines, saving the time-consuming optimization
based skeleton-to-body conversion required by the commonly used HML3D (Guo et al., 2022)
 representation in text-to-motion methods. Moreover, including the joint location components J
 facilitates solving physical constraints like reducing foot skating and joint trajectory control. Our
 motion representation also models the first-order kinematics with the temporal difference features to
 improve the motion naturalness.
 Canonicalization. We represent motion primitives in a human-centric local coordinates frame to
 canonicalize the primitive features and facilitate model learning. Each motion primitive is canonical
ized in a local coordinates system centered at the first frame body. The origin is located at the pelvis
 of the first frame body, the X-axis is the horizontal projection of the vector pointing from the left hip
 to the right hip, and the Z-axis is pointing in the inverse gravity direction. Given the pelvis, left hip,
 and right hip joints, the origin is located at the pelvis joint, and the local axis system can be derived
 by:
 Algorithm 3 Motion primitive rotation canonicalization
 Input: right_hip ∈ R3, left_hip ∈ R3
 x_axis = right_hip- left_hip
 x_axis[2] = 0
 ▷ Project to the xy plane
 normalize(x_axis)
 z_axis = [0, 0, 1]
 y_axis = cross_porudct(z_axis, x_axis)
 normalize(y_axis)
 return [x_axis, y_axis, z_axis]
 ▷ Inverse gravity direction
 Westore the canonicalization transformations and apply their inverse transformation to the generated
 motion primitives to recover global motions in the world coordinates.
 B DATASETS
 We train separate DART models on motion-text data from the BABEL (Punnakkal et al., 2021)
 and HML3D (Guo et al., 2022) dataset. Both BABEL and HML3D use motion sources from the
 AMASS(Mahmood et al., 2019) dataset. Their main difference is that BABEL features fine-grained
 frame-level text annotation while HML3D uses coarse sequence-level annotation.
 The BABELdataset contains motion capture sequences with frame-aligned text labels that annotate
 the fine-grained semantics of actions. Fine-grained text labels in BABEL allow models to learn
 precise human action controls and natural transitions among actions. We use the motion data at a
 framerate of 30 frames per second the same as prior works (Barquero et al., 2024; Athanasiou et al.,
 2022). During training, motion primitives are randomly sampled from data sequences and the text
 label is randomly sampled from all the action segments that overlap with the primitive. To alleviate
 the action imbalance in the BABEL dataset, we use the provided action labels to perform importance
 sampling during training so that the motion data of each action has roughly equal sampling chances
 despite their varying frequency in the original dataset.
 The HML3Ddataset contains short motions with coarse sequence-level sentence descriptions. We
 only use a subset of HML3D in training since its subset HumanAct12 and left-right mirroring motion
 sequences only provide joint locations instead of SMPL (Loper et al., 2015; Pavlakos et al., 2019)
 16
body sequences that our motion representation requires. The joint rotations used in the original
 HML3D are calculated using naive inverse kinematics and can not be directly used to animate
 human motions. We therefore only train using the subset where the SMPL body motion sequences is
 available. The HML3D motions are sampled at a framerate of 20 frames per second. During training,
 we randomly sample primitives with uniform probability and the text label is randomly chosen from
 one of the multiple sentence captions of the overlapping sequence.
 C MOTIONPRIMITIVE VAE
 Architecture. Our Motion Primitive VAE employs the transformer-based architecture. Both the
 encoder and decoder consist of 7 transformer encoder layers with skip connections (Chen et al.,
 2023). The transformer layers use the dropout rate of 0.1, feed-forward dimension of 1024, hidden
 dimension of 256, 4 attention heads, and the gelu activation function. The latent space dimension is
 256. After finishing training, we follow (Rombach et al., 2022) to calculate the variance of the latent
 variables using a data batch. When training the latent denoiser model, the raw latent output from the
 encoder is scaled to have a unit standard deviation.
 Losses. The motion primitive VAE is trained with the following losses:
 LVAE =Lrec +wKL×LKL+waux×Laux
 Lrec = F(ˆ X,X)
 (5)
 (6)
 The reconstruction loss Lrec aims to minimize the distance between the reconstructed future frames
 ˆ
 Xandthe ground truth future frames X, where F(·,·) denotes the distance function and we use the
 smoothed L1 loss(Girshick, 2015) in implementation.
 LKL =KL(q(z|H)||N(0,I)),
 (7)
 LKL penalizes the distribution difference between the predicted distribution and a standard Gaussian,
 where KL(·,·) denotes the Kullback-Leibler divergence (KL), and q(z|H) denotes the predicted
 distribution from the encoder E. We use a small KL term of 1e−6 following Rombach et al. (2022)
 as we aim to keep the latent space expressive and only use the small KL loss to avoid arbitrarily
 high-variance latent spaces.
 Laux = F(¯ dt, ˆ dt) + F( ¯
 dJ, ˆ dJ) + F( ¯
 dR, ˆ dR)
 (8)
 The auxiliary loss Laux regularizes the predicted temporal difference features ˆ d· of translations,
 global orientation, and joints to be close to the actual temporal differences ¯
 d· calculated from the
 predicted motions. We use wKL = 1e−6 and waux = 100 in our experiments. We train the motion
 primitive VAE with the AdamW (Kingma & Ba, 2017) optimizer and the learning rate is set to 1e−4
 with linear annealing.
 We conduct ablation studies on the impacts of the losses. We evaluate motion primitive VAEs
 trained with different loss weights on autoregressive motion sequence reconstruction error and motion
 jittering in reconstructed motions. Using large KL loss weight of wKL = 1, reduces the model
 expressiveness (the reconstruction error increases from 0.08 to 0.44 compared to wKL = 1e−6 on the
 test motions) and fails to accurately reconstruct complex sequences like cartwheeling. Using a small
 KLloss weight of wKL = 1e−6 can maintain the expressiveness of the learned VAE while allowing
 the latent distribution to deviate a bit more from a standard Gaussian. Applying the auxiliary losses
 with waux = 100 helps to reduce the jittering in the reconstructions and improve the motion quality
 compared to using waux = 0, as reflected by a smaller jerk metric of 2.45 when using waux = 100
 compared to a jerk of 3.67 when using waux = 0.
 17
D LATENT DENOISER MODEL
 D.1 LOSSES
 Wetrain the latent denoiser model using DDPM (Ho et al., 2020) with 10 diffusion steps and use a
 cosine noise scheduler. The denoiser model is trained with the following losses:
 Ldenoiser = Lsimple + wrec × Lrec + waux ×Laux
 Lsimple = F(G(zt,t,H,c),z0),
 (9)
 (10)
 where F(·,·) is a distance function and we use the smooth L1 loss (Girshick, 2015) in our implemen
tation. We train the denoiser to predict the clean latent variable with the simple objective Lsimple, and
 apply the feature reconstruction loss Lrec and auxiliary losses Laux on the decoded motion primitive
 ˆ
 X=D(G(zt,t,H,c)) to ensure the decoded motion primitives are valid.
 D.2 SCHEDULED TRAINING
 Weuse scheduled training to improve the stability of long sequence generation and the text prompt
 controllability. Long-term prediction stability is a significant challenge in autoregressive generation
 since the sample distribution can drift and accumulate during autoregressive generation. When the
 sample drifts out of the distribution covered by the learned model, the generation results can go wild.
 Our latent motion primitive model also faces the long-term stability challenge as an autoregressive
 method.
 To alleviate the out-of-distribution problems, we use the scheduled training (Ling et al., 2020; Bengio
 et al., 2015; Rempe et al., 2021; Martinez et al., 2017) to progressively introduce the test-time
 distributions during training. Specifically, we train the latent denoiser model on sequences of N
 consecutive motion primitives and use the prediction result of the previous motion primitive instead of
 the ground truth dataset to extract the history motion input H. We name such history motion extracted
 from the predicted last primitive as rollout history. Using the rollout history instead of the ground
 truth history introduces the test-time distribution which can differ from the dataset distribution, e.g.,
 unseen human poses or out-of-distribution combinations of human bodies and text labels. Exposing
 the model to such test-time distribution at training can improve the long-term generation stability and
 increase the text controllability when facing novel combinations of history motion and text prompts
 at generation time.
 The scheduled training has three stages to progressively introduce the rollout history. The first stage
 is fully supervised training where only the ground truth history is used during training. The second
 scheduled learning stage randomly replaces the ground truth history motion with rollout history
 motion by a probability linearly increasing from 0 to 1. The third stage of rollout training always
 uses the rollout history instead of the ground truth history. The scheduled training algorithm for the
 latent denoising model is shown in Alg. 5.
 We use wrec = 1 and waux = 10000 in our experiments. The denoiser model is trained using an
 AdamW optimizer. The learning rate is set to 1e−4 with linear annealing. Our denoiser model is
 trained with scheduled training (Ling et al., 2020; Bengio et al., 2015), consisting of a fully supervised
 stage of 100K iterations, a scheduled stage of 100K iterations, and a rollout stage of 100K iterations.
 Weset the maximum number of rollouts as 4. With the scheduled training, our latent motion primitive
 model can stably generate long motion sequences and better respond to the text prompt control even
 at poses that are not paired with the text prompt in the dataset.
 E TEXT-CONDITIONED TEMPORAL MOTION COMPOSITION
 E.1 EXPERIMENT DETAILS
 Weapply DART to conduct online motion generation using the rollout algorithm 1 with a default
 seed motion Hseed of rest standing and using a classifier-free guidance of w = 5. We use the
 released checkpoints of FlowMDM (Barquero et al., 2024), TEACH (Athanasiou et al., 2022),
 18
Algorithm 4 Calculate rollout probability
 1: Input: current iteration number iter, number of train iterations in the first supervised stage I1,
 number of train iterations in the second scheduled stage I2
 2: Output: rollout probability p
 3: function ROLLOUT_PROBABILITY(iter, I1, I2)
 4:
 if iter ≤ I1 then
 5:
 6:
 7:
 8:
 9:
 10:
 11:
 p ←0
 else if iter > I1 + I2 then
 p ←1
 else
 p ← iter−I1
 end if
 return p
 12: end function
 I2
 ▷ no rollout in the first supervised stage
 ▷ the third rollout stage always use rollout
 ▷ linearly scheduled rollout probability in the second scheduled stage
 Algorithm 5 Scheduled training for the latent denoising model
 1: Input: pretrained motion primitive decoder D and encoder E, latent variable denoiser GΘ
 parameterized by Θ, total diffusion steps T, optimizer O, loss criterion L, train dataset X.
 2: Scheduled training parameters: the number of train iterations in the first supervised stage I1,
 the number of train iterations in the second scheduled stage I2, the number of train iterations in
 the third rollout stage I3, the maximum number of primitive rollouts N during training.
 3:
 4: Itotal ← I1 +I2 +I3
 5: iter ← 0
 6: while iter < Itotal do
 7:
 [Hseed,X1,c1,...,XN,cN] ∼ X
 8:
 9:
 10:
 11:
 12:
 13:
 14:
 15:
 16:
 17:
 ▷ sample N consecutive motion primitives with text labels from dataset X
 H←Hseed
 for i ← 1to N do
 zi
 0 = E(H,Xi)
 t ∼U[0,T)
 ˆ
 zi
 t ← FORWARD_DIFFUSION(zi
 0,t)
 zi
 0 = GΘ(zi
 t,t,H,ci)
 ˆ
 X=D(H,ˆ zi
 0)
 ▷ initialize motion history
 ▷ number of rollouts
 ▷ compress motion primitive into latent space
 ▷ sample diffusion step t
 ▷ latent denoising model prediction
 ▷ decode predicted latent variable to future motion frames
 ∇←∇ΘL(zi
 0,ˆ zi
 0,H,Xi, ˆ Xi)
 ▷ model parameter gradient calculation
 Θ←O(Θ,∇)
 18:
 19:
 20:
 21:
 p ←ROLLOUT_PROBABILITY(iter,I1,I2)
 ▷ model update using optimizer
 ▷ update history motion using predicted or GT motion by a scheduled probability
 if rand() < p then
 22:
 23:
 24:
 25:
 26:
 27:
 28:
 29:
 30:
 31:
 ▷ use predicted rollout history
 zi
 T ← FORWARD_DIFFUSION(zi
 0,T) ▷ maximum noising simulating inference time
 ˆ
 ˆ
 zi
 0 = DDPM_SAMPLE(GΘ,zi
 T,T,H,ci)
 X=sg(D(H,ˆ zi
 0))
 H←CANONICALIZE(ˆ XF−H+1:F)
 else
 H←CANONICALIZE(XF−H+1:F)
 end if
 iter ← iter +1
 end for
 32: end while
 ▷ full DDPM denoising loop
 ▷ decode predicted latent variable and stop gradient
 ▷ use ground truth history
 and DoubleTake (Shafir et al., 2024) for baseline comparison. We adjust the handshake size and
 blending length of DoubleTake to be compatible with the shortest segment length of 15 frames. The
 history-conditioned modification of T2M-GPT is retrained on the BABEL dataset using the original
 19
hyperparameters. At generation time, the last frames of the previous action segment are encoded into
 tokens and provided as the first tokens when generating the next action segment to provide history
 conditioning of the previous action.
 Weextract the timeline of action segments for evaluation from the BABEL (Punnakkal et al., 2021)
 valid set. For each sequence in the validation set, we sort the original frame labels provided by
 BABEL to obtain a list of tuples of text prompts and durations. We randomly sample one text
 label when a segment is annotated by multiple texts. We skip the ‘transition’ text labels due to the
 ambiguous semantics and clamp the duration length with a minimum of 15 frames.
 E.2 PERCEPTUAL USER STUDIES.
 We conduct user studies to quantitatively compare our method DART with baselines to provide a
 more comprehensive and convincing evaluation. We run binary user studies on Amazon Mechanical
 Turk(AMT) to evaluate generation realism and text-motion semantic alignment. Participants are
 given generation results from two different methods and are asked to select the generation that is
 perceptually more realistic or better aligns with the action text stream in subtitles, as illustrated in
 Fig. 4. During the realism evaluation, action descriptions were not displayed to eliminate distractions.
 Wesample 256 sequences of action texts and durations from the BABEL dataset and use each method
 to generate motions given the action timelines. Participants are shown random pairs of results from
 our method and a baseline, and they are asked to choose the better generation. Each comparison is
 voted by 3 independent participants, with the video pairs randomly shuffled to ensure that participants
 do not know the method source.
 Figure 4: Illustration of the binary user study interface for evaluating motion-text semantic alignment
 (top) and perceptual realism(bottom). Participants are requested to select the generation that is
 perceptually more realistic or better aligns with the action descriptions in subtitles (only visible in
 semantic user study).
 E.3 ABLATION STUDIES.
 Weconduct ablative studies on architecture designs, diffusion steps, primitive representation, and
 scheduled training on the text-conditioned motion composition task.
 Removing the variational autoencoder (DART-VAE) and training diffusion model in the raw motion
 space results in a significantly higher jittering as reflected by the peak jerk (PJ) and Area Under Jerk
 20
(AUJ) metrics. This validates the effectiveness of integrating the variational encoder to compress the
 high-frequency noise in motion data and improve motion generation quality.
 Without the scheduled training (DART-schedule), the model can not effectively respond to text
 control and have significantly worse R-Prec and FID metrics. This is because, without scheduled
 training, the model can easily encounter out-of-distribution combinations of history motion and text
 prompts during autoregressive generation.
 Wealso include the ablation study of training a model to predict the single next frame conditioned
 on the current frame (per frame) as used in Shi et al. (2024). This is a special case of primitive
 with history length H = 1 and future length F = 1. The ablative model has significantly worse
 R-Prec and FID metrics and cannot respond to text prompts. This indicates using frame-by-frame
 prediction is less effective than primitives with a reasonable horizon (H = 2, F = 8) in learning
 text-conditioned motion space.
 DART can learn high-quality text-conditioned motion primitive models using very few diffusion
 steps because of the simplicity of the primitive representation. We conduct ablative studies of training
 DARTusing different numbers of DDPM diffusion steps. Reducing the diffusion steps from 100 to
 fewer than 10 does not significantly affect performance. However, an extremely low diffusion step
 number of 2 leads to a much higher FID, indicating poorer motion quality.
 Table 5: Ablation studies results on text-conditioned temporal motion composition. The first row
 includes the metrics of the dataset for reference. Symbol ‘→’ denotes that closer to the dataset
 reference is better and ‘±’ indicates the 95% confidence interval.
 Segment Transition
 FID↓
 R-prec↑
 DIV → MM-Dist↓ FID↓ DIV→ PJ→ AUJ↓
 Dataset 0.00±0.00 0.72±0.00 8.42±0.15 3.36±0.00 0.00±0.00 6.20±0.06 0.02±0.00 0.00±0.00
 Ours 3.79±0.06 0.62±0.01 8.05±0.10 5.27±0.01 1.86±0.05 6.70±0.03 0.06±0.00 0.21±0.00
 DART-VAE 4.23±0.02 0.62±0.01 8.33±0.12 5.29±0.01 1.79±0.02 6.73±0.23 0.20±0.00 0.96±0.00
 DART-schedule 8.08±0.09 0.39±0.01 8.05±0.12 6.96±0.03 7.41±0.10 6.58±0.06 0.03±0.00 0.18±0.00
 per frame(H=1,F=1) 10.31±0.09 0.29±0.01 6.82±0.13 7.41±0.01 7.82±0.09 6.03±0.08 0.02±0.00 0.08±0.00
 steps 2 4.44±0.04 0.60±0.00 8.20±0.15 5.38±0.01 2.24±0.02 6.77±0.07 0.05±0.00 0.20±0.00
 steps 5 3.49±0.09 0.63±0.00 8.25±0.15 5.18±0.01 2.11±0.07 6.74±0.11 0.05±0.00 0.20±0.00
 steps 8 3.70±0.03 0.62±0.01 8.04±0.13 5.25±0.03 2.15±0.08 6.72±0.15 0.06±0.00 0.20±0.00
 steps 10 (Ours) 3.79±0.06 0.62±0.01 8.05±0.10 5.27±0.01 1.86±0.05 6.70±0.03 0.06±0.00 0.21±0.00
 steps 50 3.82±0.05 0.60±0.00 7.74±0.07 5.30±0.01 2.11±0.10 6.58±0.10 0.06±0.00 0.22±0.00
 steps 100 4.16±0.06 0.61±0.00 7.82±0.15 5.32±0.02 2.20±0.05 6.43±0.10 0.06±0.00 0.21±0.00
 F LATENT DIFFUSION NOISE OPTIMIZATION-BASED CONTROL
 Optimization details. We use the Adam (Kingma & Ba, 2017) optimizer with a learning rate of 0.05
 to optimize latent noises ZT for 100 to 300 steps. The learning rate is linearly annealed to 0 and the
 gradient is normalized to stabilize optimization. The optimization running time is dependent on both
 the motion sequence length and the number of optimization steps. An example experiment of a target
 motion sequence of 60 frames and 100 optimization steps costs around 74 seconds.
 Human-scene interaction. In human-scene interaction synthesis, we use two scene constraints to
 avoid human-scene collision and foot-floating artifacts. To reduce human-scene interpenetration,
 cons_coll(·) penalizes joints colliding into scenes with negative SDF value and can be formulated as:
 cons_coll(M) = −
 (Ψ(J(M)) −J(M)thresh)−
 J
 (11)
 where Ψ(·) is the signed distance function returning the signed distance from the query point to
 the closest scene point. J(·) extracts the joint locations from the motion M. (·)thresh is the joint
dependent contact threshold value, which is determined by the joint-skin distance in the rest pose.
 (·)− denotes clipping positive values.
 To reduce the occurrence of foot-floating artifacts, cons_cont(·) encourages the foot to be in contact
 with the scene and is defined as:
 cons_cont(M) =
 (Ψ(F(M))−F(M)thresh)+
 F
 where F(·) extracts the lower foot joint location. (·)+ denotes clipping negative values.
 (12)
 21
G REINFORCEMENT LEARNING-BASED CONTROL
 Architecture. Both the actor and critic networks are 4-layer MLPs with residual connections and a
 hidden dimension of 512. We apply the tanh scaling: x = 4 · tanh(x) (Ling et al., 2020) to the actor
 output to clip the action prediction in the range of [−4,4], avoiding unbounded action predictions.
 The actor networks are initialized with close to zero weights to boost policy training following
 (Andrychowicz et al., 2020). The observation input contains a 512D text embedding, a 552D of
 history motion, a 1D observation of floor height relative to the human pelvis, a 1D observation of the
 f
 loor plane distance from the pelvis to the goal location, and a 3D unit vector of the goal direction
 in the human-centric coordinates frames. We clamp the goal distance observation with a maximum
 value of 5m and the goal direction in the range of a 120-degree field of view to simulate egocentric
 human perception.
 Rewards. The text-conditioned goal-reaching control policies are trained to maximize the discounted
 expectation of the following rewards (Zhang & Tang, 2022; Li et al., 2024; Zhao et al., 2023):
 Weuse three distance-related rewards to encourage the human agent to minimize its distance to the
 goal location.
 rdist = Di −Di−1
 (13)
 The distance reward rdist encourages the human to get closer to the goal location, where Di is the
 2D distance between the human pelvis and the goal location at step i.
 rsucc = 1 ifDi <0.3
 (14)
 0 otherwise
 The success reward succ gives a sparse but strong reward when the human arrives at the goal, where
 0.3m is the success threshold value.
 rori = ⟨pi −pi−1,g −pi−1⟩+1
 2
 ,
 (15)
 The moving orientation reward encourages the moving orientation to be aligned with the goal
 orientation, where pi is the human pelvis location at step i, and g is the goal location.
 Moreover, we apply scene constraints-related rewards to discourage unnatural behaviors such as foot
 skating and collision with the floor.
 rskate = −disp · (2 −2h/0.03),
 (16)
 The skate reward rskate penalizes foot displacements when in contact with the floor, where disp is
 the foot displacement in two consecutive frames, h denotes the higher foot height in two consecutive
 frames and 0.03m is the threshold value for contact.
 rfloor = −(|lf| − 0.03)+,
 (17)
 The floor contact reward rfloor penalizes when the lower foot distance to the floor is above the
 threshold of 0.03m, where lf denotes the height of the lower foot, | · | denotes the absolute value
 operator, and (·)+ is the clipping operator with a minimum of 0.
 The rewards are weighted with wdist = 1, wsucc = 1, wori = 0.1, wfloor = 100, wskate = 10 for
 the hopping controller and 100 for the others.
 H COMPUTING RESOURCES
 Our experiments and performance profiling are conducted on a workstation with single RTX 4090
 GPU, intel i7-13700K CPU, 64GiB memory. The workstation runs with Ubuntu 22.04.