CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character control
Authors: Guy Tevet
Sigal Raab
Setareh Cohan
Daniele Reda
Zhengyi Luo
Xue Bin Peng
Amit Haim Bermano
Michiel van de Panne
Venue: ICLR 2025 Spotlight (2025)
Keywords: RL
PPO
motion
motion generation
motion synthesis
synthesis
generative models
diffusion
animation
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Motion diffusion models and Reinforcement Learning (RL) based control for physics-based simulations have complementary strengths for human motion generation. The former is capable of generating a wide variety of motions, adhering to intuitive control such as text, while the latter offers physically plausible motion and direct interaction with the environment. In this work, we present a method that combines their respective strengths. CLoSD is a text-driven RL physics-based controller, guided by diffusion generation for various tasks. Our key insight is that motion diffusion can serve as an on-the-fly universal planner for a robust RL controller. To this end, CLoSD maintains a closed-loop interaction between two modules — a Diffusion Planner (DiP), and a tracking controller. DiP is a fast-responding autoregressive diffusion model, controlled by textual prompts and target locations, and the controller is a simple and robust motion imitator that continuously receives motion plans from DiP and provides feedback from the environment. CLoSD is capable of seamlessly performing a sequence of different tasks, including navigation to a goal location, striking an object with a hand or foot as specified in a text prompt, sitting down, and getting up.

Published as a conference paper at ICLR 2025
CLOSD: CLOSING THE LoOP BETWEEN SIMULATION AND DIFFUSION FOR MULTI-TASK CHARACTER CONTROL
Guy Tevet Sigal Raab', Setarch Cohan", Daniele Reda Zhengyi Luo', Xue Bin Pengo. Amit H. Bermano', and Michiel van de Panne "Tel-Aviv University, University of British Columbia, Wayve, *Carnegie Mellon University, SSmon Fraser University, NVIDIA guytevet wtt.emaaalaaa.....mmmmmmm
ABSTRACT
Motion diffusion models and Reinforcement Learning (RL) based control for physics-based simulations have complementary strengths for human motion gen- eration. The former is capable of generating a wide variety of motions, adhering intuitive control such text, while the latter offers physically plausible motion and direct interaction with the environment. In this work, we present a method that combines their respective strengths. CLoSD is text-driven RL physics- based controller, guided by diffusion generation for various tasks. Our key in- sight that motion diffusion can serve as an on-the-fly universal planner for a robust RL controller. To this end, CLoSD maintains a closed-loop interaction be- tween two modules - a Diffusion Planner (DiP), and a tracking controller. DiP is fast-responding autoregressive diffusion model, controlled by textual prompts and target locations, and the controller is a simple and robust motion imitator that continuously receives motion plans from DiP and provides feedback from the en- vironment. CLoSD is capable of seamlessly performing a sequence different tasks, including navigation foot as specified in a text prompt, sitting down, and getting up.
goal location, striking object with hand
stttps:/quyttveet. wtt.ihub..o/ooo...... woccoosD-ccooooop..........
INTRODUCTION
Physics-based character animation leverages simulations to generate plausible human movement, particularly terms of realistic contacts, and interactions with objects and other characters scene. To achieve set of given goals or task, control policy is often synthesized using Rein- forcement Learning (RL). As shown in recent years (Peng et al. 2018 202T), RL greatly benefits from leaning kinematic motion capture data as reference in order to learn to produce human- like behaviors. Such methods typically rely on individual clips or small-scale curated datasets to train controllers for individual tasks, this simplifies the RL exploration problem and allows for compact models of the underlying desired motion manifold. However, it is often difficult to scale RL methods leverage larger human motion datasets, such those commonly used to train kinematic motion generation models (Tevet et al. 2023 Dabral et al. 2023
In this paper. we present Closing the Loop between Simulation and Diffusion, dubbed CLoSD, method that allows for closed-loop operation of text-to-motion diffusion models with physics-based simulations. Our key insight is that motion diffusion, given textual instruction and target location, can serve as versatile kinematic motion planner. A thysicc-based motion tracking controller then serves as the executor of the plan in simulation and facilitates physically plausible interactions with the environment. However, motion diffusion models are typically used for offline motion genera- tion, with slow inference times due to the gradual denoising process. Instead, we require a model that reacts in real-time and whose movements Diffusion Planner (DiP), an auto-regressive real-time diffusion model, generating high-fidelity mo- tions using few 10 diffusion steps, while being conditioned both text and target location for
grounded in the scene. To this end, introduce

---

Published as a conference paper at ICLR 2025
"Jump to the (target]"
"Sit on the [sofa]"
"Punch the [earget]]
"Run [target]]
"Sit relaxed the [sofa]"
"Strike [[target] kicking" "
Figure 1: CLoSD is multi-task physics-baset RL controller, capable of performing object inter- action tasks and can be controlled through a text prompt and target positions. The engine CLoSD is the Diffusion Planner (DiP), a real-time diffusion model that operates in a closed feedback loop with the RL, producing high-fidelity human motions for completing diverse set of tasks.
arbitrary joints (e.g. "Strike with hook punch" + left hand target or "perform high-kick" right foot target).
We use a robust RL tracking controller to execute the DiP plan. As demonstrated in PHC (Luo ct al. 2023), a general motion tracking policy can enable physically simulated characters to follow wide spectrum of human motions. The use of physics simulation can also correct for nonphysical artifacts in the input target motion, such as floating, sliding, and penetration with objects. We syner- gize our diffusion-based planner and the motion tracker by forming closed feedback loop between them where CLoSD enables simulated characters to produce robust and responsive environment- aware behaviors. The executed motion of the simulated character via the physics simulation is fed back to the diffusion model auto-regressively and becomes the prefix of the next sequence. Then, given the text prompt and a task-related goal, e.g., the target location strike, DiP generates the next motion plan for the policy, closing the planning-execution loop. Unlike typical offline motion diffusion, the closed feedback loop CLoSD enables interactive text-driven control, where users can change text commands on the fly and our system is able to seamlessly transition between the desired behaviors.
Nonetheless, mismatch occurs as the original motion tracker was trained neither considering object interaction, nor interactions with a real-time planner. We therefore fine-tune the tracking RL policy with DiP in-the-loop. This process is universal and is performed for multiple tasks at once. The fine- tuned tracker is then robust to the closed-loop state sequences as observed for the object interactions.
We show that single instance of CLoSD successfully performs goal-reaching, target-striking, and couch interactions, each of which required a dedicated policy and considerable reward engineering in prior work (Peng al. 2022 Hassan et al. 2023). The results of our experiments demonstrate that CLoSD outpertorms both the current leading text-to-motie controller and the state-of- off-the-am art multi-task controller. Our diffusion planner generates 40-frame plans 3,500 fps, i.e., 175x real- time, allowing ample scope for real-time use even with future additions.
RELATED WORK
Recent advances kinematic and physics-base methods for human motion generation enable a variety of text-to-motion, motion imitation, and other task specification methods.
2

---

Published as a conference paper at ICLR 2025
Data-driven motion synthesis. Advances in language and generative models have led to the emer- gence of flexible and intuitive text-to-motion models for computer animation. The language and semantic understanding encapsulated in text encoders (Radford et al. [021 Devlin 2018). together with the multi-modal generative capabilities of diffusion models (Sohl-Dickstein et al. 2015 Song & Ermon 2020) have given rise to new generation of high-fidelity generative models, particularly
for images and videos (Rombach et al. [022,] Ho et al. 2022). This approach has been successfully
adapted to the motion synthesis domain (Tevet et al., 2022; 2023 Dabral et al. [023 Zhu et al.
2023). providing powerful tools for large variety of motion synthesis and editing tasks (Shafi
al. 2024 Tsngg et al. 2023: Cohan et al., 2024, Goel et al., 2024, Karunratanakul et al. (024).
Recently, MoMo (Raab et 20044). MAS KKaon et al. 2024) and SinMDM (Raab et al. 2024b)
show generalization to out-of- motions using various difrusion methods. A-MDM (Shi et al. 2024) and CAMDM (Chen et al., 2024) presented auto-regressive motion diffusion models and demonstrated interactive control tasks. Xic et al. (2023) used a diffusion model learned on limited data as reference for box loco- doco-mamipulation task. Trace and Pace (Rempe et al. 2023 used diffusion model to plan character trajectories on a 2D map for character navigation.
Diffusion models for object interaction. Works in both Human-Object Interaction (HOI) and Human- Interaction (HSI) present interaction with objects, e.g., sitting on a couch. However, HOT focuses on interaction with individual objects, while HSI addresses interaction with the entire scene. While CLoSD offers both textural control and a physics-based control, current diffusion- based HOI and HSI works typically allow either textual input (Wu et al. 2024 Peng et al. [023] Wang et al.) 2024 Yi et al. [024] Cen et al., 2024 Li & Dai 2024) or physical-based methods
(Huang et al. 2023). or neither Kulkarni al. (024))
Text controlled Simulated Characters. While data-driven motion synthesis models are able to generate life-like motions, they are still prone to exhibiting artifacts such as foot sliding and float- ing. Further, creating realistic human-object interactions using purely data-driven approaches is challenging, as the underlying contact dynamics and collisions are difficult to learn directly from
data. Using physics-simulated characters (Juravsky et al.] 2022 2024, Yao et al.] 2024 Truong
ct al. 2024 Xiao et al. 2024; Ren et al. 2023) provides a solution by enforcing physically plausible constraints motion and interaction. Among them, the PADL line of work uses natural language to control simulated characters, starting from simple instructions (Juravsky et al., 2022) to scale up to more complex instructions (Juravsky et al. 2024). MoConVQ (Yao et al. 2024) utilizes an LLM to direct characters using a pretrained motion latent space and in-context learning but neither approach supports fine-grains fine-grained human-object interactions. UniHSI (Xiao et al.) 2024) uses LLMs to specify target positions for controlling a simulated humanoid in human-object interaction tasks. However, as only the target end-effector position is used as the control interface, the style of the motion cannot be influenced by language. PDP (Truong et al. (2024) learns from an offline humanoid motion imitation dataset utilizing a diffusion policy (Chi et al. 2023) and demonstrates an early attempt at learning language- based control from offline data. Antoctor (Ren et al. (202)) utilizes differentiable physics simulator and diffusion model to achieve text-guided humanoid control. Compared to InsActor, we enable human-object interactions and alleviate artifacts introduced by the differentiable simulator. In contrast, we propose to bridge the gap between data-driven kinematic text-to-motion generation and paysics-based character control using closed-loop plan-and and-initatee system.
Humanoid Motion Imitation. Simulated humanoid motion imitation has seen significant advance- ments in recent years (Al Borno et al. 2012 Peng et al. 2018 2019 Chentanez et al. 2018
Won et al. 2020 Yuan & Kitani 2020 Fussell et al.] 2021: Luo et al.; 2023 2021] 2022
Wang
et al.] 2020). Since no ground-truth action annotations exist and physics simulators are often non- differentiable, policies, imitators, or controllers are trained to track, imitate, or mimic human motion using deep reinforcement learning (RL). From policies that can track a single clip of motion Peng et al. (2018). to large-scale datasets Won et al. (2020); Luo et al. (2023). motion imitators become more and more useful as they can reproduce an ever-growing corpus of behaviors. In this work, we utilize PHC Luo et al. (2023). an off-the-shelff motion imitator that has successfully scaled to the AMASS dataset Mahmood et al. (2019), as the foundation for our approach due its robustness in handling diverse kinematic motions. Recent works leveraged imitation policies to learn multi- purpose controllers; PULSE (Luo et al. (2024) distilled it to a latent action space for down-stream tasks, whereas MaskedMimic/Tessler et al. 2024) learned multi-modal controller based on partial information imitation.
3

---

Published as a conference paper at ICLR 2025
Closed
a: prefiz
Loop
Pelvis @ [Target]
x
prefir fir
x
pred
Diffusion step
Transformer Decoder
Pelvis [Target]
DiP
"A person is sitting down on the sofa"
Next
DistilBERT
Cross-Attention
Motion plan
concat
x: prefix
z Phed
"A person is sitting down on the sofa"
layers
action
ed
Physics Simulator
Tracking Controller
next state
x
next prefix
Figure 2: CLoSD Overview, (Left) DiP is rapid auto-regressive diffusion model conditioned on text prompt and a Target location. It generates the motion plan xred which is then performed by the RL Tracking Controller. The result frames are fed back to DiP as s Together they maintain executixe loop. (Top right) A zoom-in to single DiP denoising iteration.
PRELIMINARIES
Motion representation. We use two different motion representations, each suitable for its purpose. For the kinematic motions generated by the diffusion model, x, we follow MDM (Tevet et al. 2023) and the ML33 motion representation, in which each frame is represented relatively to the previous one. For the RL tracking policy, we follow PHC (Luo et al. (023) and represent the motion, sim (i.e. sequence of states) by their global position and velocity, at the world axis. To convert from HumanML3D to PHC we implement x æ-m = R2G(x), (stands for relative-to-llobal using simple accumulation. convert from PHC HumanML.3D we use x G2R (stands for global-to- to-llaii-- as implemented for HumanML3D. The rotations are calculated using first- order inverse kinematics and the foot contacts using a height heuristic. We simulate a humanoid robot compatible with SMPL (Loper al. 2015) joints, hence, converting the simulated results back to the SMPL mesh is straightforward (Figure
Denoising diffusion models. The DDPM (Ho et al. 2020) framework used in this paper models diffusion as a Markov noising process, {2,]==o- where To is drawn from the data distribution and ( ( (V((oorot small enough, we can approximate 2r ~ NO, I). In our context, conditioned motion synthesis models the distribution p(zplc) as the reversed diffusion process of gradually cleaning ZT. Instead of predicting é, as formulated by Ho et al. (2020), we follow MDM (Tevet et al.) (023) and use an equivalent formulation to predict the motion sequence itself, i.e., do with the simple objective (Ho et al. 2020). Siiple = Ezz (-)pp(())).ee-(...)) [| 2. Predicting 2o is found to produce better results motion data, and enables to apply a target loss as geometric loss at each denoising step.
T)) where O₁ (0, 1) are constant hyper-parameters. When oz
4 CLOSD
CLoSD is a text-driven character controller capable of performing multiple tasks involving physically-baser object interactions. An overview is given in Figure The input to CLoSD consists of (a) a text prompt describing the desired motion, including its style, content, etc., and (b) target location for one joint of the character. The combination of these two inputs provides a simple but expressive description for object interaction tasks. For example, the text prompt "strike the target using a high kick" combined with the target location for the foot at the target location, precisely defines how to strike it (Figure while the prompt "strike the target by pushing it" combined with
4

---

Published as a conference paper at ICLR 2025
the same target location, this time for one of the hand joints, defines completely different strike. Note that both inputs can be changed online to enable interactive control by the user and can be sequenced to perform series of different tasks.
CLoSD has two key components, Diffusion Planner (DiP. $4.1D. a real-time diffusion model that synthesizes kinematic motion plan, and robust motion tracking controller ($4.2) that then executes the desired motion. These then operate in a closed-loop fashion ($4.)) by providing the realized motion as an input prefix to DiP, which enables DiP to react to the physical environment. A simple state machine ($4.4) detects task completions and transitions to the next task by changing the text
prompt and the target on the fly. This enables the smooth completion of a sequence of multiple tasks (see the web page video).
4.1 MOTION DIFFUSION PLANNER
To guide CLoSD to perform multiple tasks in a physically simulated environment. we use a kine- matic motion generator as a planner. Motion diffusion models such as MDM (Tevet et al. 2023). are a leading method for motion synthesis but are not suitable for several reasons. They are designed to generate end-to-end motions in an offiin manner and are therefore slow, making them incompatible with real-time interaction with physics-based simulation. Furthermore, the large text-to-motion benchmarks on which MDM was trained, lack any description or awareness of the environment.
To overcome these limitations, we designed diffusion planner (DiP), real-time diffusion model that is aware of both the current state of the agent and the task to be performed. State awareness is achieved by autoregressively updating the motion plan, i.c. the diffusion planner receives input motion prefix representing the previous motion frames and outputs a prediction of the frames ahead. The model is further conditioned on (1) text prompt specifying the task (e.g. "A person is sitting on the sofa") and (2) target locations for one of the body joints (e.g. pelvis joint at the location of the sit). The latter serves ground the task description specific physical locations.
DiP follows the principles presented by Chen et al. 2024) in the design of a real-time auto-regressive diffusion model with transformer decoder backbone. Figure (top right) illustrates a single de- noising diffusion step. At each step t E [o, the model gets as input the clean prefix motion predx to be completed, the noisy motion prediction a.!red to be denoised, the diffusion step 6, and the two conditions- a text prompt, and a target location (optionally more than one). Here, prefix and upped with fixed lengths No and Ny, respectively, are concatenated, added to standard posi- tional embedding tokens to indicate their indices, and input to the transformer decoder backbone, such that each frame is represented by single input token. From the transformer's 's output, the last N₂ frames corresponding to .r.ree are the clean motion prediction a-prod whereas the first N, output frames corresponding to the prefix are not use.
Simultaneously, the model's conditions are constructed and injected through the cross-attention blocks of each of the transformer's 's layers. The text condition is encoded using a fixed instance of DistilBERT (Sanh 2019) to a sequence of latent tokens, followed by learned linear layer to coordinate the dimensions, resulting in Cueat E R Noten xd where d is the transformer's dimension. DistilBERT is lighter version of the popular text encoder BERT (Devlin 2018), which is not biased towards a global description of images, as CLIP (Radford et al. 2021 and as a result, was widely
used for text-to-motion (Petrovich et al. 2022 Athanasiou et al. 2022
The diffusion step and the target condition are embedded through two separate shallow fully con- nected networks to C, and Ciarger correspondingly, added together and concatenated Ciex con- form the full condition sequence (C2 + Cuarges, Cwex) E R (Nuiinn +1)xd This condition is injected into each of the transformer decoder's s layers through their cross-attention blocks.
Adaptive target conditioning. The target condition ej E TR3 specifies the desired location of joint j E J at the last predicted frame, where J is the subset of end-effector joints and the pelvis. Since different tasks require different joints, we want to adaptively condition DiP on different subsets of joints, and be able to change them on the fly as the task changes. To this end, we add boolean validity signal By per joint which indicates if it is currently used as a condition. In addition, we condition DiP on the heading angle of the body in the XY plane, co R. and its validity signal vo respectively. The full target condition is hence a concatenation of all joint locations and heading angle (Lcg)ies; (ug (ij)iescco,t
5

---

Published as a conference paper at ICLR 2025
Sit Down
Target strike
Reaching
Get up
"Performing cartwheel"
Figure 3: (Left) CLoSD generates versatile text-prompted physics-based motions. The SMPL- compatible physics model is rendered with the SMPL mesh. (Right) CLoSD perform sequence of RL tasks (see the web page video). Task transitions are user-specified via interactively changing the text, or via state machine, with transitions task done signal.
Target loss. Reaching the target condition the last predicted frame 2o [N is enacted via an additional loss term Luayet-- Since DiP predicts the motion ën at each diffusion step I, rather than the noise 6,, can follow MDM and implement the target loss geometric loss as follows:
Largot =0,l (2))) - -cyll - ( [ JEJ
where @ denotes angles difference and R22 is differentiable relative-to-gloaal representation converter. The last is needed since HumanML3D representation (See Section 3) defines each frame relative to the previous frame whereas Curget defines the target in global coordinates system.
Training. During training we sample (motion, text) pairs from the dataset, and a denoising timestep t 410, T). The motion is cropped
and =o respectively. xb is noised length N₁ + Ng with random offset, then split to Prefi resulting xe[rod The text prompt is encoded to Cjext using the frozen DistilBERT and to C₁ using the dedicated shallow network. To learn the random target condition, we take the heading angle and joint position of a randomly sampled joint j E J from the last frame of prd encode them the input Carge and use them the ground truth ej and ca for the target loss Lurget- DiP gets the above and predicts apred The full loss applied to the model is then L Lsimple Anarget Cuarett
Inference. We follow the standard DDPM iterative inference for ag prediction model: Initializing pred ~ (0, I). then for each / E [T. 1] we let DiP to predict agree partially noise it to x and repeat. At the last iteration, use DiP's output the final prediction .ared
Unlike MDM, which known for its high runtime cost, DiP is substantially faster. It generates 2 seconds of reference motion in 11.4 ms, i.e., 175x faster than real time. This acceleration is achieved by predicting shorter motion segments and reducing the number of diffusion steps. Pre- dicting shorter segments helps since the self-attention runtime depends quadratically on the sequence length. Additionally, as shown by Chen et al. (2024). auto-regress MDM requires fewer diffusion steps: able to reduce the number of diffusion-denoising steps from 50 in MDM to 10, and accelerate generation by a factor of
4.2 UNIVERSAL TRACKING POLICY
To perform DiP's planning in a physically simulated environment, we leverage a motion tracking policy that translates the kinematic motions to control forces that can be used to drive the movement of a simulated humanoid character. To this end, we use slim version of the Perpetual Humanoid Control (PHC) (Luo et al. (023), i.e. single PHC policy without the suggested mixture of experts. PHC a universal RL tracking policy, capable of imitating the large AMASS motion capture dataset with 99% success rate, where successful imitation is assumed when the maximal per-joint error less than 0.3 for all joints during the entire motion. PHC implements a Markov Decision Process (MDP) policy тиис that at each simulation step observes the current humanoid pose x [n] and the current pose to be imitated [n] and outputs the humanoid action a(n) driving PHC's keypoint-only state representation
it perform (e [n] at the next step. In detail,

---

Published as a conference paper at ICLR 2025
s(n] = [x= - [n= [n]). PHC uses proportional derivative (PD) controller at each DoF of the humanoid such that the action a(n] specifies the PD target. The goal reward is the negative exponent of the distance between [n] and =-in(n + 1] accompanied by adversarial reward as defined in AMP (Peng et al., 2021) and energy penalty. PHC is trained using the Proximal policy optimization (Schulman et al. 2017) (PPO). For full details of the implementation, we refer the
reader to (Luo et al. (023).
While PHC uses progressive neural networks with multiple primitives (policies) achieve robust fall recovery, training single primitive for a longer time yields a comparable imitation success rate. Since we desire a lightweight tracking policy that can be easily finetuned for multi-task training and we do not require fall recovery, we use the single primitive тPHс-
4.3 CLOSING THE LOOP
As shown in Figure the reference motion provided input to тHhс is the DiP predicted plan, converted to the PHC representation, n = () Whereas DiP generates Ng long motion segments at a time, ТPнС processes them trame-by-framen for each frame n, лтнс observes [n][[]] and 1:5mm(m and outputs a/n]. The simulator receives the action and outputs the next state of the humanoid which is buffered as - [n]. When the policy-simulator iterations reach the end of zt The last N, frames of v are cropped and used as the new input p G2R(2 :I) to DiP, after converting back to the relative representation, using the G2R block. DiP in turn generates a new pooedd to be consumed by ттнс-
Fine-tuning, Despite PHC's impressive motion tracking capabilities, there is a gap between the motion tracking task тhhС was trained on and the capabilities required by CLoSD, First, лтиС was trained using a relatively realistic motion capture dataset. While motion diffusion models are able to generate high-fidelity motions, they are still prone to generating physically inaccurate motions (uuan et al. 2023). It means that Thhe will need to fix such artifacts on the fly and avoid motions that might cause falling. Second, тPнС was trained to track motion capture without object interaction, and therefore does not learn the control required for interaction.
For those reasons, we fine-tune ттиС closed loop manner using Proximal Policy Optimization (PPO) (Schulman et al., (017)). Since both CLoSD and the imitation Ттнс are universal components, and since neither reward nor reset engineering is needed, we can fine-tune the same policy for multiple tasks simultancously. In practice, during fine-tuning, we fix DiP and for each episode randomly choose a task, then set the object, text prompt, and target location accordingly. We use the original PHC rewards and reset conditions, i.e., it is still trained to track a target motion, where the target motion is specified by DiP and involves different types of object interactions. The fast inference time of DiP and its relatively low memory usage enable training without compromising the large number of parallel environments required for PPO.
4.4 HIGH-LEVEL PLANNING USING STATE MACHINE
CLoSD is a multi-task agent. Each task is defined by the text prompt and the target location condi- tions. Since those conditions can be changed on the fly by the user, CLoSD is capable of performing sequence of different tasks at test time by simply transitioning between different target conditions. We automate the transitions using a simple state machine to demonstrate this. An example state- machine is illustrated in Figure 3 To choose meaningful transition point between tasks, each task signals when it is done. For example, sitting is done when the pelvis is on the sitting area of the sofa and striking done when the target hits the ground. When task done, the state machine can randomly select new one. Because the planning burden is taken care of by DiP, high-level planning for CLoSD can generally be quite simple.
EXPERIMENTS
We measure the performance of CLoSD using two experiments. Task performance ($.22) measures its success rate in goal reaching, target striking, sitting, and getting up. Text-to-motion $.3 mea- sures its ability to apply to text prompts and span the motion manifold according to the popular
7

---

Published as a conference paper at ICLR 2025
HumanML3D (Guo et al., 2022) benchmark and compares it to purely data-driven models. The code is available at ht : /tggttttubbbb// ctn.a/uui..co///yeeee//////a.
5.1 IMPLEMENTATION DETAILS
Motion representations. We use two different motion representations, each suitable for its purpose. For the kinematic motions generated by DiP, we follow MDM (Tevet al., (023) and use the HumanMI motion representation. Each frame n (i.e. a single pose) of the motion sequence x defined by xxn]] (fte fit fe ,,,,, "","""" "","",,,,, f) E R.,, where ja R is the root angular velocity along the Z-axis. prt p=, E R are root linear velocities on the XY-plane, and rM E R is the root height. j E [R( j" E RR(J-1) and j" E R3J are the local joint positions, velocities, and rotations with respect to the root, and E R are binary features denoting the foot contact labels for four-foot joints (two for each leg). For the RL tracking policy, we follow PHC (Luo et al. 2023) and represent frame (i.e. a single state) 5: [n] [j where jap E R the joint's global positions, and j5D E RS are the joint's 's linear velocities relative to the world (as opposed to j" which is relative the root).
j",
DiP is implemented as an 8 layer transformer decoder with a latent dimension of 512 and 4 attention heads. The text encoding is done using fixed instance of DistilBERT (Sanh) 2019). It was trained on the HumanML3D text-to-min eot-to-motion dataset using the MDM (Tevet et al. (023) framework for 600K diffusion steps on a single NVIDIA GeForce RTX 3090 GPU. We find the 200K checkpoint optimal for CLoSD and use it across our experiments. We use 10 diffusion steps, prefix length N, 20 and generation length N₂ = 40 as we find it a good trade-off between quality and runtime (Table ! This instance of DiP impressively outputs 3500 frames per second on the same GPU. The classifier- free guidance parameter used for DiP-only, the task experiments and the text-to-motion experiment 7.5, 2.5, and 5 respectively.
Tracking policy. We follow PHC (Luo et al., 2023) and train the joints only (without rotation angles tracking) single-primitive tracking policy over the AMASS (Mahmood al., 2019) dataset for 62K
PPO epochs. We further fine-tune it in the closed loop as described in Section 4.3 for 4K more
epochs. We train using the Isaac Gym simulator (Makoviychuk et al. (021 with 3072 parallel environments on a single NVIDIA A100 GPU.
Baselines. To understand the merits of the closed loop design, we suggest the open loop baseline, in which DiP generates the motion offline, fed by its own prediction, and then the tracker follows this fixed trajectory. In addition, report the performance of CLoSD before fine-tuning and exper- imenting with longer (Ng 80) and shorter (N, = 10) plans, as ours is 40 frames.
5.2
TASK EVALUATION
The following are the descriptions of the tasks to be evaluated. We fine-tune the tracking controller (closed loop) for all tasks simultancously, without any reward engineering. Each environment ran- domly initialized to perform one of the tasks. DiP performs planning according to the relevant text prompt and target location and the policy is fine-tuned with PPO with the original PHC objectives.
Goal reaching
Object striking
Sittnn
Getting
R-precision ? (top-3)
Runtime | [msec]
Speed
FID
[fps]
AMP 202
AMP
strike
reach
0.88
1.0
DiP (ours)
0.78
0.28
11.4
3.5 110
InterPhys
2023
sit
0.76
Diff steps
0.76
-10
UnittST (2024
0.96
20 Diff stef-steps
0.8
0.28
23
1.7 -1-
CLoSD (Ours)
1.0
0.9
0.98
N. 40
0.74
0.7
13
3.1 -1
shorter-loop
0.86
0.71
0.61
0.95
N 20
0.78
0.26
11.4
1.7 10
longer-loop
0.99
0.86
0.92
N₂
0.74
1.18
5-10"
w.o. fine-tuning
1,0
0.81
0.66
0.53
with PHC+ 2024
0.99
0.71
0.50
0.21
open-loop
1,0
0.8
0.19
0.23
Table 2: DiP ablation study. We use DiP
with prefix length N,, 20, generation length N, = 40, and 10 diffusion steps. DiP performs surprisingly well even with few as 5 diffusion steps.
Table 1: Task success rates. Bold and under- score relate to multi-task only. CLoSD signif- icantly excels on Striking and Get-up, which require careful object interaction.
8

---

Published as a conference paper at ICLR 2025
UniHSI [Xiao 2024]
Open loop
Close loop (No fine- fine-tuning)
Ours (with fine-tuning)
Comparisons of the getup task. The pelvis target is marked in cyan. CLoSD is able get up successfully with a human-like motion. Before fine-tuning on object interaction with the closed-loop, it was able to sit but struggled to get up. The open-loop baseline struggles with any interaction with the sofa due to the lack of re-planning. UniHSI (Xiao et al., (0224) was designed to minimize contact-poini distance and thus lifts the pelvis instead of getting up from the sofa.
Figure 4:
Goal Reaching, By specifying random target for the pelvis joint on the XY plane (without forcing a specific height) while specifying a text prompt defining the desired locomotion type (e.g. running, walking, jumping. dancing waltz, etc.) and its style (c.g. "rapidly", "like zombic", "slalom" ), CLoSD performers goal-reaching. While the target can be far away, and the motion plan x xred is typically up to two seconds long, we heuristically limit the velocity of the character by assigning intermediate targets to the last frame pred for each DiP iteration. The success criterion this task is when the pelvis joint XY location is at distance of 0.3 m] from the target.
0.3(m]
Object Striking, In this task, the character needs to strike down a standing kickboxing bag. The success criterion of this task is when the bag angle with the ground plane is less than 75 For each episode, we randomly sample one of the hands or feet end-effector joints and a text prompt aligned with it. For example, an optional text for the left hand might be "A person is punching the target with the left hand.". The target for the end-effector joint will be the center of the bag. For robustness, we randomly perturb the bag's location during both train and test.
Sitting down. In this task, the character needs to sit down on sofa. The target for the pelvis joint the center of the sofa's seat and the text specifies sitting down. The success criterion of the task is when the pelvis joint is on top of the sitting area. For robustness, we randomly perturb the sofa's 's location and orientation during both train and test.
Getting up. In this task, the initial pose is sitting down on sofa object and the goal is to get up to standing position. The target for the pelvis is placed 0.3[m] in front of the sofa, a standing height (0.9/ml). The text is designed appropriately, e.g. "A person is standing up from the sofa and remains standing.". The success criterion of this task when the pelvis joint is at a distance 0.3(mL from the target, which empirically implies successful standing up. In practice, and getting up on the same episode, when we start at a standing rest pose, a sitting target, and sitting text, then when the sitting success criterion is met, wait for another seconds and switch the target and text prompt to get up.
learn the sitring down
Results. Examples of behaviors learned by the models presented in Figures I 2 and the web page video. Table [ compares the success rate performance of CLoSD to single- and multi-task agents over 1K episodes per task. We trained UniHSI (Xiao al. 2024 our tasks for 12K PPO epochs until convergence. UniHSI models their multi-task agent by reaching a sequence of contact points. Without a semantic description of the task, UniHSI touches the target instead of striking it, and bending the pelvis instead of getting up. which leads to low success rates and is reflected Figure The open-loop baseline struggles successfully perform the interaction tasks, particularly for the sitting and get-up tasks, due to the inability to adapt to the actual motion in progress. Using the motion tracking controller without fine-tuning (both our single-primitive version and PHC+ (Luo et al. 2024), as provided by its authors) for closed-loop behavior also results in significantly degraded performance for interaction tasks. To stress test the robustness of the learned sitting controls, conducted
additional experiment in which randomly perturbed the sit's height between 20 to 60 60[cm] at test time only. The sitting and getting up success rates dropped slightly, from (0.88, 0.98) to (0.83, 0.95) respectively.

---

Published as a conference paper at ICLR 2025
Text-to motion
Physics based metrics
R-precision
FID
Muli-Modal Distance
Diversity
Physics model
Penetration [mm]
Floating [umm]
Skating
Top
Top 2
Top
[mm]
Ground Truth
0.405
0.632
0.746
0.001
2.95
9.51
0.0
22.9
206 -10
MDM (2023)
0.406
0.603
0,719
0.423
3,53
9.52
0.147
28.6
330-10-7
MoConvQ 2024
22224
0.309
0.504
0.614
3.279
3,97
8.01
/
0.249
32.0
294 -10
CLoSD (Ours)
0.385
0.578
0.693
1.755
3.65
8.23
0.021
20.1
1.2 .10
DiP only
0.464
0.668
0.777
0.283
3.15
9.21
x
0.083
23.6
629 029-10-"
shorter-loop
0.303
0.473
0.586
3.481
4.18
7.93
0.018
21.5
0.3 -10
longer-loop
0.369
0.559
0.674
1.671
3.72
8.20
0.015
19.8
1.2 122-10 8
open-loop
0.367
0.557
0.672
1.445
3,71
9.38
0.007
19.9
1-10
w.o. fine-tuning
0.367
0.555
0.670
2.154
3,75
9.34
0.016
20.2
1,6-10
Table 3: Text-to-motion on the HumanML3D benchmark (Guo et al. 2022). ) alongside the PhysDiff
metrics (Yuan al., 2023), which evaluate aspects of physical correctness. Diversity values closer to the ground truth are preferred.
5.3 TEXT-TO-MOTION EVALUATION
For text-to- -ot-ttoomoion evaluation, we disable the target conditioning and direct CLoSD to perform var- ious behaviors by conditioning on text only. We the popular HumanML3D benchmark, and compare our model to data-driven kinematic motion synthesis models and to MoConVQ, Yao et al. 2024), which is a state-of-the-art physics-based text-to-motion controller.
Text-to-motion metrics. We evaluate the text-to-motion performance using the HumanML.3D (Guo al. 2022) benchmark, which is widely used for this task. The benchmark uses text and motion encoders, mapping to the same latent space and learned using contrastive loss. The following metrics are calculated in that latent space. R-precision measures the proximity of the motion to the text was conditioned on. A generation is considered successful if its text is in the top-n closest prompts compared to 31 randomly sampled negative examples. FID measures the distance of the generated motion distribution to the ground truth distribution, Diversity measures the variance of generated motion, and MultiModel distance is the mean 1,2 distance between the pairs of text and conditioned motion. For full details, we refer the reader to the original paper.
Physics-based metrics. We report three physics-based metrics suggested by Yuan et al. (0233) to evaluate the physical plausibility of generated motions: Penetration measures the average distance between the ground and the lowest body joint position below the ground. Floating measures the average distance between the ground and the lowest body joint position above the ground. We use tolerance of 5mm to account for geometry approximation. Skating measures the average horizontal displacement within frames that demonstrate skating. Two adjacent frames are said to have skating if they both maintain contact to ground (minimum joint height < 5mm).
Results. Table 3 compares CLoSD to current methods. Generally, physics-based texl-to-motion significantly improves the physical metrics, but degrades the data distribution metrics. The latter is expected as daivenriven models model the distribution directly. To evaluate MoConVQ (Yao al. 2024). we collected samples using the code and model released by the authors, following their described procedures. In comparison to MoConVQ. which is physicc--assee based method, our method excels in all metrics. Our ablation study (bottom of Table 3 indicates that fine-tuning improves the results. Additionally, pure text-to-motion benefits from longer plans. Since closed-loop feedback crucial for object interaction, find our 2 second plans to be good trade-off.
6 CONCLUSIONS AND LIMITATIONS
We have presented method combine the flexible planning capabilities of diffusion models with the grounded realism and responsiveness of physics-based simulations. CLoSD performs text- prompt directable multi-task sequences involving physical interactions with objects in the envi- ronment, including navigation. striking an object with specified hands or feet. and sitting down or getting up. Adding further scene awareness, i.e., vision or height maps, the diffusion-based planner and the tracking controller will be important for achieving wider range of tasks. Further investigation could tackle longer-time-seee ning. Additionally, planning at latent action space, such as the recent PULSE (Luo et al. 20244) instead of the pose space, might enable a more straightforward tracking. Lastly, further adaptivity could be incorporated into the loop. For example, temporally adaptive loop could help with fast motions and settings where closer feedback important. Conversely, longer tracking horizons could mitigate occasional artifacts.
planning, as we focused on mid- and low-level plan-
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This research was partly supported by the Israel Science Foundation (1337/22), Len Blavatnik and the Blavatnik family foundation, The Tel Aviv University Innovation Laboratories (TILabs), and the NSERC Discovery Grant (DGECR-2023-0028)) RGPIN-2020-059299,
REFERENCES
Mazen Al Bomo, Martin De Lasa, and Aaron Hertzmann. Trajectory optimization for full-body movements with complex contacts. IEEE transactions visualization and computer graphics, 19(8):1405-1414, 2012.
Nikos Athanasiou, Mathis Petrovich, Michael J. Black, and Gül Varol. TEACH: Temporal Action Compositions for 3D Humans. In International Conference on 3D Vision (3DV), 2022.
Zhi Cen, Huaijin Pi, Sida Peng, Zehong Shen, Minghui Yang. Shuai Zhu, Hujun Bao, and Xiaowei Zhou. Generating human motion in 3d scenes from text descriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1855-1866, 2024.
Rui Chen, Mingyi Shi, Shaoli Huang, Ping Tan, Taku Komura, and Xuelin Chen. Taming diffusion probabilistic models for character control. In SIGGRAPH, 2024.
Nuttapong Chentanez, Matthias Müller, Miles Macklin, Viktor Makoviychuk, and Stefan Jeschke. Physics-based motion capture imitation with deep reinforcement learning. Proceedings MIG 2018: ACM SIGGRAPH Conference on Motion, Interaction, and Games, 2018.
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shu- ran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.041377 2023.
Setareh Cohan, Guy Tevet, Daniele Reda, Xue Bin Peng, and Michiel van de Panne. Flexible motion in-betweening with diffusion models. In ACM SIGGRAPH 2024 Conference Papers, pp- 1-9, 2024.
Rishabh Dabral, Muhammad Hamza Mughal, Vladislav Golyanik, and Christian Theobalt. Mo- fusion: A framework IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9760-9770, 2023.
motion synthesis. Proceedings of the
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding.
arXiv preprini arXiv:1810.0480 2018.
Levi Fussell, Kevin Bergamin, and Daniel Holden. Supertrack: motion tracking for physically simulated characters using supervised learning. ACM Trans. Graph., 40:1-13, 2021. ISSN 0730- 0301.
Purvi Goel, Kuan-Chieh Wang, C Karen Liu, and Kayvon Fatahalian. Iterative motion editing with natural language. In ACM SIGGRAPH 2024 Conference Papers. pp. 1-9, 2024.
Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng, Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5152-5161, June 2022.
Mohamed Hassan, Yunrong Guo, Tingwu Wang, Michael Black, Sanja Fidler, and Xue Bin Peng. Synthesizing physical character-scens scene interactions. In ACM SIGGRAPH 2023 Conference Pro- ceedings, SIGGRAPH '23, New York, NY, USA, 2023. Association for Computing Machin- ery. ISBN 9798400701597. doi: 10.1145/358843233551222 URL https: //do:. http:///oo...o...../ 1145 001553558432. 3591525
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models, Advances in Neural Information Processing Systems, 33:68400666 -6551, 2020.

---

Published as a conference paper at ICLR 2025
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey A. Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans. Im- agen video: High definition video generation with diffusion models. ArXiv, abs/2210.02303, 2022.
Siyuan Huang, Zan Wang. Puhao Li, Baoxiong Jia, Tengyu Liu, Yixin Zhu, Wei Liang, and Song- Chun Zhu. Diffusion-basee generation, optimization, and planning in 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16750-16761, 2023.
Jordan Juravsky, Yunrong Guo, Sanja Fidler, and Xue Bin Peng. Padl: Language-directed physics-based character control. In SIGGRAPH Asia 2022 Conference Papers, SA '22, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450394703. doi:
10.1145/3500499.555399 URL https:://doi.. org/ 1ttp://..1111155540333333333
Jordan Juravsky, Yunrong Guo, Sanja Fidler, and Xue Bin Peng. Superpadl: Scaling language- directed physics- based control with progressive supervised distillation. SIGGRAPH 2024 Con- ference Papers (SIGGRAPH '24 Conference Papers).. 2024.
Roy Kapon, Guy Tevet, Daniel Cohen-Or, and Amit Bermano. Mas: Multi-view ancestral sam- pling for 3d motion generation using 2d diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1965-1974, 2024.
Korrawe Karunratanakul, Konpat Preechakul, Emre Aksan, Thabo Beeler, Supasorn Suwajanakorn, and Siyu Tang. Optimizing diffusion noise can serve as universal motion priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1334-1345, 2024.
Nilesh Kulkarni, Davis Rempe, Kyle Genova, Abhijit Kundu, Justin Johnson, David Fouhey, and Leonidas Guibas. Nifty: Neural object interaction fields for guided human motion synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
947-957 2024.
Lei Li and Angela Dai. Genzi: Zero-shot 3d human-scene interaction generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20465-20474, 2024.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael Black. SMPL: A skinned multi-person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia), (:66):24:11224:111 16, October 2015.
Zhengyi Luo, Ryo Hachiuma, Ye Yuan, and Kris Kitani. Synamics-regulated kinematic policy for egocentric pose estimation. In Advances in Neural Information Processing Systems, 2021.
Zhengyi Luo, Shun Iwase, Ye Yuan, and Kris Kitani. Embodied scene-aware human pose estimation. In Advances in Neural Information Processing Systems, 2022.
Zhengyi Luo, Jinkun Cao, Kris Kitani, Weipeng Xu, et Perpetual humanoid control for real- time simulated avatars. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 10895-10904, 2023.
Zhengyi Luo, Jinkun Cao, Josh Merel, Alexander Winkler, Jing Huang, Kris M. Kitani, and Weipeng Xu. Universal humanoid motion representations for physics- control. In The Twelfth Inter- national Conference on Learning Representations, 2024. URL httttps://ppenrvvieee net/ Forum? 1id-mmdddddrdddddee
Naureen Mahmood, N. Ghorbani, N. Troje, Gerard Pons-Moll, and Michael J. Black. Amass: Archive of motion capture surface shapes. 2019 IEEE/CVF International Conference on Com- puter Vision (ICCV), pp. 5441-5450, 2019.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey. Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv: 108. 10470, 2021.
12

---

Published as a conference paper at ICLR 2025
Xiaogang Peng, Yiming Xie, Zizhao Wu, Varun Jampani, Deqing Sun, and Huaizu Jiang. Hoi-diff: Text-driven synthesis of 3d human-object interactions using diffusion models. arXiv preprint arXiv:2312.06551, 2023.
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: Example- guided deep reinforcement learning of physics-based based character skills. ACM Trans. Graph., 37
(4):143:1-144:14 July 2018. ISSN 0730-0301. doi: 10.1145/3173117.3200 //doi acm. org/ 0. 1114/31195751 9920///1111177321111
URLhttpp
Xue Bin Peng, Michael Chang, Grace Zhang, Pieter Abbeel, and Sergey Levine. Mcp: Learning composable hierarchical control with multiplicative compositional policies. Advances in neural information processing systems, 32, 2019.
Xue Bin Peng, Ze Ma, Picter Abbeel, Sergey Levine, and Angjoo Kanazawa. Amp: Adver- sarial motion priors for stylized physics-based character control. ACM Trans. Graph, 40 (4), July 2021. doi: 10.1145/3450626.445555 3450626. 3459670
URL httt:://ddooo acm. 1tc..oorg/10.115555.
Xue Bin Peng, Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters. ACM Trans. Graph., 41(4), July 2022.
Mathis Petrovich, Michael J. Black, and Gül Varol. TEMOS: Generating diverse human motions from textual descriptions. In European Conference on Computer Vision (ECCV), 2022.
Sigal Raab, Inbar Gat, Nathan Sala, Guy Tevet, Rotem Shalev-Arkushin, Ohad Fried, Amit H Bermano, and Daniel Cohen-Or. Monkey see, monkey do: Harnessing self-attention in motion diffusion for zero-shot motion transfer. arXiv preprint arXiv:2406. 06600000000060088, 2024a.
Sigal Raab, Inbal Leibovitch, Guy Tevet, Moab Arar, Amit H Bermano, and Daniel Cohen-Or. Single motion diffusion. The Twelfth International Conference on Learning Representations (ICLR), 2024b. BtRL:ttps:/ /twp///eeeeeeeeeeeeeeeeeeee /
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763, PMLR, 2021.
Davis Rempe, Zhengyi Luo, Xue Bin Peng. Ye Yuan, Kris Kitani, Karsten Kreis, Sanja Fidler, and Or Litany. Trace and pace: Controllable pedestrian animation via guided trajectory diffusion. In Conference on Computer Vision and Puttern Recognition (CVPR), 2023.
Jiawei Ren, Mingyuan Zhang, Cunjun Yu, Xiao Ma, Liang Pan, and Ziwei Liu. Insactor: Instruction- driven physics-based characters. NeurIPS, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Sanh. Distilbert, distilled version of bert: Smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.011 108, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06344 2017. URL http:://bbppp -utpe.ieri.t..r....... html #Schu manWDRK17
de/ /t//b//jouraala wwwr
707 http:///rraaaaa/coo/////oo000000
Yoni Shafir, Guy Tevet, Roy Kapon, and Amit Haim Bermano. Human motion diffusion as gen- erative prior. In The Twelfth International Conference on Learning Representations, 2024. URL
net Forum Forrmmiidddpppr.da
Yi Shi, Jingbo Wang. Xuekun Jiang, Bingkun Lin, Bo Dai, and Xue Bin Peng. Interactive character control with auto-regressive motion diffusion models. ACM Trans. Graph., 43, jul 2024.
13

---

Published as a conference paper at ICLR 2025
Jascha Sohl-Dicksteiis Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing. pp. 2256-2265, PMLR, 2015.
Yang Song and Stefano Ermon. Improved techniques for training score-based based generative models. Advances in neural information processing systems, 33: 13:12438-12448, 2020.
Chen Tessler, Yunrong Guo, Ofrr Nabati, Gal Chechik, and Xue Bin Peng. Unified physics-based character control through masked motion inpainting. arXiv preprint arXiv:2409, 143333,, 2024.
Maskedmimic:
Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or, Motionclip: Ex- posing human motion generation clip space. Computer Vision-ECCV 2022: 17th Euro- pean Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXII, pp. 358-374. Springer, 2022.
Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffusion model. In The Eleventh International Conference Learning Repre- sentations, 2023. URL https:
pttttps://ppereeee /wwp//peppeeeeeeeeoeeee net wtw.:evwwwiieee/oouuuiiiiiee Forummid SJ1kSdd2 jwu
Takara E Truong, Michael Piseno, Zhaoming Xie, and C Karen Liu. Pdp: Physics- character animation via diffusion policy. arXiv preprint arXiv:2406. 240.006066 2024.
Jonathan Tseng, Rodrigo Castellon, and Karen Liu. Edge: Editable dance generation from music. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 448-458, 2023.
Tingwu Wang, Yunrong Guo, Maria Shugrina, and Sanja Fidler. Unicon: Universal neural controller for physics-based character motion. arXiv, 2020. ISSN 2331-8422.
Zan Wang, Yixin Chen, Baoxiong Jia, Puhao Li, Jinlu Zhang, Jingze Zhang, Tengyu Liu, Yixin Zhu, Wei Liang, and Siyuan Huang. Move as you say interact as you can: languagge-guiedd human motion generation with scene affordance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 433 444, 2024.
Jungdam Won, Deepak Gopinath, and Jessica Hodgins. A scalable approach to control diverse behaviors for physically simulated characters. ACM Trans. Graph., 39, 2020. ISSN 0730- 1371,155
Qianyang Wu, Ye Shi, Xiaoshui Huang, Jingyi Yu,, Lan Xu, and Jingya Wang. Thor: Text to human- object interaction diffusion via relation intervention. arXiv preprint 1rXiv:2403.1/208, 2024.
Zeqi Xiao, Tai Wang, Jingbo Wang, Jinkun Cao, Wenwei Zhang, Bo Dai, Dahua Lin, and Jiangmiao Pang. Unified human-scene interaction via prompted chain-of-contacts. In The Twelfth Interna- tional Conference Learning Representations, 2024. URL /tttps:/cppenrete reviee.wwweeew.....e..e.... net/
forum? www.m:ddwwwwwnwx111mmmm
Zhaoming Xie, Jonathan Tseng, Sebastian Starke, Michiel van de Panne, and C Karen Liu. Hier- archical planning and control for box loco- aeoo-manipulation. Proceedings of the ACM on Computer Graphics and Interactive Techniques, 6(3):1-18, 2023.
Heyuan Yao, Zhenhua Song, Yuyang Zhou, Tenglong Ao, Baoquan Chen, and Libin Liu. Moconvq: Unified physics- based motion control via scalable discrete representations. ACM Transactions on Graphics (TOG), 43-4):1-21, 2024.
Hongwei Yi, Justus Thies, Michael J Black, Xue Bin Peng, and Davis Rempe. Generating human interaction motions in scenes with text control. arXiv preprint arXiv:2404.10685. 2024.
Ye Yuan and Kris Kitani. Residual force control for agile human behavior imitation and extended motion synthesis. Advances in Neural Information Processing Systems, 33:21763-211 2020.
Ye Yuan, Jiaming Song. Umar lqbal, Arash Vahdat, and Jan Kautz. Physdiff: Physics-guided human motion diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
14

---

Published as a conference paper at ICLR 2025
Wentao Zhu, Xiaoxuan Ma, Zhaoyang Liu, Libin Liu, Wayne Wu, and Yizhou Wang. Motionbert: A unified perspective on learning human motion representations. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 15085-15099, 2023.
15


Reviews:
[{"review_id": "G5fTiHKSTX", "reviewer": "Reviewer_MtoF", "summary": "This paper presents CLoSD, a text-driven reinforcement learning (RL) controller for character motion, merging the generative capabilities of diffusion models with the physical realism of RL-based control in a closed-loop framework. The proposed framework has two components: a Diffuion Planner (DiP), and a RL tracking controller. DiP is an auto-regressive diffusion model that is controlled by textual prompts and target locations and outputs motion plans. The RL tracking controller, based on the Perpetual Humanoid Control (PHC) model, interprets DiP’s motion plans and executes them in a physics simulation.", "questions": "Please see weaknesses above.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written, with the key contributions listed clearly in the introduction. The authors also identified relevant literature in this line of research and highlighted their contributions in the method. The authors also included low-level technical details which aid to the reproducibility of the work. Experimental evaluations highlight CLoSD’s superior performance on multi-task benchmarks, including goal-reaching, object striking, sitting and getting up. In tasks requiring environmental interaction, CLoSD significantly outperforms both state-of-the-art open-loop diffusion models and other text-to-motion RL systems, demonstrating its ability to maintain stable, responsive control across diverse physical interactions. Overall, I believe this paper is a good step towards understanding the interaction between generative models and physical simulation.", "weaknesses": "Recent generative models are capable of generating plans in much larger observation spaces. Can the authors comment on how this work can involve sensory inputs (if at all), such as vision, to enhance CLoSD's adaptability in more complex environments? Moreover, can the authors comment on the potential failure modes of their method if their method was used for a longer planning horizon to generate even more intricate motion sequences?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4r8fMZ8EQg", "reviewer": "Reviewer_U9rt", "summary": "This work presents a method that combines a diffusion-based motion planner with a tracking controller in a closed-loop setup. The motion planner is an autoregressive diffusion model that predicts future character poses based on previous states and conditioning inputs, such as joint target positions and text descriptions. The controller is an off-the-shelf state-of-the-art tracking controller finetuned to the motions generated by the DiP. The result is a text-to-motion controller that can be used to solve different tasks while having control over the style of the motions.", "questions": "**Tracking Policy**\n\nI have a few questions regarding the PHC setup. It is stated that a simplified version of PHC was used, excluding the mixture of experts. \n* Does this imply that only a single primitive policy was trained? If so, is the controller essentially a standard policy trained solely with AMP and tracking rewards? \n* Additionally, how would the full PHC version perform in comparison? \n* Would an alternative controller, such as the one proposed by Chentanez et al. 2018, which incorporates future frames, potentially be more effective?\n\nThere has been more recent work on character control for tracking reference motions, including newer controllers developed in 2024. \n* Consider referencing newer motion tracking controllers to provide a more current perspective. \n* Additionally, other methods have also explored controllers that operate without a reference motion. While these are not directly relevant, It would be beneficial to discuss such approaches, including whether the integration of a DiP-like approach is feasible in this context. \n\n**Comparisons with UniHSI**\n\nUniHSI uses an LLM with scene information to design a chain of contacts. Their demonstrated results look less smooth than CLoSD; however, they look much better than those presented here. For example, their pipeline shows successful „stand-up “ performance. As far as I understand, in the comparisons here, the LLM Planner was omitted, and instead, a fixed target label for the pelvis alone is defined. \n* The discrepancy between UniHSI’s original results and the lower-quality outcomes here needs clearer explanation and precise clarification. \n* Further UniHSI shows more complex chains of tasks, sitting on different objects, lying down etc. Can CLoSD also sit on variable hights? where are the limits of versatility?\n\n**Text-to-Motion**\n* PhysDiff’s metrics are used to evaluate the physical correctness of the text-to-motion quality compared to MDM and MoConVQ. It seems more intuitive to use PhysDiff (MDM + Physical Projection) as an additional comparison to understand how much the closed-loop approach improves over the proposed physical projection in PhysDiff.\n* As far as I understand, the CLoSD results are the motions performed in the physics-based simulator. This makes me wonder why there is still such a high floating error of 2cm on average.\n* It would be helpful to visualize a side-by-side comparison of DiP and MDM motions, as DiP appears to significantly outperform MDM in this aspect.\n* A comparison between DiP and CAMDM is needed to clarify why DiP might be better suited than the Chen et al. 2024 architecture mentioned in line 236.\n\n**Motion Quality**\n* The presented motions in the video are not very smooth and contain a lot of shakiness and artifacts that seem to trick the physics (e.g., fast vibrations allowing the character to drift - 02:42)\n* In the renderings with the SMPL mesh, strong artifacts in hand pose and feet are visible. Can you explain what is causing those artifacts that are less severe in MDM.\n* In the last motion in the video, “A person is waving goodbye,” the model attempts a simple motion that MDM typically handles with high precision, yet here it appears less accurate. This raises the question of whether CLoSD is overfitted to the specific tasks, potentially limiting its ability to leverage the full capabilities of DiP/MDM in terms of motion quality and diversity.\n\n**Limitations**\n* MDM can generate a diverse range of dynamic motions, and PHC can track many AMASS motions. However, the motions presented here appear more constrained. A more detailed explanation of the method’s limitations would be helpful.\n* The method’s applications rely heavily on various tricks, making it challenging to determine how much of the outcome is due to human engineering versus the method’s inherent capabilities. For instance, to ensure the object is hit, the user must specify a joint target location and an appropriate prompt. Consequently, the approach appears overfitted to this specific set of motions/tasks and does not fully utilize the generative capabilities of the diffusion model. It would be helpful if the authors could discuss these limitations further.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The goal of the paper is clear: combine the strength of a DIffusion Model (MDM) with a physics-based controller (PHC).\n* The method is simple yet interesting enough.\n* The paper is well-written, and the method is clearly explained.\n* Code will be publicly available.", "weaknesses": "* While the method is well explained, the experimental section lacks clarity and precision. (see *Questions*)\n* The experiments could be better designed. \n* The video shows lower motion quality than state-of-the-art physics-based character control, which the paper does not address.\n* The limitations are not clearly stated and discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l5a5uwfjql", "reviewer": "Reviewer_1o2G", "summary": "This paper presents CloSD, a method that combines a diffusion planner with a reinforcement learning tracking policy to enable multi-task character control. The results demonstrate that CloSD achieves natural and versatile motions.", "questions": "- I am curious about the difference between open-loop and closed-loop approaches. In Table 1, the open-loop setting performs well on “goal reaching” and “object striking” tasks but struggles with “sitting” and “getting up” tasks. Could you elaborate on why re-planning is necessary in these cases?\n- $x^{\\text {prefix }}$ is defined as “current frames,” but does this risk the model forgetting its previous plans? In other words, might it be more effective to define $x^{\\text {prefix }}$ as motion tracking errors, or are there alternative design choices that could work better?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper combines a diffusion planner with an RL motion tracker to achieve natural and versatile physics-based character control across multiple tasks.\n- CloSD outperforms previous methods, such as InterPhys and UniHSI, and provide a solid framework for future research.\n- Extensive results and ablation studies demonstrate the effectiveness of each design choices and influence of hyper-parameters.\n- The paper is overall well-structured and well-written.", "weaknesses": "Please refer to the questions part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gHjrWe0PkP", "reviewer": "Reviewer_N8zK", "summary": "The paper presents CLoSD, a multi-task character motion generation approach that combines text-driven motion geneation models and reinforcement learning to control physics-based simulations of human-like motion. By combining motion diffusion, which generates a variety of text-based motions, and RL, which enables interaction with the environment, CLoSD ensures both diversity and physical accuracy in character motion. The system operates in a closed loop, where a Diffusion Planner (DiP) generates motion plans from text prompts and target locations, and a tracking controller uses these plans to adapt the character’s actions in real-time. This setup enables motion generation across tasks like navigating to a target, interacting with objects, and completing actions like sitting or getting up. The approach demonstrates high task success rates, adaptability to on-the-fly text changes, and a significant improvement over existing controllers, as it allows characters to perform varied tasks while adhering to realistic physical constraints.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces a closed-loop integration between motion diffusion and RL, leveraging each method’s strengths and overcoming traditional limitations, such as the need for pre-planned motion sequences or manual fine-tuning.\n2. CLoSD’s capability to execute multiple tasks through simple text prompts and target positions highlights its flexibility and generalization, reducing the need for task-specific controllers.\n3. By optimizing the Diffusion Planner to be both autoregressive and fast, CLoSD maintains responsiveness in dynamic environments, a crucial factor for real-world applications.", "weaknesses": "1. The paper could benefit from better contextualization. While the integration of CAMDM and PHC for multitask character control is intriguing and effectively combines a diffusion model for language-conditioned motion generation with a tracker to ensure physical feasibility, the Related Work section would benefit from a deeper discussion since similar architecture has been explored in previous works like Trace and Pace or InsActor. Specifically, while it briefly mentions these methods, it lacks depth on the specific contributions of CLoSD that differ from such hierarchical decomposition in language-conditioned motion generation as explored in prior works. Including insights on how CLoSD advances or contrasts with these methods, such as motion in-painting, goal-reaching, and motion chaining demonstrated in InsActor, could enhance readers’ understanding of the unique contributions of this work.\n2. The novelty of the work could be better conveyed. While the combination of CAMDM and PHC for multitask character control aligns with existing architectures, CLoSD’s main distinguishing feature is its real-time generation capability, which largely stems from the use of CAMDM. A clearer exposition of the unique contributions and advancements CLoSD introduces—beyond real-time generation—would help emphasize its originality and innovation.\n3. The experimental evaluation is somewhat limited in scope. Although the paper includes experiments on motion generation with HumanML3D, a broader range of tasks would provide a more comprehensive validation of the approach. Since the method is intended to enable character interactions with environmental objects, including more diverse interaction tasks beyond the four basic ones presented would greatly enrich the empirical support and demonstrate the approach’s robustness and generalizability.\n\nOverall, the paper presents an interesting approach to interactive character control. However, I would appreciate a deeper discussion that thoroughly compares the proposed work with prior works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)