CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character control

Authors: Guy Tevet, Sigal Raab, Setareh Cohan, Daniele Reda, Zhengyi Luo, Xue Bin Peng, Amit Haim Bermano, Michiel van de Panne
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: RL, PPO, motion, motion generation, motion synthesis, synthesis, generative models, diffusion, animation

Abstract:
Motion diffusion models and Reinforcement Learning (RL) based control for physics-based simulations have complementary strengths for human motion generation. The former is capable of generating a wide variety of motions, adhering to intuitive control such as text, while the latter offers physically plausible motion and direct interaction with the environment. In this work, we present a method that combines their respective strengths. CLoSD is a text-driven RL physics-based controller, guided by diffusion generation for various tasks. Our key insight is that motion diffusion can serve as an on-the-fly universal planner for a robust RL controller. To this end, CLoSD maintains a closed-loop interaction between two modules — a Diffusion Planner (DiP), and a tracking controller. DiP is a fast-responding autoregressive diffusion model, controlled by textual prompts and target locations, and the controller is a simple and robust motion imitator that continuously receives motion plans from DiP and provides feedback from the environment. CLoSD is capable of seamlessly performing a sequence of different tasks, including navigation to a goal location, striking an object with a hand or foot as specified in a text prompt, sitting down, and getting up.

Decision: accept (spotlight)
Reviewer scores: 8, 8, 8, 6

Reviews and discussion:
<thread>
creation date:
1730143377987

thread_type:
official_review

<thread_object>
creator: reviewer_n8zk

<content>
summary:
The paper presents CLoSD, a multi-task character motion generation approach that combines text-driven motion geneation models and reinforcement learning to control physics-based simulations of human-like motion. By combining motion diffusion, which generates a variety of text-based motions, and RL, which enables interaction with the environment, CLoSD ensures both diversity and physical accuracy in character motion. The system operates in a closed loop, where a Diffusion Planner (DiP) generates motion plans from text prompts and target locations, and a tracking controller uses these plans to adapt the character’s actions in real-time. This setup enables motion generation across tasks like navigating to a target, interacting with objects, and completing actions like sitting or getting up. The approach demonstrates high task success rates, adaptability to on-the-fly text changes, and a significant improvement over existing controllers, as it allows characters to perform varied tasks while adhering to realistic physical constraints.

strengths:
1. The paper introduces a closed-loop integration between motion diffusion and RL, leveraging each method’s strengths and overcoming traditional limitations, such as the need for pre-planned motion sequences or manual fine-tuning.
2. CLoSD’s capability to execute multiple tasks through simple text prompts and target positions highlights its flexibility and generalization, reducing the need for task-specific controllers.
3. By optimizing the Diffusion Planner to be both autoregressive and fast, CLoSD maintains responsiveness in dynamic environments, a crucial factor for real-world applications.

weaknesses:
1. The paper could benefit from better contextualization. While the integration of CAMDM and PHC for multitask character control is intriguing and effectively combines a diffusion model for language-conditioned motion generation with a tracker to ensure physical feasibility, the Related Work section would benefit from a deeper discussion since similar architecture has been explored in previous works like Trace and Pace or InsActor. Specifically, while it briefly mentions these methods, it lacks depth on the specific contributions of CLoSD that differ from such hierarchical decomposition in language-conditioned motion generation as explored in prior works. Including insights on how CLoSD advances or contrasts with these methods, such as motion in-painting, goal-reaching, and motion chaining demonstrated in InsActor, could enhance readers’ understanding of the unique contributions of this work.
2. The novelty of the work could be better conveyed. While the combination of CAMDM and PHC for multitask character control aligns with existing architectures, CLoSD’s main distinguishing feature is its real-time generation capability, which largely stems from the use of CAMDM. A clearer exposition of the unique contributions and advancements CLoSD introduces—beyond real-time generation—would help emphasize its originality and innovation.
3. The experimental evaluation is somewhat limited in scope. Although the paper includes experiments on motion generation with HumanML3D, a broader range of tasks would provide a more comprehensive validation of the approach. Since the method is intended to enable character interactions with environmental objects, including more diverse interaction tasks beyond the four basic ones presented would greatly enrich the empirical support and demonstrate the approach’s robustness and generalizability.

Overall, the paper presents an interesting approach to interactive character control. However, I would appreciate a deeper discussion that thoroughly compares the proposed work with prior works.

questions:
See weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730433845893

thread_type:
official_review

<thread_object>
creator: reviewer_1o2g

<content>
summary:
This paper presents CloSD, a method that combines a diffusion planner with a reinforcement learning tracking policy to enable multi-task character control. The results demonstrate that CloSD achieves natural and versatile motions.

strengths:
- This paper combines a diffusion planner with an RL motion tracker to achieve natural and versatile physics-based character control across multiple tasks.
- CloSD outperforms previous methods, such as InterPhys and UniHSI, and provide a solid framework for future research.
- Extensive results and ablation studies demonstrate the effectiveness of each design choices and influence of hyper-parameters.
- The paper is overall well-structured and well-written.

weaknesses:
Please refer to the questions part.

questions:
- I am curious about the difference between open-loop and closed-loop approaches. In Table 1, the open-loop setting performs well on “goal reaching” and “object striking” tasks but struggles with “sitting” and “getting up” tasks. Could you elaborate on why re-planning is necessary in these cases?
- $x^{\text {prefix }}$ is defined as “current frames,” but does this risk the model forgetting its previous plans? In other words, might it be more effective to define $x^{\text {prefix }}$ as motion tracking errors, or are there alternative design choices that could work better?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730567391047

thread_type:
official_review

<thread_object>
creator: reviewer_u9rt

<content>
summary:
This work presents a method that combines a diffusion-based motion planner with a tracking controller in a closed-loop setup. The motion planner is an autoregressive diffusion model that predicts future character poses based on previous states and conditioning inputs, such as joint target positions and text descriptions. The controller is an off-the-shelf state-of-the-art tracking controller finetuned to the motions generated by the DiP. The result is a text-to-motion controller that can be used to solve different tasks while having control over the style of the motions.

strengths:
* The goal of the paper is clear: combine the strength of a DIffusion Model (MDM) with a physics-based controller (PHC).
* The method is simple yet interesting enough.
* The paper is well-written, and the method is clearly explained.
* Code will be publicly available.

weaknesses:
* While the method is well explained, the experimental section lacks clarity and precision. (see *Questions*)
* The experiments could be better designed. 
* The video shows lower motion quality than state-of-the-art physics-based character control, which the paper does not address.
* The limitations are not clearly stated and discussed.

questions:
**Tracking Policy**

I have a few questions regarding the PHC setup. It is stated that a simplified version of PHC was used, excluding the mixture of experts. 
* Does this imply that only a single primitive policy was trained? If so, is the controller essentially a standard policy trained solely with AMP and tracking rewards? 
* Additionally, how would the full PHC version perform in comparison? 
* Would an alternative controller, such as the one proposed by Chentanez et al. 2018, which incorporates future frames, potentially be more effective?

There has been more recent work on character control for tracking reference motions, including newer controllers developed in 2024. 
* Consider referencing newer motion tracking controllers to provide a more current perspective. 
* Additionally, other methods have also explored controllers that operate without a reference motion. While these are not directly relevant, It would be beneficial to discuss such approaches, including whether the integration of a DiP-like approach is feasible in this context. 

**Comparisons with UniHSI**

UniHSI uses an LLM with scene information to design a chain of contacts. Their demonstrated results look less smooth than CLoSD; however, they look much better than those presented here. For example, their pipeline shows successful „stand-up “ performance. As far as I understand, in the comparisons here, the LLM Planner was omitted, and instead, a fixed target label for the pelvis alone is defined. 
* The discrepancy between UniHSI’s original results and the lower-quality outcomes here needs clearer explanation and precise clarification. 
* Further UniHSI shows more complex chains of tasks, sitting on different objects, lying down etc. Can CLoSD also sit on variable hights? where are the limits of versatility?

**Text-to-Motion**
* PhysDiff’s metrics are used to evaluate the physical correctness of the text-to-motion quality compared to MDM and MoConVQ. It seems more intuitive to use PhysDiff (MDM + Physical Projection) as an additional comparison to understand how much the closed-loop approach improves over the proposed physical projection in PhysDiff.
* As far as I understand, the CLoSD results are the motions performed in the physics-based simulator. This makes me wonder why there is still such a high floating error of 2cm on average.
* It would be helpful to visualize a side-by-side comparison of DiP and MDM motions, as DiP appears to significantly outperform MDM in this aspect.
* A comparison between DiP and CAMDM is needed to clarify why DiP might be better suited than the Chen et al. 2024 architecture mentioned in line 236.

**Motion Quality**
* The presented motions in the video are not very smooth and contain a lot of shakiness and artifacts that seem to trick the physics (e.g., fast vibrations allowing the character to drift - 02:42)
* In the renderings with the SMPL mesh, strong artifacts in hand pose and feet are visible. Can you explain what is causing those artifacts that are less severe in MDM.
* In the last motion in the video, “A person is waving goodbye,” the model attempts a simple motion that MDM typically handles with high precision, yet here it appears less accurate. This raises the question of whether CLoSD is overfitted to the specific tasks, potentially limiting its ability to leverage the full capabilities of DiP/MDM in terms of motion quality and diversity.

**Limitations**
* MDM can generate a diverse range of dynamic motions, and PHC can track many AMASS motions. However, the motions presented here appear more constrained. A more detailed explanation of the method’s limitations would be helpful.
* The method’s applications rely heavily on various tricks, making it challenging to determine how much of the outcome is due to human engineering versus the method’s inherent capabilities. For instance, to ensure the object is hit, the user must specify a joint target location and an appropriate prompt. Consequently, the approach appears overfitted to this specific set of motions/tasks and does not fully utilize the generative capabilities of the diffusion model. It would be helpful if the authors could discuss these limitations further.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730584880915

thread_type:
official_review

<thread_object>
creator: reviewer_mtof

<content>
summary:
This paper presents CLoSD, a text-driven reinforcement learning (RL) controller for character motion, merging the generative capabilities of diffusion models with the physical realism of RL-based control in a closed-loop framework. The proposed framework has two components: a Diffuion Planner (DiP), and a RL tracking controller. DiP is an auto-regressive diffusion model that is controlled by textual prompts and target locations and outputs motion plans. The RL tracking controller, based on the Perpetual Humanoid Control (PHC) model, interprets DiP’s motion plans and executes them in a physics simulation.

strengths:
The paper is well-written, with the key contributions listed clearly in the introduction. The authors also identified relevant literature in this line of research and highlighted their contributions in the method. The authors also included low-level technical details which aid to the reproducibility of the work. Experimental evaluations highlight CLoSD’s superior performance on multi-task benchmarks, including goal-reaching, object striking, sitting and getting up. In tasks requiring environmental interaction, CLoSD significantly outperforms both state-of-the-art open-loop diffusion models and other text-to-motion RL systems, demonstrating its ability to maintain stable, responsive control across diverse physical interactions. Overall, I believe this paper is a good step towards understanding the interaction between generative models and physical simulation.

weaknesses:
Recent generative models are capable of generating plans in much larger observation spaces. Can the authors comment on how this work can involve sensory inputs (if at all), such as vision, to enhance CLoSD's adaptability in more complex environments? Moreover, can the authors comment on the potential failure modes of their method if their method was used for a longer planning horizon to generate even more intricate motion sequences?

questions:
Please see weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732263001501

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We thank the reviewers for their comprehensive reviews and positive feedback. 
Thank you for noting our work “achieve natural and versatile physics-based character control across multiple tasks” (1o2G), “overcoming traditional limitations, such as the need for pre-planned motion sequences or manual fine-tuning” (N8zK), “outperforms previous methods” (1o2G), and “demonstrates high task success rates, adaptability to on-the-fly text changes, and a significant improvement over existing controllers” (N8zK). MtoF concluded that “this paper is a good step towards understanding the interaction between generative models and physical simulation.”. You also mentioned that the paper is well-written (MtoF, U9rt, 1o2G), and the method is clearly explained (MtoF, U9rt).

We have revised the paper according to your feedback (changes are highlighted in blue) and provided more detailed responses in the individual messages.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents CLoSD,  method combining motion diffusion models with RL-based control for physics-based character motion generation. The key contribution is a closed-loop framework integrating a Diffusion Planner (DiP) and tracking controller for real-time text-driven motion generation.

Strengths:
+ Novel integration of diffusion models and RL to achieve fast, closed-loop simulated character motion generation. 
+ Strong experimental results on diverse scenarios and motion complexity
+ Well-documented implementation details

Weaknesses:
- Initial concerns about comparison with and differentiation from prior work like Trace & Pace and InsActor, though addressed in discussion
- Motion artifacts and jitters in some cases, particularly in SMPL mesh visualization
- Limited experimental evaluation scope, though expanded through additional tests
- Some technical details needed clarification, especially around control signal choices

The reviewers recommended acceptance after authors provided thorough responses addressing their concerns.