Motion-Agent: A Conversational Framework for Human Motion Generation with LLMs

Authors: Qi Wu, Yubo Zhao, Yifan Wang, Xinhang Liu, Yu-Wing Tai, Chi-Keung Tang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: 3D human motion, multimodal LLM, motion generation, conversational AI

Abstract:
While previous approaches to 3D human motion generation have achieved notable success, they often rely on extensive training and are limited to specific tasks. To address these challenges, we introduce **Motion-Agent**, an efficient conversational framework designed for general human motion generation, editing, and understanding. 
Motion-Agent employs an open-source pre-trained language model to develop a generative agent, **MotionLLM**, that bridges the gap between motion and text. This is accomplished by encoding and quantizing motions into discrete tokens that align with the language model's vocabulary. With only 1-3% of the model's parameters fine-tuned using adapters, MotionLLM delivers performance on par with diffusion models and other transformer-based methods trained from scratch. By integrating MotionLLM with GPT-4 without additional training, Motion-Agent is able to generate highly complex motion sequences through multi-turn conversations, a capability that previous models have struggled to achieve.
Motion-Agent supports a wide range of motion-language tasks, offering versatile capabilities for generating and customizing human motion through interactive conversational exchanges.

Decision: accept (poster)
Reviewer scores: 6, 8, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730129856533

thread_type:
official_review

<thread_object>
creator: reviewer_omaw

<content>
summary:
This paper introduces Motion-Agent, an efficient framework for general human motion generation, and it also develops a generative agent, Motion LLM, which bridges the gap between motion and text. Experiments show that with few-shot fine-tuning, MotionLLM can achieve state-of-the-art performance. Motion-Agent can generate complex motion sequences.

strengths:
1. This paper presents a framework to bridge text and motion. This part is novel in the opinion of the reviewer. The proposed motion tokenization approach and Motion-language Agent have some level of novelty.

2. Table 1 summarizes the advantages of this work against several recent competitors. It seems that this work supports multi-turn editing, reasoning, and composition. Also, Motion LLM can generate longer motion, which is superior to previous works.

3. Motions in this work and its supplementary (including video results) look cool. This indicates that the proposed model can generate complex motions.

weaknesses:
1. The website in the supplementary cannot function well (videos cannot be rendered). The reviewer suggests authors publish their websites on anonymous hosts (such as Google Pages). Moreover, it's better to show case more diverse prompts.

2. More ablation studies are welcomed. For example, different types of tokenizers, detailed comparison between LLaMa and Gemma, etc.

questions:
1. What's the rendering engine to generate those visualizations?

2. Can this framework benefit down stream tasks such as robotics and policy learning?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730643091097

thread_type:
official_review

<thread_object>
creator: reviewer_qach

<content>
summary:
Motion Agent introduces a conversational framework for generating, editing, and reasoning about human motion in multiple steps. This framework is built on a MotionLLM fine-tuned with LoRA. With a motion tokenizer-detokenizer and a text detokenizer, Motion Agent can create new motions and generate text that explains or describes the motions in natural language. Fine-tuning the large LLM allows Motion Agent to generalize better than current state-of-the-art methods, produce longer motion sequences, and handle language-guided edits.

strengths:
1) The paper is well-written and supported by experiments, with a good presentation of ablation studies and applications. The supplementary material includes videos for temporal generations, which helps in reviewing the work better.

2) This research is timely, as using LoRA fine-tuning on LLMs for multimodal human motion generation allows for models that don’t rely on task-specific data. 

3) Multi-turn editing and the ability to generate long motion sequences are valuable directions for the field.

weaknesses:
1) Lines 280-282: Multiple adapters are mentioned to work together, and fig 2 shows two LoRA adapters after fine-tuning. It looks like they’re applied to the base LLM in separate branches. How does MotionLLM.generate or MotionLLM.caption work—are they called every time during inference, or just when needed? This information would help clarify the pipeline.

2) Line 323 states that there is no ground truth for these tasks, but the videos show apparent issues like body interpenetration, lack of physical constraints (e.g., floor contact), and body deformation. There are also no quantitative metrics for multi-turn edited generations. Even though the motion quality seems better than the current SOTA, these limitations stand out. One way to address this would be to explain why these interpenetrations occur, especially since the motion tokenizer/detokenizer was trained on motion reconstruction losses, which should ideally prevent this. Another approach could be to generate multi-turn prompts based on test set motion sequences and provide metrics like FID, where the test sample could act as a pseudo ground truth.

3) The examples in fig 3 show different genders/body types across three instances. How does Motion Agent choose each identity or gender? Are all identities compatible in terms of body kinematics, or could retargeting cause issues if aiming for consistent identity? This needs more explanation.

4) Figure 5 shows in-betweening, but there’s no quantitative metric to evaluate it. Using a metric like NPSS (e.g., https://arxiv.org/pdf/1809.03036, https://arxiv.org/pdf/2102.04942) could provide useful information on the in-between motion quality.

5) Tab 2 - Generation Task: FID scores are lower than the state-of-the-art. As noted in Lines 416-418, could the motion length be constrained to match other methods to provide a clearer comparison of the FID scores?

questions:
1) What are the benefits of using VQVAE over other models like diffusion/VAE?

2) Which body joint representations were used for training?

3) The extended motion sequences keep the original motion unchanged. This is visible in all the provided videos. For example, with prompt p1, motion m1 is generated; when p2 is added, it generates a new motion m2, so the combined motion is m1 + m2. How is m1 preserved—does the new generation only respond to the new prompt?

4) Lines 191-192 mention that the motion detokenizer is important for smoothing transitions between different motion sequences. Are the motions generated by MotionLLM concatenated and sent to the motion detokenizer after each prompt?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730681062269

thread_type:
official_review

<thread_object>
creator: reviewer_deqa

<content>
summary:
The paper introduces Motion-Agent, a novel conversational framework for 3D human motion generation, editing, and understanding that leverages large language models (LLMs). The framework employs a pre-trained language model called MotionLLM, which bridges the gap between motion and text by encoding and quantizing motions into discrete tokens aligned with the language model's vocabulary. By fine-tuning only 1–3% of the model's parameters using adapters, MotionLLM achieves performance comparable to diffusion models and other transformer-based methods trained from scratch. Integrating MotionLLM with GPT-4 without additional training, Motion-Agent enables the generation of highly complex motion sequences through multi-turn conversations, a capability previous models lacked. The framework supports a wide range of motion-language tasks, offering versatile capabilities for generating and customizing human motion through interactive conversational exchanges. Experiments demonstrate that Motion-Agent effectively handles intricate, multi-turn interactions and achieves strong results in both motion generation and captioning tasks.

strengths:
Originality: The paper presents an innovative approach by integrating large language models into 3D human motion generation, a relatively unexplored area. The conversational framework allowing multi-turn interactions for motion generation and editing is particularly novel.
Quality: The proposed method effectively leverages pre-trained LLMs with minimal fine-tuning, demonstrating competitive performance with significantly reduced computational resources compared to models trained from scratch.
Clarity: The methodology is well-articulated, with clear explanations of the motion tokenizer/detokenizer and how MotionLLM integrates with GPT-4. The inclusion of qualitative examples and ablation studies aids in understanding the framework's capabilities.
Significance: The ability to generate complex, customizable human motions through conversational interactions has substantial implications for animation, virtual reality, and human-computer interaction. The framework's versatility enhances its practical relevance in various applications.

weaknesses:
Evaluation Metrics: The paper primarily presents qualitative results and some standard metrics but lacks comprehensive quantitative comparisons with state-of-the-art models on standardized benchmarks, making it challenging to fully assess performance improvements.
Limited Ablation Studies: While an ablation study is included, more extensive experiments isolating the contributions of each component (e.g., the impact of different LLMs, the motion tokenizer) would strengthen the evaluation.
Dependence on Proprietary Models: Utilizing GPT-4 as the conversational LLM may limit reproducibility and accessibility, as GPT-4 is not openly available to all researchers.
Scalability and Efficiency: The paper does not thoroughly discuss the computational efficiency and real-time performance of the framework, which is critical for interactive applications.

questions:
Can the authors provide more quantitative evaluations comparing Motion-Agent with state-of-the-art models on standard benchmarks?
How does the choice of the pre-trained LLM (e.g., GPT-4 vs. open-source models) impact the performance of the framework? Have experiments been conducted with other LLMs?
What are the computational requirements (e.g., inference time, memory usage) of Motion-Agent, especially for real-time interactive applications?
How does the framework handle motions involving interactions with the environment or multiple agents (e.g., multi-human interactions)?
Are there limitations regarding the diversity or realism of the generated motions when handling highly complex or extended sequences?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730686389335

thread_type:
official_review

<thread_object>
creator: reviewer_7jda

<content>
summary:
This work presents a framework capable of understanding, generating, and editing 3D human motion based on dialogue. Specifically, the model includes a more advanced LLM (GPT-4) play a role of "coordinator" and a motion/language translator fine-tuned from a lightweight language model (LoRA). The experiments (including the ablation study) are thorough, and the results (including the demos) demonstrate excellent performance.

From my perspective, the reason for achieving this performance isn’t due to improvements in the language-motion model itself, but is because the coordinator achieves the high level of understanding, decomposing, and recording the tasks before language-motion model, significantly enhancing the results (or the user’s perceived experience) without change to the existing data or methods. This is very interesting and smart (though perhaps slightly tricky). I am in favor.

strengths:
+ Very interesting idea and very reasonable design.

+ Thorough experiments and good performance.

+ Good writing.

weaknesses:
- I am very curious about the long motion sequence generation. As illustrated by authors, "By decomposing descriptions of long motions into a series of short motions using LLMs and subsequently concatenating these short motions into longer sequences, our Motion-Agent
can theoretically achieve infinite motion generation." Though I can generally understand the meaning, the details are unclear. Is "decomposition" done by GPT4 during long motion seq generation? If so, how it is achieved? Could authors provide detailed output of GPT4 of an example (like Figure 3)? Meanwhile, during training, will GPT4 be used to decompose HumanML3D data into shorter atomic units? If so, how does such labeling achieved and how to guarantee the alignment between motion and text?

- An important work is missed in related works as one of the first works that quantize/tokenize motion into GPT: 
Bailando: 3d dance generation by actor-critic GPT with choreographic memory, CVPR 2022;
Bailando++: 3d dance GPT with choreographic memory, TPAMI 2023

questions:
see above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730710230396

thread_type:
official_review

<thread_object>
creator: reviewer_dbek

<content>
summary:
In this paper, the authors propose Motion-Agent, a conversational framework for human motion that utilizes large language models (LLMs). By incorporating MotionLLM, a generative agent fine-tuned with adapters, Motion-Agent enables bidirectional motion-text translation and supports multi-turn conversational interactions for tasks such as generation, captioning, and editing. The framework leverages the HumanML3D and KIT datasets to demonstrate its effectiveness, achieving competitive results by integrating GPT-4 for coordination without additional task-specific tuning.

strengths:
1. The paper introduces Motion-Agent, a conversational framework that leverages pre-trained large language models (LLMs) for bidirectional human motion generation and understanding, achieving competitive results across various motion-language tasks.

2. The authors demonstrate Motion-Agent’s effectiveness through an extensive evaluation of the HumanML3D and KIT datasets, highlighting the model's competitive ability to generate and caption human motion sequences.

weaknesses:
1. The proposed Motion-Agent framework seems to focus primarily on conversational motion generation, a capability that, according to the authors, could also be achieved using additional datasets for task-specific instruction tuning. While the authors assert that Motion-Agent is efficient, the experiments presented offer insufficient evidence to substantiate this claim.

2. The authors are encouraged to expand the discussion on the potential advantages of the proposed Motion-Agent framework. For instance, how does the model address out-of-domain motion concepts in comparison with current methods? A more thorough analysis of the model's generalization capabilities would contribute to a deeper understanding of its overall effectiveness.

3. The authors claim that Motion-Agent can theoretically achieve infinite motion generation. However, plots or tables illustrating changes with increasing conversation turns and motion lengths, should be provided to substantiate this claim.

4. The paper would benefit from ablation studies on the motion tokenizer, as well as comparisons with state-of-the-art RVQ-VAE models, such as MoMask which also employs RVQ-VAE to convert motion into a discrete representation.

5. The paper lacks comparisons on additional MotionGPT benchmarks, such as motion composition tasks. Tuning MotionLLM with these task is something that can be easily done.

questions:
1. Did the authors attempt to fine-tune all parameters of the language models within MotionLLM? If so, what were the resulting outcomes, and how did they compare to the model tuned with LoRA?

2. Is it feasible to extend the Motion-Agent framework to incorporate vision and audio modalities, given that GPT-4o is known to support multimodal inputs? What impact might this integration have on the model's performance and potential applications?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732206833745

thread_type:
official_comment

<thread_object>
creator: reviewer_omaw

<content>
title:
No author responses?

comment:
Hi, all. 
I received an email to check the author's responses, but I cannot find any of this paper. Can anyone confirm that authors did not submit them?
Best
</content>
</thread_object>

</thread>
<thread>
creation date:
1732250102903

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We thank all reviewers for their insightful comments. We refer to dbek, 7jDa, DEqA, QaCH, and oMaw as R1, R2, R3, R4 and R5, respectively.

We would like to first reiterate some key aspects of Motion-Agent that may address some of the reviewers' questions. The primary goal of Motion-Agent is to efficiently handle complex and flexible motion-language tasks through multi-turn conversations using LLMs. LLMs such as GPT-4 do not generate or understand human motions, so we employ a translation agent, which is our *MotionLLM*, to bridge this gap. MotionLLM enables bi-directional generation, which can support the Motion-Agent framework for the generation and understanding tasks. Among existing models capable of both motion generation and motion captioning, MotionLLM delivers state-of-the-art performance.
MotionLLM is built upon a lightweight LLM and fine-tuned using LoRA, thus avoiding the need for pre-training from scratch. It is then integrated with another LLM (e.g., GPT-4) for conversational tasks, without requiring additional training. Then GPT-4 and MotionLLM work hand-in-hand, where GPT-4 interprets user requests and decomposes them into plans consisting of simple calls to MotionLLM. The generated motions can be concatenated to produce significantly longer sequences. By breaking down complex tasks, such as editing, composition, or question answering, into manageable components for MotionLLM, our framework achieves remarkable efficiency without requiring task-specific data or further training.

In the following, we provide additional experiments requested by reviewers.

**Ablation study on different tokenizers (R1, R3, R5)**

We conducted additional experiments replacing our current VQ-VAE tokenizer with the RVQ-VAE tokenizer used in MoMask. RVQ-VAE differs from traditional VQ-VAE by using $Q+1$ ordered codebooks. The first codebook is used for base tokens, similar to traditional VQ, while the remaining $Q$ codebooks are used to represent residuals for enhancing fidelity.

Following MoMask's approach, the MotionLLM in our case is aware of the base layer tokens. To predict the residual layer token sequences, we follow MoMask to use an additional non-autoregressive (NAR) transformer that takes as input the base layer token sequences from the LLM output. This NAR transformer can thus be considered as part of our detokenizer, which does not affect the LLM’s inference or training process. 

The results are shown as follows:
|Models|Trainable Params|Top 1 R Precision ↑|Top 3 R Precision ↑| FID ↓| Multimodal Dist ↓| Diversity ↑|
|-|-|-|-|-|-|-|
|Gemma2-2b R=32 VQ|41.5M|0.415|0.750|0.712|2.938|11.251|
|Gemma2-2b R=64 VQ|83.1M|0.422|0.762               | 0.658  | 2.929             | 11.195      |
| Gemma2-9b R=32 VQ          | 108M             | 0.439               | 0.776               | 0.438  | 2.872             | 11.151      |
| Gemma2-2b R=64 RVQ         | 83.1M + 13.4M    | 0.429               | 0.768               | 0.647  | 2.857             | 10.126      |
| Gemma2-2b Full VQ          | 2697.4M          | 0.423               | 0.774               | 0.591  | 2.913             | 11.138      |
*More Comparisons on KIT-ML.*

Our results show that using RVQ-VAE does not significantly improve performance compared to the original VQ-VAE, validating that the VQ-VAE model is already sufficiently effective for our tasks. While RVQ-VAE provides some potential improvements in fidelity, it introduces additional training overhead (the 13.4M parameters are from the NAR transformer). Specifically, the NAR transformer required for residual token prediction is challenging to train and adds complexity to the system (namely, requiring an additional training step for the NAR transformer). 

Notwithstanding, we believe that both VQ-VAE and RVQ-VAE are effective for motion tokenization, and that the choice between them is more of an engineering decision based on the specific trade-offs involved. Our Motion-Agent framework remains flexible and can support different tokenizers, as long as they can convert motions into discrete representations. Future advancements in tokenization methods may further enhance the framework's performance and capabilities.

**Ablation study on LoRA vs Full fine-tuning (R1)**

We also experimented fine-tuning all the parameters of LLM, the corresponding result is also shown in the table above. When using a 2b backbone, full fine-tuning does improve overall scores but at the costly expense of significantly increasing training overhead, which is in stark contrast to  LoRA fine-tuning a 9b backbone model which requires over 20 times fewer parameters trained. Fully training such a large model also demands substantially more computational power and memory, making it nearly impossible on consumer-level GPUs. This result indicates that LoRA is indeed a more efficient and effective approach than full fine-tuning, suggesting it is a better alternative for scaling up to a larger backbone in the future.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732733909564

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revision

comment:
We thank the reviewers and have addressed their suggestions in the revision by:

- Adding citations for Bailando and Bailando++.
- Including the additional ablation studies.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The submission introduces a framework for human motion modeling based on conversations.  Reviewers were in general positive about the submission, appreciating the novel idea and the good writing.  The rebuttal was also helpful in convincing the reviewers to level up their support.  Reviewer DEqA was less positive, though their review is low-quality, and they did not engage in the discussion.  The AC agreed with the majority and recommended acceptance.