Motion-Agent: A Conversational Framework for Human Motion Generation with LLMs
Authors: Qi Wu
Yubo Zhao
Yifan Wang
Xinhang Liu
Yu-Wing Tai
Chi-Keung Tang
Venue: ICLR 2025 Poster (2025)
Keywords: 3D human motion
multimodal LLM
motion generation
conversational AI
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: While previous approaches to 3D human motion generation have achieved notable success, they often rely on extensive training and are limited to specific tasks. To address these challenges, we introduce **Motion-Agent**, an efficient conversational framework designed for general human motion generation, editing, and understanding. 
Motion-Agent employs an open-source pre-trained language model to develop a generative agent, **MotionLLM**, that bridges the gap between motion and text. This is accomplished by encoding and quantizing motions into discrete tokens that align with the language model's vocabulary. With only 1-3% of the model's parameters fine-tuned using adapters, MotionLLM delivers performance on par with diffusion models and other transformer-based methods trained from scratch. By integrating MotionLLM with GPT-4 without additional training, Motion-Agent is able to generate highly complex motion sequences through multi-turn conversations, a capability that previous models have struggled to achieve.
Motion-Agent supports a wide range of motion-language tasks, offering versatile capabilities for generating and customizing human motion through interactive conversational exchanges.

Published as a conference paper at ICLR 2025
MOTION-AGENT! A CONVERSATIONAL FRAMEWORK FOR HUMAN MOTION GENERATION WITH LLMs
Qi Wu': Yubo Zhao's Yifan Wang', Xinhang Liu', Yu- WinWing Tai,, Chi-Keung Tang' The Hong Kong University of Science and Technology
2 Dartmouth College
ABSTRACT
While previous approaches to 3D human motion generation have achieved no- table success, they often rely on extensive training and are limited specific tasks. To address these challenges, we introduce Motion-Agent, an efficient conversational framework designed for general human motion generation, edit- ing, and understanding. Motion-Agent employs open-source pre-trained lan- guage model to develop a generative agent, MotionLLM, that bridges the gap between motion and text. This is accomplished by encoding and quantizing motions into discrete tokens that align with the language model's vocabulary. With only 1-3% of the model's parameters fine-tuned using adapters, Motion- LLM delivers performance on par with diffusion models and other transformer- based methods trained from scratch. By integrating MotionLLMM with GPT- 4 without additional training, Motion-Agent is able to generate highly com- plex motion sequences through multi-turn conversations, capability that pre- vious models have struggled to achieve. Motion-Agent supports wide range of motion-language tasks, offering versatile capabilities for generating and cus- tomizing human motion through interactive conversational exchanges. Project page: httpss//knox/hhoogiiuub/o///oo
INTRODUCTION
Large Language Models (LLMs) have recently attracted much attention in both industry and academia. Many LLMs, such as GPT-4 (Achiam et al. 2023). LLaMA (Touvron al.] (023). Gemma (Team et al. 2024a), have shown their advanced capabilities, robustness and generaliza- tion across various downstream tasks. These progresses have motivated researchers explore the application of LLMs in multimodal tasks, integrating them with modalities such as images (Koh) al. 2024). videos Zhang et al. (2023a), audio (Borsos et al. [023] Huang et al., 2023), and more, resulting in promising outcomes in understanding these different modalities. However, the utiliza- tion of LLMs in the context of multimodal generation, particularly of 3D human motion, remains underexplored, which is crucial for advancing robots and humanoid applications.
Research in 3D human motion has explored various language-related tasks, including text- conditioned motion generation (Zhang et al. 2023b Guo et al., 2022a Tevet al. 2023 Zhang ct al. 2022; Shafir et al. 2024 Guo et al. 2024; Jiang et al. 2024b), motion captioning (Guo et al, 20226 Jiang al. 2024b), motion reasoning (Edo et al. 2023 Jiang et al. 2024c). However, existing methods often require extensive training, leading to high computational demands and in- efficiency. These models typically trained task-specific data, making them data-dependent and limiting their ability to generalize across diverse scenarios. They also struggle with handling long, complex prompts with performance degradation. Furthermore, most existing models lack the capability to support multi-tum conversational interactions, thus limiting both the generation and refinement processes, and restricting the ability to create dynamic, interactive systems that can seamlessly generate and allow editing motions through dialogue.
Moving forward with the most recent LLM and MILM development, in this work, we propose Motion-Agent, multimodal framework that leverages the generalization and flexibility of pre-
"Equal contribution.

---

Published as a conference paper at ICLR 2025
Users person doing the gyn? the type of the exercise and what are the training goals?
cont'd
Users Mell, I don't want the person to do push-ups now, make the person do squats instead? Motion-Agents Aegents
Notion-Agents Agents The person is running on treadnill. Running treadrill is prinarily focuses on cardiovascular fitness. training goal can vary but often includes isproving heart health, increasing stanina and endurance, losing weight, or preparing
aerobic exercise
After the exercise, the is tired and praceeds to sit down to rest. Notion-Agents
conpetitive running
Users Can you generate notion containing an exapple of both an aerobic exercise and anaerobic exercise? Nation-Agents
Users Replace squats with running in place, and I want the sit chair front instead ground. Notion-Agents
Users Explain the notion you have just generated in detail.
Mation-Agenté
sequence generated two exercises. First, the person perforss jurping physical jumping exercise where the jacks, which noved repetitively
body's the hands clap overhead, then quickly return to the sides. This exercise is aerobic, helping to increase Second,
improve cardiovascular the person does push-ups, which involve repeatedly raising and lowering the body using the arns while lying face This exercise anaerobic, focusing strongthoning upper body and core resisting body weight.
Figure 1: Multi-turn Conversation Between User and Motion-Agent. First Turn: Motion Un- derstanding; Second Turn: Motion Generation; Third Turn: Motion Understanding with Previously Generated Motion; Fourth Turn: Motion Editing; Fifth Turn: Continue Motion Generation; Last Turm: Motion Editing on Long Sequence. Note that all turns are continuous.
trained I.Mss. Central to the framework is our new generative agent, MotionL.MM the incorpo- ration of which eliminates the need for extensive pre-training by employing lightweight adapter- based fine-tuning of a pre-trained LLM. Unlike MotionChain (Jiang et al. 2024c), which requires pre-training and large datasets for extensive instruction tuning achieve conversational control, Aoteon-Agent integrates MotionLLM with GPT-4 and leverages the LLM's inherent conversational capabilities without additional training. This enables efficient, customizable motion generation, un- derstanding, and multi-turn editing across various tasks.
In Motion-Agent, first train a pair of motion tokenizer and detokenizer. The motion tok- enizer encodes motions into motion embeddings and quantizes them into a set of discrete LLM- understandable tokens using codebook, while the detokenizer reconstructs tokens back to their original continuous forms. This tokenizer-detokenize pair enables the translation between continu- ous motion sequences and discrete tokens, facilitating interaction with the LLM while still allowing for the recovery of the original motions from the tokens. MotionLLM is trained by enriching pre-trained LLM's vocabulary with these additional motion tokens, while keeping the original text tokens unchanged. Given that motions can be represented as temporal sequences, our tokenization process converts motions into token sequences akin to sentences natural language. MotionI.MM translates between text token sequences and motion token sequences. On top of this, GPT-4 acts as a coordinator, decomposing user instructions to determine the number of calls to MotionLLM and how to structure those calls effectively. The resulting motion token sequences from multiple calls are concatenated and decoded by the detokenizer to produce the final output.
Our Motion-Agent framework leverages pre-trained LLMs in two key ways: (1) fine-tuning lightweight LLM via adapters to serve as a text-motion translation agent, and (2) using an LLM for conversational interactions without training, thus facilitating multi-turn dialogue for refining
2

---

Published as a conference paper at ICLR 2025
generated motions and producing extended motions by iteratively generating and concatenating quences. Despite training only a small number of parameters, MotionLLM can achieve competitive results in motion generation (text to motion) compared to those trained-fromm from-scratch models with specialized architectures. In motion captioning (motion to text), MotionI.LMM achieves state-of-the- art performances, generating semantically accurate and contextually appropriate text descriptions. MotionLLM enables bidirectional translation between text and motion, outperforming other au- toregressive models while using fewer trainable parameters, making an ideal fit for the overall Motion-Agent framework. By combining Mo with GPT-4, Motion- Agent enables versa- tile dialogue- based motion generation and reasoning, without requiring specific datasets or extra training for these tasks.
To summarize, our contributions include:
We introduce simple, efficient conversational framework, Motion-Agent, that utilizes pre- trained LLMs and produces strong results in various motion-language tasks.
We demonstrate the flexibility and versatility of our method by achieving highly customiz- able
tasks, including long and complex motion generation, multi-turn editing, and multi-turn reasoning.
2 RELATED WORK
Multimodal LLMs Recent advancements have integrated large language models (LLMs) with mul- tiple modalities such as image, video. music, audio, and point cloud using different approaches (Liu jtt al., 2024; Hann et al. 2024] Wu et al.] [023b] [Chnn et al.) 2023a; Gao al.) 2023). Various approaches have been proposed to align different modalities. For instance, Video-LLaMe (Zhang cet al. 2023a) leverages Q-formers to bridge the gap between modalities. PointLLM (Xu et al, 2023b) utilizes projector to align the feature space of point clouds with the feature space of the LLM. VALLE-X /Zhang et al. (023d) and LlamaGen (Sun et al., 2024) tokenize inputs from var- ious modalities connect them with language. On the other hand, emerging research (Wu al., 2023a Lu et al.) 2024 Du & Kaelbling 2024 Liu et al., (2025) demonstrates promising results with compositional language models. These models, often composed of smaller specialized components, excel in data efficiency and perform well on unseen distributions, aligning with the design of our framework.
3D Human Motion Synthesis Modern works can generate human motions based on a variety of
inputs such as action labels (Petrovich al. 2021: Lee al. 2023] Guo et al., 2020 Xu et al.
2023a), textual descriptions (Jiang et al. 2024b] Wang et al.) 2023] Zhang et al., 2023b Guo et al.
2022b Zhou et al., [202]] Tevet et al.] 2023] 2022 Guo et al. 2204; Zhang et al. [002]] Dabral
et al. 2023 Petrovich et al. [022]] Zhang al. 2023c Pinyoanuntapong
al. (024), control sig-
nals (Xie et al. 2024; Wan et al. 2023) Petrovich et al. 2024 2024|Huang et al. 2024 Goel et al.
2023
music audio (Dabral et al. 2023; Tseng et al. 2022; Zhou & Wang 2023 Siyao et al. 2022
2023). and others (Zhong et al. 2024). Particularly, text- text- guided 3D motion generation or text-to-
motion has garnered significant interest. Notably, some diffusion models have emerged powerful tools, such as Tevet et al. (2023); ; [Shafir et al.] (2024); Wang et al. (2023); Zhou et al. (2023); Xie al. (2024): Zhang al. 20022. Despite the proficiency generating motions, diffusion models necessitate manual length control of the generated motions with limited flexibility. In addition to diffusion models, which employ continuous motion representation, discrete token-based methods utilizing Vector Quantized Variational Autoencoders (VQ-VAEs) have also demonstrated promis- ing results. Notable examples include TM2T (Guo et al. (0226), T2M-GPT (Zhang et al., 2023b), MotionGPT (Jiang et al. (024b) and MoMask (Guo et al.] 2024). Most existing works in both approaches focus on conditional generation translate between modalities. our work, we em- phasize generating human motion through complex, customized user conversations while proposing a training-efficce -fffiiient approach to bridge these modalities using pre-trained LLMs.
Conversational Control For Human Motion Generating 3D human motion through conversation is more flexible, which allows users to customize versatile requests and control motion via iterative refinement. While models like MotionGPT (Jang al. 2024b) handle some simple single-turn tasks using instruction tuning, and MotionChain (Jiang et al. (022cc) supports multi-turn interactions by sampling single-turn data into multi-turn training data, both methods rely heavily on extensive
3

---

Published as a conference paper at ICLR 2025
person jumps down, then be runs in circle. after that he du backllips.
the job man
doing?
input notion_ file
GPT4
Mo
Motion Tolumier
Test Tokenia
1ist coliioaal commands to call
</>
NationL www.onnn....nnnn...eememmm generatee weneratee""essriitttttee ""eee""rprppoiion..... MotsonL captiii (notion, File)
MotionyTeext tokens
Mo
Mixed
LLM+LORG
Generated
Motlon Detokentoer
Text Detokentor
takens
The probably sportsman
Figure 2: Motion-Agent pipeline. GPT-4 can interact with the translation agent (i.e., MotionILLM) to generate or interpret motions based on input requirements. The generated motion tokens are con- catenated and decoded, and the textual caption produced by MotionLLM returned and processed by GPT-4.
instruction tuning and additional data. In contrast, our Motion- framework uses a composition of LLMs to eliminate extra training. By training the translation agent, MotionLLM, solely on the original text-motion paired data, our method eliminates the need further data or training, resulting in higher efficiency and broader generalizability.
METHOD
As shown in Fig. [ our Motion-Agent framework primarily consists of three components: an LLM (i.e., GPT-4) for conversational interaction and prompting control, pair of motion tokenizer/deto- kenizer, and a translation agent (i.e., MotionLLMM). The text tokenizer is inherited from the LLMs and remains unchanged, while the motion tokenizer and detokenizer proper reconstruction of motion sequences. Once trained, the motion tokenizer and detokenizer are kept fixed. The motion detokenizer plays key role in smoothing the transitions between different motion sequences, ensuring scamless integration of motion outputs.
trained together to ensure
To ensure bidirectional understanding, our framework also enables motion comprehension. Thus, the agent should also be capable of generating textual captions from given motions upon request. This bidirectional translation is crucial for applications such as answering questions about motions or generating descriptions, where models that can only perform motion generation are not suitable, On the other hand, our proposed MotionI_LM can indeed good fit, ensuring bidirectional translation within unified architecture.
3.1 THE MOTION- AGENT FRAMEWORK
In this framework, GPT-4 serves as the coordinator of both motion generation and comprehension, enabling seamless interaction between users and a multimodal text-motion agent. The agent is responsible for translating between text and motion modalities. Within the conversation, the input to GPT-4 consists of two components: a fixed instruction prompt p. which provides guidelines for interacting with the text-motion agent, and a customized request from the user. Based on this input, GPT-4 generates a structured plan, determining whether the agent should perform tasks such motion generation captioning. It also decides how many times invoke the agent and specifies the arguments for each invocation. This plan, formatted as a JSON file, is then parsed and executed by the agent to carry out the specified tasks.
4

---

Published as a conference paper at ICLR 2025
For generation, the agent generates motion token sequences corresponding to each set of arguments, and these sequences are concatenated for universal decoding. Specifically, let G represent the agent and |a,] the arguments for each of the N calls determined by GPT-4. The resulting motion token sequences 7; = ((a:) are concatenated to form single sequence z = (Z1. 22, ZN). This sequence is decoded by the decoder D to produce the final motion, Dzz), will be outlined in the tokenization section.
For motion understanding and reasoning. the agent generates textual captions of the motions, which are then returned to GPT-4. This allows GPT-4 to interpret the motion and respond to user queries accordingly, enabling seamless interactions between users and the system through both motion gen- eration and comprehension.
Since LLMs such as GPT-4 possess strong multi-turn conversational abilities, users can continuously ask the model to refine, edit, or extend previous generations, as well as pose additional questions. In response, GPT-4 will re- r--geerrate the plan or provide answers, thus providing an interactive and adaptive system. This dynamic interaction leads to unified framework that supports exception- ally wide variety of combinations, lengths, and task complexities, offering enhanced flexibility and customization across both motion generation and comprehension.
3.2 MOTION TOKENIZATION
In order to align better with LLM's next-token prediction mechanism, we tokenize motions into discrete representations using Vector Quantization (VQ) and Variation AutoEncoders (VAE). This VQ-VAE approach is widely adopted by Guo et al. (002b)), (Siyao et al. (2022) Siyao et al. (2023)
Zhang et al. (2033b). Jiang ct al. (004b)) and Guo et al. (2024).
In our motion tokenization, motion sequence is represented mI.T E R and is first encoded using an encoder E to motion embeddings 1::7/N E R///xdd where N is the downsampling rate and d is the number of the hidden dimensions. Then the motion embeddings are quantized by quantizer using codebook C {K where K is the codebook size and each E IRd. The quantization results can be represented as 11.T/N, where
i. = arg min ||z, C2ll2
&EE
The original sequence can be reconstructed by the decoder D: m..T D(
We follow Zhang et al. (2033)) to optimize the VQ-VAF, using reconstruction loss together with a commitment loss. We also add additional regularization on the joint positions p enhance the generation performance. The loss can be formulated as:
Lug
m
+
+33 1| - sg[z]lla
Cre
E.
Commat
where sgl:] is the -ooddena operation, a and B are weighting factors. The codebooks are trained using exponential moving average (EMA) and codebooks reset following T2M-GPT (Zhang et al.) 2023b). After training, the tokenizers are frozen for further usage.
3.3 LLM-BASED MOTION-LANGUAS AGENT
Following tokenization, the motion representation is discretized into K distinct motion tokens. We utilize the indices of these motion tokens from the codebook construct the motion token vocab- ulary Vm = {< Motion i > } In addition, we introduce special tokens "<Mot iMMttion>" onn" and "</Mot on>" to denote the start and end of a motion token sequence. These special tokens, to- vocabulary set Va of size K + 2. This vocabulary will
gether with the motion tokens, form then be appended to the pre-trained LLM's vocabulary.
After expanding the LLM's vocabulary, motion can now be denoted as token sequence that is un- derstandable by the LLM. During the generation process, the LLM predicts the succeeding token by maximizing the probability Po( x c). where Z1:7 is the target token sequence and represents the prompt. This prediction is performed iteratively in an autoregressive manner. Consequently. the training objective aims to maximize the log-likelihood LILM Σ log po pa( c))
5

---

Published as a conference paper at ICLR 2025
Methods
Motion Generation
Captioning Multi-turn Editing
Reasoning
Composition
MotionGPT Jiang et al. 2024b
short
x
MoMask
Luo et al. 422
short
MotionChain
Juing et at,
2024c
short
Ours
long
Table 1:
Comparison on functionalities among recent motion generation models. Italicized model indicates the corresponding model requires pre-training and task-specific tuning.
During the inference process, approach utilizes instructive prompts such as "Generate motion that matches the following input human motion description." accompanied by a sentence describing the desired motion. The LLM then proceeds predict tokens autoregressively until it predicts the "< "</Mot on>" token, indicating the completion of the motion generation. This autoregressive pro- cess allows for the generation of motions with variable lengths, adapting the specific requirements of the given description.
To fine-tune the LLM, we employ LoRA (Hu al. 2021). Throughout the whole training process, the tokenizer, the embeddings, and the output layer of the original text tokens remain unchanged and frozen. Only the additional adapters are trained. These LoRA adapters are trained for the task at hand (generation or captioning) while maintaining general architecture where multiple adapters can coexist harmoniously. This approach allows to leverage the power of LLMs while tailoring them to specific motion-language tasks, ensuring efficient and effective training without altering the core components of the LLM.
4 EXPERIMENTS
We assess our Motion-Agent framework with general and complex conversational user inputs, demonstrating its ability to handle intricate, multi-turn interactions. We also evaluate MotionLLM on single-turn motion generation and motion captioning tasks.
4.1 EXPERIMENT SETUP
Datasets. Our experiments on MotionLLM conducted with KIT Motion Language Dataset (KIT-ML) (Plappert et al. 2016). HumanM1.3D (Guo et al.) 2022a). KIT-MI contains 3,911 human motion sequences, while HumanML3D dataset, obtained from AMASS (Mahmood et 2019 and
HumanAct12 (Guo et al. 2020), contains 14,616 human motions sequences with 44,970 textual
descriptions. For Motion Motion-Agent, Agent, we use the MotionL.LMM model which is trained on HumanML.3D.
Evaluation Metric. For motion generation, we follow T2M (Guo et al., 2022a). Global represen- tations of motion and textual descriptions are first extracted with the pre-trained network in (Guo) et al. 2022a) and then measured in the following: 1) Text matching: R-precision (Top-1, Top-2.
and Top-3 accuracy) by ranking Euclidean distances between motion and text embeddings. and MM Dist, which measures the average distance between text and generated motion embeddings. 2) Gen- eration diversity: quantifies the variance of generated motions across all descriptions. 3) Motion fidelity: FID assesses the distance between the distribution of real and generated motions, reflecting how closely they match real motion distributions. For motion captioning, we follow TM2T (Guo) 2022b) to evaluate the quality of motion captioning by facilitating linguistic metrics from nat- al.
ural anguage studies, including Bleu (Papineni et al. 2002), Rouge (Lin 2004), Cider (Vedantam
al 2015). and Bert Score (Zhang et al. 2020).
Implementation Details. We utilize GPT-4 (Achiam et al. 2023) as the conversational LLM in our Motion-Agent framework, which offers enhanced textual control and interaction capabilities. In our tokenizer, we set the downsampling rate N to 4, the hidden dimension d to 512, and the codebook size K to 512. The weighting factors and B for Cp and Coommit are set to 0.5 and 0.02 respectively. For MotionLLM, we employ Gemma2-26 2b-it (Team et al. 2024b). a lightweight open-source LLM from Google, which offers accessibility and can be deployed LoRA rank is set to 64 for generation and 32 for captioning, the values of alpha remain the same with the rank. All of our experiments are conducted NVIDIA RTX4090s.
single consumer-level GPU. The

---

Published as a conference paper at ICLR 2025
Users Generate another person down and then stands up to fight back by slapping and kicking. Notion-Agents
Users Generate a notion of a person perforning a Floor exencise in artistic gymastics, and nake it long. Notion-Agents
Users A Football player is doing the following celebration Can who is?
Notion-Agents Based on the given goal celebration notion involving neditation, the likely Erling Haaland. reditation celebration has become one of this iconic goal celebration.
Figure 3: Motion-Agent can comprehend abstract, complex user prompts and generate accurate, long motions. It also understands and answers questions based real-world knowledge. No- tably, the three turns in this figure stem from a continuous conversation, demonstrating the flexibility of its multi-tum capability in scenarios that should not be influenced by previous turns.
Users Generato a notion where a golfer hits the ball, runs to the hole to check, and then colebrates by junping and waving hands.
NotionGPT:
Motion- Agents
Figure 4: Comparison with Other Methods. Our Motion-Agent accurately generates motions involving a series of actions, while other models struggle with more complex descriptions like this, resulting in short and unclear motions.
4.2 RESULTS OF MOTION-AGENT
In this section, we present the results of our Motion-Agent framework, demonstrating its ability to generate long outputs through complex combinations of tasks via multi-turn conversations. important note that established ground truth exists for such tasks, aside from text-motion translation, where we do not conduct additional training for these extended tasks.
As shown in Table Motion-Agent is proficient in various motion-language tasks, generating long motion sequences through natural conversational user interactions. MotionGPT (Jiang et al. (024b) supports bidirectional translation but lacks versatility, while MoMask (Guo et al. (2024) excels in generation but is limited to this task. Although MotionChain (Jiang et al.) (024c) can perform similar functions, requires additional datasets for task-specific instruction tuning. These methods, along with most existing approaches, are restricted to relatively short motion sequences. In contrast, without training on additional datasets, our Motion-A Agent can generate longer sequences, accurately matching the given prompts, as indicated in Figure 4 While HumanMI.3D (Guo et al. 2022a) contains a wide range of human motions, sequences are generally short and atomic, lasting less than 10 seconds. By decomposing descriptions of long motions into series of short motions using LLMs and subsequently concatenating these short motions into longer sequences, our Motion-Agann Agent theoretically achieve infinite motion generation. This decompose-an ann-oss--nndnntgrrttee approach can thoroughly leverage existing data for long motion generation, mapping known data distributions to unknown ones and enhancing both efficiency and scalability.
The integration of LLMs also improves the system's 's ability to interpret vague, abstract, or complex motion descriptions, allowing for iterative refinement through multi-turn conversations. Figure

---

Published as a conference paper at ICLR 2025
Users I will provide two notions, I'd Like you to generate notion that facilitates smooth transition between then.
Direct Concatenations
Transition by Notion-Agents
Motion1
Motion2
Figure 5: Motion-Agon can compose motions with smooth transitions. this example, the two motions "a person falls down the back" and "a person is walking" are provided to Motion-Agent in two turns. The system then generates a "stand up" motion to facilitate a seamless composition of the two motions.
already illustrates our framework's 's strong multi-turn contextual capabilities, enabling it to under- stand, extend, and edit the results of previous turns effectively. Additionally, our multi-turn func- tionality facilitates non-contextual requests, evidenced by the results in Figure 3 which were generated within a single conversation comprising multiple turns. This flexibility allows users to avoid restarting for new requests. Furthermore, the results in Figure demonstrat that our method can accommodate general, customized, and complex user requests through conversational and iter- ative exchanges. Our Motion-Agent is also capable of generating transition motions to connect and compose movements seamlessly, as shown Figure | an ability that previous motion generation models struggled to achieve. This further demonstrates the motion understanding and generation capabilities of our method.
More qualitative results are presented in the appendix A.1.1 and the supplementary material.
4.3 EVALUATIONS OF MOTIONLLM
We evaluate MotionLLM on both -oxtton-motion and motion-to-Ee teeton-to-text tasks validate that it achieves satisfactory results. MotionLLM is focused on enabling bidirectional translation with minimal train- ing load, while still maintaining competitive performance across key benchmarks. Quantitative re- sults are shown in Table
For generation, we compare our model with state-of-thh-arr (SOTA) approaches, including di diffusion models (Tevet et al. 2023: Chen et al. (023b Zhang et al. (022) and token-based models (Zhang) ctt al. 2023b: Jiang et al. 2024b; Guo al. 2024). Despite fine-tuning only a small number of parameters, model performs competitively against these models trained from scratch. This demonstrates our advantages of leveraging the generalization and robustness capabilities of LLMs. Additionally, our model exhibits low MMDist, high R Precision and high Diversity, indicating strong motion-language understanding and generative capabilities. Note that MoMask (Guo et al., 2024) and the diffusion models are non-autoregressivv, requiring known target lengths for generation, and evaluate using ground truth lengths. However, since the FID metric measures the distance between the distribution of generated results and ground truth, variable length generated by autoregressive models can lead to higher FID scores. Yet, our MotionL achieves a lower FID than some other autoregressive models such as MotionGPT (Tiang et al. 2024b) with only about one-third of train- able parameters. Additionally, the autoregressive nature of our model offers advantages over non- autoregressive models when ground truth motion lengths are not provided. This makes MotionLLM better fit for our Motion-Agent framework, as it eliminates the need for specifying motion lengths.
In Sec. we provide further analysis and evidence that increasing the model size can lead to overall improvemens performance scores. For more economical choice, selected one of the smallest LLMS (Gemma2-25) available to the public.
For captioning, we compare models capable of bidirectional generation. Leveraging the strong text processing capabilities LLMs, MotionLLM produces accurate descriptions of human motions. We assess the generated captions using linguistic metries from Guo et al. (2022b), which calculate semantic similarities to ground truth captions. To ensure an accurate evaluation. we follow ]aang et al. (2024b) by using the unprocessed ground truth texts, Guo et al. (0022)) ignores gram-
8

---

Published as a conference paper at ICLR 2025
R Precision
Tasks
Methods
FID
MultiModal Dist
Diversity
Top
3
T2M Guo et al.
2022n
0.457 .002
0.740
1.067#.002
3.3410=.00 .006
9.188=.00
TM2T (Go0
2022b
0.424
0.72g
1.501
3,467 001
8. 8.589=-076
MDM
Tevet
2125
0
0.611
0.544
5. 5.506*-400
9.559
MLD hen et al
2023b
0.481 003
0.7724.uuz
0.473
3.196#.010
9.724 052
Generation
MotionDiffuse
T2M-GPT (Zhang et
ZZhang al.
20233
2023
0.491 =000
0.491 0003
0.775#.002
0.782#.um
0.116:t.004
0.630
3.118#.011
3.113#
9.410#.049
#0083
MotionGPT Duing
2024b
0.492
0.778
0,232#
3.096 2008
9. 9.528#-071
MotionChain Jiang al
20240
0.501
0.790
0,248
3.033
9.470#-075
MoMask Guo et al.
2024
0,521 2002
0.807 002
0.045
2.958 008
9.620
Mo
0.515
0.801
0. 0
2.967
9.908
Bleu@ 1
Blcu@41
Rouget
Cidert
Bert Score
TMZT [Guo et al..
2022b
48.90
8.27
15.80
32.2
Captioning
MotionGiP Drang cl 2024h
48.20
12.47
37.4
29.20
32.4
MotionChain Drang arai 20240
48.10
12.56
33.9
33.70
36.9
MoLionLLM
54.53
17.65
48.7
33.74
42.63
Table 2: Quantitative evaluation of MotionLLM on the HumanMI.3D (Guo et al.) 2022a) test
set. For motion generation, follow T2M (Guo et al., (222aa) for the evaluation metrics. The evaluations are conducted 20 times to obtain 95% confidence interval. Methods indicated italics utilize the ground truth lengths for estimation. Models above capable of bidirectional generation also included in the captioning evaluation. For motion captioning. we use the ground truth captions without pre-proces rre-processing and linguistic metrics suggested by Guo al. (2022b) for evaluation. Best scores are highlighted in boldface, while underscore refers to the second best.
matical tense and plural forms. As demonstrated in Tab. 2 our method outperforms previous SOTA approaches across all metrics by large margin, thanks to the language abilities of pre- trained LLMs.
4.4 ABLATION STUDY
Ablation on Motion-Agent Theoretically, the MotionLLM agent in our Motion-Agent frame- work can be replaced with any model capable of motion-text translation. However, models like MoMask (Guo et al. 2024). which require manual motion length input, may encounter issues (see A.1.2. making autoregressive models preferable. In this study, substitute MotionLLM with MotionGPT (Jiang et al. 2024b), which also supports bidirectional translation. After integrating with Motion-Agent, we observe that MotionGPT is capable of generating longer and more complex motions compared to its original implementation. However, it still falls short of the accuracy and smoothness achieved by using MotionLLM. For example (Figure 6). in the user prompt "A person lies face up to rest and then stands up after a while." GPT-4 decomposes this into two components: "lying face up for while" and "transition from lying face up to standing up." While MotionGPT correctly generates the first part, it incorrectly generates the second as "from lying face down." This results in an abrupt and unsmooth transition between the two motions. contrast, MotionLLM accurately generates both parts, ensuring smooth, seamless motion transition.
Users person 1ies Face up rest then stands up after while.
Notion-Agent (with MotionLLM)S
Notion-Agont (uith NotionEPT)S
Figure 6: Motion-Agent Ablation Study. We substituted MotionLLM with MotionGPT and no- ticed that MotionGPT cannot generate smooth motion transition.
Additionally, our framework can be adapted to use different LLMs for conversation. We tested sub- stituting GPT-4 with various models, including Llama (ouvron et al. 2023). Gemma (Team et al, 2024b), and Mixtral (Jiang al. 2024a). Most of these models successfully generated reasonable outputs and are capable of facilitating multi-turn interactions. Some smaller models may struggle with producing the correct JSON format. This ablation study demonstrated that our framework is

---

Published as a conference paper at ICLR 2025
applicable to all other LLMs. not just GPT-4. The performance of framework can improve alongside the development of LLMs. More details are in Sec A.2
Nonetheless, our framework can be summarized combination of larger LLM for conversation and motion-language translation agent, providing flexible choices for different components.
Ablation MotionLLM We conducted an ablation study examine the impact of different LLM backbones and adapter sizes. The results are shown in T from which we may conclude that using larger backbone models or increasing the LoRA rank leads overall improvements in the metrics. Additional ablation studies on different tokenizers and a comparison between LoRA and full fine-tuning are presented in Section A.6
R Precision f
Models
Trainable Params
FID
Multimodal Dist
Diversity
Top
Top 3
T2M-GPT Zhang 2023h
228.4M
0.416
0.745
0.514
3.007
10.921
MotionGPT
[Jaag
2024b
220M
0.366
0.680
0.510
3.527
10.350
Gemma2 2b R=16
20.8M
0.411
0.738
0.745
2,994
11.313
Gemma2-2h R=32
41.5M
0.415
0.750
0.712
2.938
11.251
Gemma2-22 R=64
83.1M
0.422
0.762
2.929
11.195
LLaMA3-8B R=32
0.381
0.737
0.646
3.046
11.210
Gemma2-99 R=32
108M
0.439
0.776
0.438
2.872
11.151
Table 3: More comparisons and ablation study on the KIT-ML (Plappert et al., 2016) dataset. Gemma (Tem al. 2024a) and LLaMA (Tourron et al. 2023) chosen LLM backbones. R indicates the LoRA rank, the value of alpha is kept the same with the rank. Two other autoregressvie transformer models are included for reference.
5 DISCUSSION
Limitations and Future Work. Our Motion-Agent specializes in generating motions of articu- lated 3D human body, without incorporating 3D visual understanding, such as interaction with the surrounding environment (e.g.. "a person puts his hand on the table"). Also, Motion-Agent does not include detailed hand or facial movements. Nonetheless, our framework demonstrates high flexibil- ity, making it well-suited to incorporate additional agents for handling these tasks in future exten- sions. Additionally, while we have conducted preliminary trials on multi- multi-human motion generation using Motion-Agen Aggett framework with some initial results (see Appendix (A.3) this has not yet been fully explored. Therefore, this paper still focuses on single-human motion generation. We left the extension for human-environment interaction and multi-humaa interaction for future work.
Concluding Remarks. In this work, we propose novel LLM-based multimodal, conversational motion-language learning framework, offering both flexibility and generalizability. By harness- ing the linguistic comprehension and generation capabilities of pre-trained LLMs, our Motion- L.LM achieves strong results in bidirectional translation between motion and natural language. The Motion-Agent framework is easily expandable across various tasks through conversational inter- actions. Our approach not only casy to train and adaptable but also user-friendly, making it versatile solution for motion-language learning applications. Motion-Agent offers comprehensive solution for enhancing LLMs' capabilities understanding, generating. and editing human motion, aligning with goal of teaching LLMs to interpret human motion effectively.
ACKNOWLEDGMENTS
This work supported in part by Dartmouth College A&S Startup fund and by the Research Grant Council of the Hong Kong SAR under neeme-based Research Scheme, grant no. T22-606/23R.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical
10

---

motion synthesis. In Computer Vision and Published as a conference paper at ICLR 2025
report. arXiv preprint 2rX1v:2303.08774,20000
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Shar- ifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. Audiolm: a language modeling approach to audio generation. arXiv preprint (rXiv:2209.031433 2023.
Feilong Chen, Minglun Han, Haozhi Zhao, Qingyang Zhang, Jing Shi, Shuang Xu, and Bo Xu. X-Ilm: Bootstrapping advanced large language models by treating multi multi-modalities as foreign languages. arXiv preprint arXiv:2305.04100 2023a.
Xin Chen, Biao Jiang, Wen Liu, Zilong Huang. Bin Fu, Tao Chen, and Gang Yu. Executing your commands via motion diffusion in latent space. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18000-18010, 2023b.
Rishabh Dabral, Muhammad Hamza Mughal, Vladislav Golyanik, and Christian Theobalt. Mo- fusion: A framework for Pattern Recognition (CVPR), 2023.
Yilun Du and Leslie Kaelbling. Compositional generative modeling: A single model all you need. arXiv preprint arXiv:2402.0/ 103, 2024.
Mark Endo, Joy Hsu, Jiaman Li, and Jiajun Wu. Motion question answering modular motion programs. In International Conference on Machine Learning, pp. 9312-9328. PMLR, 2023.
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu,
Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304. 15010, 2023.
Purvi Goel, Kuan-Chich Wang, C. Karen Liu, and Kayvon Fatahalian. Iterative motion editing with natural language. arXiv preprint 2. 1538, 2023.
Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. Action2motion: Conditioned generation of 3d human motions. In Proceedings the 28th ACM International Conference on Multimedia, pp. 2021 2021-2029, 2020.
Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1.52-5161, June 2022a.
Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng. Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts. In European Conference on Computer Vision, pp. 1800597..00 Springer, 2022b.
Chuan Guo. Yuxuan MU, Muhammad Gohar Javed, Sen Wang, and Li Cheng. Momask: Gener- ative masked modeling of 3d human motions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1900-1910, 2024.
Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. Onellm: One framework to align all modalities with language. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 1rXiv22106.09685, 2021.
Rongjie Huang. Mingze Li, Dongchao Yang. Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong. Jiawei Huang, Jinglin Liu, Yi Ren, Zhou Zhao, and Shinji Watanabe. Au- diogpt: Understanding and generating speech, music, sound, and talking head. arXiv preprint arXiv:2304. 12995, 2023.
11

---

Published as a conference paper at ICLR 2025
Yiming Huang, Weilin Wan, Yue Yang. Chris Callison-Burch, Mark Yatskar, and Lingjie Liu. Como: Controllable motion generation through language guided pose code editing. arXiv preprint arXiv:2403.13990, 2024.
Albert Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford. Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint orXiv:2401.04088 2024a.
Biao Jiang. Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as foreign language. Advances in Neural Information Processing Systems, 36, 2024b.
Biao Jiang, Xin Chen, Chi Zhang. Fukun Yin, Zhuoyuan Li, Gang YU, and Jiayuan Fan. Motionchain: Conversational motion controllers via multimodal prompts. arXiv preprint arXiv:2404.01700, 2024c.
Jing Yu Koh, Daniel Fried, and Russ R Salakhutdinov. Generating images with multimodal language models. In Advances in Neural Information Processing Systems, volume 36, 2024.
Taeryung Lee. Gyeongsik Moon, and Kyoung Mu Lee. Multiact: Long-term 3d human motion generation from multiple action labels. In AAAI Conference on Artificial Intelligence (AAAI), 2023.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics. URL http:////aaaaaaacl.....ooomo wwwp://-ogg//000000 1013
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. arXiv preprint arXiv:2402.08268 2024.
Xinhang Liu, Yu-Wing Wi-g Tai, and Chi-Keung Tang. Chatcam: Empowering camera control through conversational ai. Advances in Neural Information Processing Systems, 37:54483-54506, 2025.
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai- Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language mod- els. Advances in Neural Information Processing Systems, 36, 2024.
Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Mold Moll, and Michael J. Black. AMASS: Archive of motion capture surface shapes. In International Conference Computer October 2019.
Vision, pp. 5442-5451,
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318 2002.
Mathis Petrovich, Michael J. Black, and Gül Varol. Action- cedioncoonitiooned 3D human motion synthesis with transformer VAE. International Conference on Computer Vision (ICCV), 2021.
Mathis Petrovich, Michael J. Black, and Giil Varol. TEMOS: Generating diverse human motions from textual descriptions. In European Conference Computer Vision (ECCV), 2022.
Mathis Petrovich, Or Litany, Umar lqbal, Michael J. Black, Gül Varol, Xue Bin Peng, and Davis Rempe. Multi-track timeline control for text-driven 3d human motion generation. In CVPR Workshop on Human Motion Generation, 2024.
Ekkasit Pinyoanuntapong, Pu Wang, Minwoo Lee, and Chen Chen. Mmm: Generative masked motion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1556-1555 1555, 2024.
Matthias Plappert, Christian Mandery, and Tamim Asfour. The KIT motion-language dataset. Big Data, 4(4):236-252, dec 2016. doi: 20009///i.2216.0000 URL http::/tpx..xo/ doi.orii.g/gg/ooooooooo.oo.o 10. 2016. 0028
Yonatan Shafir, Guy Tevet, Roy Kapon, and Amit H. Bermano. Priormdm: Human motion diffusion as generative prior. In International Conference on Learning Representations, 2024.
12

---

Published as a conference paper at ICLR 2025
Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang, Chen Qian, Chen Change Loy, and Ziwei Liu. Bailando: 3d dance generation by actor-critic gpt with choreographic memory. Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11050- 11059, 2022.
Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang, Chen Qian, Chen Change Loy, and Ziwei Liu. Bailando++: 3d dance gpt with choreographic memory. IEEE Transuctions on Pattern Analysis and Machine Intelligence, 2023.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.00 06525, 2024.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, al. Gemma: Open models based on gemini research and technology. arXiv preprint 0rXiv:2403.08295 2024a.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.001 18, 2024b.
Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or. Motionclip: Ex- posing human motion generation to clip space. arXiv preprint arXiv:2203. 2022.
Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H. Bermano. Mdm: Human motion diffusion model. International Conference on Learning Representations, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023.
Jonathan Tseng, Rodrigo Castellon, and Karen Liu. Edge: Editable dance generation from music. arXiv preprini arXiv:2211. 2022.
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. arXiv preprint arXiv:1411.5726, 2015.
Weilin Wan, Zhiyang Dou, Taku Komura, Wenping Wang, Dinesh Jayaraman, and Lingjie Liu. Ticontrol: Trajectory and language control for human motion synthesis. arXiv preprint arXiv:2311.17135. 2023.
Yin Wang. Zhiying Leng, Frederick W. B. Li, Shun-Cheng Wu, and Xiaohui Liang. Fg- (2m: Fine-grained text-driven human motion generation via diffusion model. arXiv preprint arXiv:2309.06284 2023.
Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Vi- sual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671, 2023a.
Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multi- modal 1lm. arXiv preprint arXiv:2309.05519 2023b.
Yiming Xie, Varun Jampani, Lei Zhong, Deqing Sun, and Huaizu Jiang. Omnicontrol: Control any joint at any time for human motion generation. In ICLR, 2024.
Liang Xu, Ziyang Song, Dongliang Wang, Jing Su, Zhicheng Fang, Chenjing Ding, Weihao Gan, Yichao Yan, Xin Jin, Xiaokang Yang, et al. Actformer: gan-based transformer towards general action-condie coniiincndiiooned 3d human motion generation. ICCV, 2023a.
Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointllm: Empowering large language models to understand point clouds. arXiv preprint arXiv:2308. 16911, 2023b.
13

---

Published as a conference paper at ICLR 2025
Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction tuned audio-visual language model for video understanding. arXiv preprint arXivv2230..22: 02858, 2023a.
Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, and Xi Shen. T2m-gpt: Generating human motion from textual descriptions with discrete representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023b.
Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208. 15001, 2022.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evalu- ating text generation with bert. arXiv preprint arXiv:/904.09675, 2020.
Yaqi Zhang, Di Huang, Bin Liu, Shixiang Tang, Yan Lu, Lu Chen, Lei Bai, Qi Chu, Nenghai Yu, and Wanli Ouyang. Motiongpt: Finetuned Ilms are purrpss motion generators. arXiv preprint arXiv:2306. 10900, 2023c.
Ziqiang Zhang, Long Zhou, Chengyi Wang. Sanyuan Chen, Yu Wu, Shujie Liu, Zhuo Chen, Yan- qing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. Speak foreign lan- guages with your own voice: Cross-lingual neural codec language modeling. arXiv preprint 0rXi::2303.03926. 2023d.
Lei Zhong, Yiming Xie, Varun Jampani, Deqing Sun, and Huaizu Jiang. Smoodi: Stylized motion diffusion model. arXiv preprint arXiv:2407.12777 2024.
Wenyang Zhou, Zhiyang Dou, Zeyu Cao. Zhouyingcheng Liao, Jingbo Wang, Wenjia Wang, Yuan Liu, Taku Komura, Wenping Wang, and Lingjie Liu. Emdm: Efficient motion diffusion model for fast, high-quality motion generation. arXiv preprint urXiv:231..02256, 2023.
Zixiang Zhou and Baoyuan Wang. Ude: A unified driving engine for human motion generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5632-5641, 2023.
14

---

Published as a conference paper at ICLR 2025
A
APPENDIX
In the appendix, we present:
Section
A.1
More Qualitative Results.
Section
A.2
Ablation study different LLMs
Section
A.3
Preliminary Trials on Multi-human Motion Generation using Motion-Agent
Section
A
4
More details about the evaluation metrics.
Section
A.5
5
More details regarding our implementation.
Section
More Ablation Studies on MotionLLM
A.1
QUALITATIVE RESULTS
Rendered original videos of all examples shown in the paper can be found the corresponding folder of the supplementary material.
A.1.1 NOTION-AGENT
More examples of Motion-Agent presented in Figure 7 8 9 and corresponding videos can be found in the supplementary material.
Users what is the person doing?
Motion-Agents parson sindcking chicken by putting their hands their arspits. Userš you have any ideas why doing like this?
notion-Agents The person could be acting like chicken for various reasons such as performing in play, entertaining children, participating in other possibilities for this notion besides siaicking chickan?
contest, or siaply having fun in husorous or light- situation.
Users Are there specifåc description the notion with hands placed under arspits pritarily suggests ninicking
Not iot-Antents a chicken. However, it's possible, though less likely, that this notion could be used in other creative or hunorous contexts not directly related initating chicken, such unconventional dance soves part of physical routine.
Userd
difforence between notion previous
Notion-Agents The differonce between this notion and the previous ane is that in this notion, the person explicitly raises both ares and flaps then like bird, which suggests broader initation of bird's flying notion. In notion involved placing under ampits acting like chicken, which is rather outright flapping.
specific chicken behavior and includes minickingg wings
Figure 7: More examples of Motion-Agent.
A.1.2 MOTIONLLM
Motion Generation Figure 10 presents the comparison on no-length given motion generation. More qualitative results are in the supplementary material.
Motion Captioning Our model is capable of generating high-quality motion captions, demonstra- tions provided in Table 4
15

---

Published as a conference paper at ICLR 2025
Users possible scenarios for this nation.
Nation Nation-Agents The person could be playing
sneaking on soneone, or possibly injured and trying to nove
location.
person Nation-Agents After crawling forward, the person night try to stand up, continue crawling, or reach for sonothing front of then. The exact next action would depend on the context of why they are crawling. scenarios you said, prodict longer notion.
Users you
Users Imagine this Notion-Agents
Users you explain
scenario where person sneaking up on saseone or trying nove cautiously, sequence notions could be as follows: First, the person is slouly crawling forward. Than, they stand up fron the crauling position. After standing they look around cautiously to ensure the enviroment is safe to locate their target. Finally, they walking sladly continue their cautious away. Generate longer
Nation-Agants
Users Then say person noticed Notion-Agents
Users Can you briefly write in the style of news report about what's happening? Nation-Agents In surprising
appeared stealthy approach, slouly crawling forward events, individual another person. kitnesses reported individual stood up cautiously after brief noment, looking around to assess the situation. However, the stealthy maneuver did not go unnoticed. Upon realizing they had been detected, the individual quickly turned and fled the onlookers bewildered the unexpected
engaging
attenpt sneak
sneaking
Figure 8: More examples of Motion-Agent.
Motion
Model
Caption
Ground Truth
person walks forward just like mummy
TM2T
person walk in counterclockwise circle with their arm the side the person is walking like a mummy from the dead
demo
MotionGPT
Ours
person walks forward while holding arms out as if to be zombie
Ground Truth
person walks forward slowly while their right hand is slightly elevate person slowly walk forward while hold onto something with their left hand person walks forward slowly, placing one foot front the other, on a helt that circulates, enabling the person to effectively slowly walk in place. the person is walking on balance beam
TM2T
demo 2
MotionGPT
Ours
Ground Truth
person moves side side zigzag fashion backwards
TM2T
person does a cartwheel to right a person is practing defense moves.
dcmo_3
MotionGPT
Ours
person walks backwards in zig-zag motion
Ground Truth
person makes and drinks cup of coffee
person hold something with their right hand and make sawing motion their left hand
TM2T
demo.4
MotionGPT
person is cating something
person uses their left hand open bottle, drinks from then places the bottle back down
Ours
Table 4: Comparsion of motion captioning ability across different models. Original motions can be found in supplementary material.
A.2 ABLATION STUDY ON DIFFERENT LLMS
In this study, replace GPT-4 with several other LLMs, including Llama (Touvron et al.) 2023), Gemma (Team et al. 2024b), and Mixtral (Jang et al. (024a). The experiment involved a straight- forward two-turn conversation. In the first turn, the prompted, "Generate motion that person
16

---

Published as a conference paper at ICLR 2025
Users Now generate notion of person stepping to the left.
Motion-Agemn
Users Then he picks up sonething froe the ground. Motion-
Users Then walks forward then places sorething on the ground.
Mot Son-Agents
Users stretches Motion-Agent5
Usend Then the person wallss as 1f he drunk. Motion-
Figure 9: More examples of Motion-Agent.
doing exercise." In the second turn, we provided a motion that person is slowly crawling forward and asked, "Briefly explain the possible scenarios for this motion." " The decomposed arguments from the agent in the first turn and the response from the second turn are presented in Table
Overall, we observe that different LLMs can generate reasonable outputs in response to user re- quests. However, smaller models, such as Llama-3-7B and Mixtral-8x7B, while capable of produc- ing some acceptable responses, struggled adhere strictly the instructed JSON format. As result, the agent was unable to parse their outputs successfully.
A.3 MULTI-HUMAN WITH MOTION-AGENT
In this section, we present the results of our preliminary trials on multi-human motion generation using the Motion-Agent framework, specifically focusing generating motions for two individuals.
In our implementation, each person represented in the HumanML format (Guo et al. 2022a), with their motions defined separately. To uniquely define the motions of both individuals, we incorporate location information for the first frame. represented by a tuple of three parameters, relative r = (0, Here, the first person is always positioned at the origin in 3D space, and the relative tuple r determines the position of the second person concerning the first. The parameter denotes the rotation radius, while and represent the coordinates (with the y-axis vertical). Therefore, the
17

---

Published as a conference paper at ICLR 2025
MoMask
Ours
Time
8
A person stands notionless and then takes step backwards to the left.
Figure 10: Comparison between MM and MoMask (Guo et al. 2024), which is non- autoregressive. During regular inference, MoMask uses a length estimator predict the length conditioned on the text. This estimator is likely to fail. In this example, their incorrect predicted length causes severe drifting.
LLM
Turn 1 Decomposition Results
Turn 2 Reasoning Response
GPT-4-Turbo
'A person does jumping jacks', 'A person does push-ups." 'A person does sit-ups.,, person stretches."
"The person could be participat- ing in military training exercise, playing a game that involves crawl- ing, navigating through confined space, or engaging in a fúness rou- tine that includes floor-based move- ments."
Llama-3-70B
'A person stands up straight'' 'A person raises their arms above their head', 'A person bends their knees slightly', 'A person starts jogging place.' 'A person lowers their arms to their sides.'
"The person could be participat- ing in military or law enforce- ment training exercise, engaging in crawling competition, playing a game that involves crawling, or navigating through a confined or obstacle-filled space."
Gemma2-27B
'A person does jumping jacks.' 'A person does push-ups!, 'A person does sit-ups.'
Possible scenarios include: army training exercise, searching for something lost on the ground, play- ing a game, navigating a tight space, or enacting dramatic scene.
Llama-3.1-8B
'A person starts in a standing posi- tion', 'A person bends their knees.', 'A person lowers their body', 'A person stands up.', 'A person peats the motion."
"The person could be crawling through tight space, crawling the floor for exercise, or crawling as a form of slow and deliberate move- ment in a situation where walking rumning is not feasible."
Llama-3-7B
Failed to follow the format
Failed to follow the format
Mixtral-8x77
Failed to follow the format
Failed to follow the format
Table 5: Comparison of Using Different LLMs on a Two-Turn Prompt.
18

---

Published as a conference paper at ICLR 2025
you generate people waving Motion-Agents
Figure 11: Multi-human Motion Generation using Motion-Agent.
motion of each person together with can uniquely determine the whole motion. In this context, GPT-4 is tasked with generating three outputs: the arguments for MotionLLM for each person and the relative tuple r.
Figure II shows an example of human generation using Motion-Agent. In this case, the two arguments are "A person waves.", and (3.14, 14, 0, 1), indicating that the second person rotates 180 degrees (since 3.14 z ло from facing the 2 direction (hence positioned face to face with the first person) and standing meter away from the first person.
A.4 EVALUATION METRIC
We detail the calculation of several evaluation metrics proposed in Guo et al. (2022a). We denote ground-truth motion features, generated motion features, and text features as Jat. Spred. and fieu- Note that these features extracted with pretrained networks in Guo et al. (002aa)
Multimodal Distance (MM-Dist). MM-Dist is widely used to evaluate the motion generation abil- ity of the model. MM-Dist measures the distance between the text embedding and the generated motion feature. Given N randomly generated samples, the MM-Dist measures the feature-level dis- tance between the motion and the text. It computes the average Euclidean distances between each text feature and the generated motion feature from this text:
N
MM-Dist
Σ
11
fiext
N
where fpred,i and fiest, are the features of the i-th text-motion pair.
Frechet Inception Distance (FID). FID measures the distance of motion features distribution be- tween real and generated motions. We calculate FID by
FID = ll Mpreall2 2 Tr(2ig + Epred 2(E Epreaa))
where Pgt and Mered the means of fgt and fpred- E is the covariance matrix and Tr denotes the trace of a matrix.
R precision Given the motion sequence and 32 text descriptions (1 ground-truth and 31 randomly selected mismatched descriptions). we rank the Euclidean distances between the motion and text embeddings to get Top-1, Top-2, and Top-3 accuracy of motion-text;
Diversity. Diversity measures the variance of the whole motion sequences across the dataset. We randomly sample Sais pairs of motion and each pair of motion features is denoted by Spred, and fredd The diversity can be calculated by
Diversity
ppedd
Spred,ell
Sais
In our experiments, we set Sai to 300 as (Guo et al.
2022a
Linguistic metrics. Linguistic metrics including Bleu (Papineni et al. 2002). Rouge Lin 2004
Cider (Vedantam et al. 2015) and Bert Score (Zhang et al. (020), we follow TM2T (Gu et al. 20226), using NE.PEva to calculate. Readers refer to their papers for further details.
19

---

Published as a conference paper at ICLR 2025
A.5
MORE IMPLEMENTATION DETAILS
Prompts For MotionI.LM.
We use different prompts for different tasks.
Task
Prompts
Generate a motion matching the following input human motion description.
Motion Generation
Generate a caption matching the following input human motion token sequence.
Motion Captioning
Table Instructing prompts for Mo training and inference.
Hyper-parameters.
Our hyper-parameters settings for different tasks.
Hlyper-paaamete
Motion Generation
Motion Captioning
Batch size
6
Learning rate
le-5
le-5
LoRA rank
64
32
LoRA alpha
32
32
LoRA dropout
0.1
0.1
Codebook size
512
512
Codebook dim
512
512
Total vocab size
256514
256514 14
Table 7: Hyper-paramettrs of our models used in main experiments. Other VQ training settings are borrowed from T2M-GPT (Zhang et al. 2023b)
A.6 MORE ABLATION STUDIES ON MOTIONLLM
Ablation study different tokenizers We conducted additional experiments replacing cur- rent VQ-VAE tokenizer with the RVQ- VAE tokenizer used in MoMask (Guo et al.) (024). RVQ- VAE differs from traditional VQ-VAE by using Q+1 ordered codebooks. The first codebook is used for base tokens, similar to traditional VQ, while the remaining Q codebooks are used to represent residuals for enhancing fidelity.
Following MoMask's approach, the MotionLLM in our case is aware of the base layer tokens. To predict the residual layer token sequences, follow MoMask use additional autoregressive (NAR) transformer that takes as imput the base layer token sequences from the LLM output. This NAR transformer can thus be considered as part of our detokenizer, which does not affect the LLM's inference or training process.
The results are shown in Table Our results show that using RVQ-VAE does significantly improve performance compared to the original VQ-VAE, validating that the VQ-VAE model is al- ready sufficiently effective for our tasks. While RVQ VAE provides some potential improvements in fidelity, it introduces additional training overhead (the 13.4M parameters are from the NAR trans- former). Specifically, the NAR transformer required for residual token prediction is challenging to train and adds complexity to the system (namely, requiring an additional training step for the NAR transformer).
Notwithstanding, we believe that both VQ-VAE and RVQ-VAE are effective for motion tokeniza- tion, and that the choice between them more of an engineering decision based on the specific trade-offs involved. Our Motion-Agent framework remains flexible and can support different tok- enizers, as long as they can convert motions into discrete representations. Future advancements in tokenization methods may further enhance the framework's performance and capabilities.
20

---

Published as a conference paper at ICLR 2025
Precision
Models
Trainable Params
FID L Multimodal Dist +
Diversity f
Top
Top
Gemma2-2b R=32 VQ
41.5M
0.415
0.750
0.712
2.938
11.251
Gemma2-22 2b R=64 VQ
83.1M
0.422
0.762
0.658
2.929
11.195
Gemma2-9h 9hh R=32 VQ
108M
0.439
0.776
0.438
2.872
11.151
Gemma2-26 2b R=64 RVQ
13..44
0.429
0.768
0.647
2.857
10.126
Gemma2-22 Full VQ
2697.4M
0.423
0.774
0.591
2.913
11.138
Table 8: More ablation studies conducted on KIT-ML (Pappert et al., 2016).
Ablation study on LoRA Full fine-tuning We also experimented fine-tuning all the parameters of LLM, the corresponding result is shown in Table [ When using 2B backbone, full fine-tuning does improve overall scores but the costly expense of significantly increasing training overhead, which is in stark contrast to LoRA fine-tuning 9b backbone model which requires over 20 times fewer parameters trained. Fully training such large model also demands substantially more com- putational power and memory, making it nearly impossible indicates that LoRA is indeed a more efficient and effective approach than full fine-tuning, suggest- ing it is better alternative for scaling up to a larger backbone in the future.
consumer-level GPUs. This result
21


Reviews:
[{"review_id": "pLRINSNPMO", "reviewer": "Reviewer_dbek", "summary": "In this paper, the authors propose Motion-Agent, a conversational framework for human motion that utilizes large language models (LLMs). By incorporating MotionLLM, a generative agent fine-tuned with adapters, Motion-Agent enables bidirectional motion-text translation and supports multi-turn conversational interactions for tasks such as generation, captioning, and editing. The framework leverages the HumanML3D and KIT datasets to demonstrate its effectiveness, achieving competitive results by integrating GPT-4 for coordination without additional task-specific tuning.", "questions": "1. Did the authors attempt to fine-tune all parameters of the language models within MotionLLM? If so, what were the resulting outcomes, and how did they compare to the model tuned with LoRA?\n\n2. Is it feasible to extend the Motion-Agent framework to incorporate vision and audio modalities, given that GPT-4o is known to support multimodal inputs? What impact might this integration have on the model's performance and potential applications?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper introduces Motion-Agent, a conversational framework that leverages pre-trained large language models (LLMs) for bidirectional human motion generation and understanding, achieving competitive results across various motion-language tasks.\n\n2. The authors demonstrate Motion-Agent’s effectiveness through an extensive evaluation of the HumanML3D and KIT datasets, highlighting the model's competitive ability to generate and caption human motion sequences.", "weaknesses": "1. The proposed Motion-Agent framework seems to focus primarily on conversational motion generation, a capability that, according to the authors, could also be achieved using additional datasets for task-specific instruction tuning. While the authors assert that Motion-Agent is efficient, the experiments presented offer insufficient evidence to substantiate this claim.\n\n2. The authors are encouraged to expand the discussion on the potential advantages of the proposed Motion-Agent framework. For instance, how does the model address out-of-domain motion concepts in comparison with current methods? A more thorough analysis of the model's generalization capabilities would contribute to a deeper understanding of its overall effectiveness.\n\n3. The authors claim that Motion-Agent can theoretically achieve infinite motion generation. However, plots or tables illustrating changes with increasing conversation turns and motion lengths, should be provided to substantiate this claim.\n\n4. The paper would benefit from ablation studies on the motion tokenizer, as well as comparisons with state-of-the-art RVQ-VAE models, such as MoMask which also employs RVQ-VAE to convert motion into a discrete representation.\n\n5. The paper lacks comparisons on additional MotionGPT benchmarks, such as motion composition tasks. Tuning MotionLLM with these task is something that can be easily done.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ofOgxFmiUR", "reviewer": "Reviewer_7jDa", "summary": "This work presents a framework capable of understanding, generating, and editing 3D human motion based on dialogue. Specifically, the model includes a more advanced LLM (GPT-4) play a role of \"coordinator\" and a motion/language translator fine-tuned from a lightweight language model (LoRA). The experiments (including the ablation study) are thorough, and the results (including the demos) demonstrate excellent performance.\n\nFrom my perspective, the reason for achieving this performance isn’t due to improvements in the language-motion model itself, but is because the coordinator achieves the high level of understanding, decomposing, and recording the tasks before language-motion model, significantly enhancing the results (or the user’s perceived experience) without change to the existing data or methods. This is very interesting and smart (though perhaps slightly tricky). I am in favor.", "questions": "see above", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "+ Very interesting idea and very reasonable design.\n\n+ Thorough experiments and good performance.\n\n+ Good writing.", "weaknesses": "- I am very curious about the long motion sequence generation. As illustrated by authors, \"By decomposing descriptions of long motions into a series of short motions using LLMs and subsequently concatenating these short motions into longer sequences, our Motion-Agent\ncan theoretically achieve infinite motion generation.\" Though I can generally understand the meaning, the details are unclear. Is \"decomposition\" done by GPT4 during long motion seq generation? If so, how it is achieved? Could authors provide detailed output of GPT4 of an example (like Figure 3)? Meanwhile, during training, will GPT4 be used to decompose HumanML3D data into shorter atomic units? If so, how does such labeling achieved and how to guarantee the alignment between motion and text?\n\n- An important work is missed in related works as one of the first works that quantize/tokenize motion into GPT: \nBailando: 3d dance generation by actor-critic GPT with choreographic memory, CVPR 2022;\nBailando++: 3d dance GPT with choreographic memory, TPAMI 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mX9oJfNkqZ", "reviewer": "Reviewer_DEqA", "summary": "The paper introduces Motion-Agent, a novel conversational framework for 3D human motion generation, editing, and understanding that leverages large language models (LLMs). The framework employs a pre-trained language model called MotionLLM, which bridges the gap between motion and text by encoding and quantizing motions into discrete tokens aligned with the language model's vocabulary. By fine-tuning only 1–3% of the model's parameters using adapters, MotionLLM achieves performance comparable to diffusion models and other transformer-based methods trained from scratch. Integrating MotionLLM with GPT-4 without additional training, Motion-Agent enables the generation of highly complex motion sequences through multi-turn conversations, a capability previous models lacked. The framework supports a wide range of motion-language tasks, offering versatile capabilities for generating and customizing human motion through interactive conversational exchanges. Experiments demonstrate that Motion-Agent effectively handles intricate, multi-turn interactions and achieves strong results in both motion generation and captioning tasks.", "questions": "Can the authors provide more quantitative evaluations comparing Motion-Agent with state-of-the-art models on standard benchmarks?\nHow does the choice of the pre-trained LLM (e.g., GPT-4 vs. open-source models) impact the performance of the framework? Have experiments been conducted with other LLMs?\nWhat are the computational requirements (e.g., inference time, memory usage) of Motion-Agent, especially for real-time interactive applications?\nHow does the framework handle motions involving interactions with the environment or multiple agents (e.g., multi-human interactions)?\nAre there limitations regarding the diversity or realism of the generated motions when handling highly complex or extended sequences?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: The paper presents an innovative approach by integrating large language models into 3D human motion generation, a relatively unexplored area. The conversational framework allowing multi-turn interactions for motion generation and editing is particularly novel.\nQuality: The proposed method effectively leverages pre-trained LLMs with minimal fine-tuning, demonstrating competitive performance with significantly reduced computational resources compared to models trained from scratch.\nClarity: The methodology is well-articulated, with clear explanations of the motion tokenizer/detokenizer and how MotionLLM integrates with GPT-4. The inclusion of qualitative examples and ablation studies aids in understanding the framework's capabilities.\nSignificance: The ability to generate complex, customizable human motions through conversational interactions has substantial implications for animation, virtual reality, and human-computer interaction. The framework's versatility enhances its practical relevance in various applications.", "weaknesses": "Evaluation Metrics: The paper primarily presents qualitative results and some standard metrics but lacks comprehensive quantitative comparisons with state-of-the-art models on standardized benchmarks, making it challenging to fully assess performance improvements.\nLimited Ablation Studies: While an ablation study is included, more extensive experiments isolating the contributions of each component (e.g., the impact of different LLMs, the motion tokenizer) would strengthen the evaluation.\nDependence on Proprietary Models: Utilizing GPT-4 as the conversational LLM may limit reproducibility and accessibility, as GPT-4 is not openly available to all researchers.\nScalability and Efficiency: The paper does not thoroughly discuss the computational efficiency and real-time performance of the framework, which is critical for interactive applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2Eh5o1Kcwt", "reviewer": "Reviewer_QaCH", "summary": "Motion Agent introduces a conversational framework for generating, editing, and reasoning about human motion in multiple steps. This framework is built on a MotionLLM fine-tuned with LoRA. With a motion tokenizer-detokenizer and a text detokenizer, Motion Agent can create new motions and generate text that explains or describes the motions in natural language. Fine-tuning the large LLM allows Motion Agent to generalize better than current state-of-the-art methods, produce longer motion sequences, and handle language-guided edits.", "questions": "1) What are the benefits of using VQVAE over other models like diffusion/VAE?\n\n2) Which body joint representations were used for training?\n\n3) The extended motion sequences keep the original motion unchanged. This is visible in all the provided videos. For example, with prompt p1, motion m1 is generated; when p2 is added, it generates a new motion m2, so the combined motion is m1 + m2. How is m1 preserved—does the new generation only respond to the new prompt?\n\n4) Lines 191-192 mention that the motion detokenizer is important for smoothing transitions between different motion sequences. Are the motions generated by MotionLLM concatenated and sent to the motion detokenizer after each prompt?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The paper is well-written and supported by experiments, with a good presentation of ablation studies and applications. The supplementary material includes videos for temporal generations, which helps in reviewing the work better.\n\n2) This research is timely, as using LoRA fine-tuning on LLMs for multimodal human motion generation allows for models that don’t rely on task-specific data. \n\n3) Multi-turn editing and the ability to generate long motion sequences are valuable directions for the field.", "weaknesses": "1) Lines 280-282: Multiple adapters are mentioned to work together, and fig 2 shows two LoRA adapters after fine-tuning. It looks like they’re applied to the base LLM in separate branches. How does MotionLLM.generate or MotionLLM.caption work—are they called every time during inference, or just when needed? This information would help clarify the pipeline.\n\n2) Line 323 states that there is no ground truth for these tasks, but the videos show apparent issues like body interpenetration, lack of physical constraints (e.g., floor contact), and body deformation. There are also no quantitative metrics for multi-turn edited generations. Even though the motion quality seems better than the current SOTA, these limitations stand out. One way to address this would be to explain why these interpenetrations occur, especially since the motion tokenizer/detokenizer was trained on motion reconstruction losses, which should ideally prevent this. Another approach could be to generate multi-turn prompts based on test set motion sequences and provide metrics like FID, where the test sample could act as a pseudo ground truth.\n\n3) The examples in fig 3 show different genders/body types across three instances. How does Motion Agent choose each identity or gender? Are all identities compatible in terms of body kinematics, or could retargeting cause issues if aiming for consistent identity? This needs more explanation.\n\n4) Figure 5 shows in-betweening, but there’s no quantitative metric to evaluate it. Using a metric like NPSS (e.g., https://arxiv.org/pdf/1809.03036, https://arxiv.org/pdf/2102.04942) could provide useful information on the in-between motion quality.\n\n5) Tab 2 - Generation Task: FID scores are lower than the state-of-the-art. As noted in Lines 416-418, could the motion length be constrained to match other methods to provide a clearer comparison of the FID scores?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C6bqRiIGBw", "reviewer": "Reviewer_oMaw", "summary": "This paper introduces Motion-Agent, an efficient framework for general human motion generation, and it also develops a generative agent, Motion LLM, which bridges the gap between motion and text. Experiments show that with few-shot fine-tuning, MotionLLM can achieve state-of-the-art performance. Motion-Agent can generate complex motion sequences.", "questions": "1. What's the rendering engine to generate those visualizations?\n\n2. Can this framework benefit down stream tasks such as robotics and policy learning?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper presents a framework to bridge text and motion. This part is novel in the opinion of the reviewer. The proposed motion tokenization approach and Motion-language Agent have some level of novelty.\n\n2. Table 1 summarizes the advantages of this work against several recent competitors. It seems that this work supports multi-turn editing, reasoning, and composition. Also, Motion LLM can generate longer motion, which is superior to previous works.\n\n3. Motions in this work and its supplementary (including video results) look cool. This indicates that the proposed model can generate complex motions.", "weaknesses": "1. The website in the supplementary cannot function well (videos cannot be rendered). The reviewer suggests authors publish their websites on anonymous hosts (such as Google Pages). Moreover, it's better to show case more diverse prompts.\n\n2. More ablation studies are welcomed. For example, different types of tokenizers, detailed comparison between LLaMa and Gemma, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)