MotionGPT: Human Motion as a Foreign Language
Authors: Biao Jiang
Xin Chen
Wen Liu
Jingyi Yu
Gang YU
Tao Chen
Venue: NeurIPS 2023 poster (2023)
Keywords: 3d motion
motion generation
human motion synthesis
text-driven
text-to-motion
TL;DR: We explore a unified and user-friendly motion-language model to learn the semantic coupling of two modalities and generate high-quality motions and text descriptions on multiple motion tasks.

Abstract: Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multimodal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.

MotionGPT: Human Motion as a Foreign Language
Biao Jiang 1.2
Xin Chen 2*
Wen Liu? Tencent
Jingyi Yu
Gang Yu² ShangghaiTech University
Tao Chen'l
Fudan University
https://github com/ homm/DDeattottoonaaa//oooooooooo
Abstract
Though the advancement of pre-trained large language models unfolds, the explo- ration of building unified model for language and other multimodal data, such motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as form of body language. By fusing language data with large-scale motion moteln, language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, unified, versa- tile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process word tokens. Building upon this "motion vocabulary", perform language mod- eling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, pre-train MotionGPT with a mixture of motion- loniinn-language data and fine-tune it on prompt-based based question- and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including texi-driven motion generation, motion captioning, motion prediction, and motion in-between.
Introduction
Recent years have witnessed significant breakthrough in pre-trained large language models such as GPT [34, 35, 3, 27], BERT [7], and T5 [36, 5], which lead to the convergence of language [59, 47], image [33, 50, 20], mesh [55, 26] and mutlimodal [8] modeling. Nevertheless, general pre-trained model for human motion and language has yet to emerge. This pre-trained motion-language model, capable of supporting numerous motion-relevant tasks through prompts, should benefit diverse fields like gaming, robotics, virtual assistant, and human behavior analysis.
Previous research on human motion has explored various tasks, including motion generation [29, 10, 46, 52, 57], motion captioning [9, 11], and motion prediction [56, 61, 24]. Recent text-to-motion works[46, 58, 30, 52] have attempted to employ pre-trained language-relevant models [7, 33]. For instance, MDM [46] learns a motion diffusion model with conditional text tokens from CLIP [33], while MLD [52] integrates motion latent space to improve the efficiency of motion diffusion process. On the other hand, MotionCLIP [45] and TM2T [11] concentrate on modeling the coupled relationship between motion and text description. However, the above approaches treat motion and language as separate modalities, which often require strictly paired motion and text data. Moreover, since the supervisions task-specifie, they can hardly generalize effectively to unseen tasks data, they lack a comprehensive understanding of the relationship between motion and language. We thus focus on building a pre-trained motion-language model, which can generalize various tasks and learn in-depth motion-language correlation knowledge from more feasible motion and language data.
Two challenges crucial and need to be solved for pre-training promising motion-languee model. The first is modeling the relation between language and motion, and the second is building a uniform multi-task framework that can generalize to new tasks. Fortunately, human motion exhibits semantic
"Contributed equally work done while Biao Jiang was Research Intern with Tencent PCG tCorresponiing author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Can you generate more from this motion?
Can you generate motion that a person kneeling the ground gets up?
Can you translate this motion to text?
Describe the motion of someone as you will.
A person kicks two times on his left then kicks forward two times.
A man stands and moves his left leg back- ward, then try to throw something hard.
Figure 1: MotionGPT can address diverse motion-relevee tasks uniformly given different instructions. We provide the results on text-to-motion (the upper left), motion captioning (the bottom left), motion completion (the upper right), and the language question-to-answer (the bottom right). The left right of motion represents the time order. Blue motion denotes the input, and yellow is the generation.
coupling similar to human language, often interpreted form of body language. Building upon this observation, follow vision-language pre-training from BEiT-3 [50] to treat human motion as specific foreign language. By integrating motion and language data together and encoding them within a single vocabulary, the relationship between motion and language becomes more apparent. Therefore, with recent significantly larger-scale language data and models, the motion-languag pre- training has great potential to improve the performance on motion tasks. Meanwhile, this pre-training on language enables textual instructions like prompts in InstructGPTT [27] and makes the model more versatile and eser-friendly for various motion tasks.
In this work, propose uniform motion-language framework, namely MotionGPT, that leverages the strong language generation and zero-shot transfer abilities of pre-trained language models for doing human motion-related tasks. To enable MotionGPT comprehend and generate human- -ike motions, we first learn motion-specific vector quantized variational autoencoder (VQ-Va VAE) model to construct "motion vocabulary". akin to English vocabulary and then convert motion data into sequence of motion tokens. These tokens are then processed by pre-trained language model [36, 5] that learns the underlying grammar and syntax of the motion language, well its relationship with the corresponding textual descriptions. To effectively integrate language and motion in MotionGPT, we design two-stage training scheme. We first pre-train the language model on the raw motion dataset to learn the basic grammar and syntax of the motion language. For prompt tuning, we fine-tune the language model on an instruction dataset, which contains both textual descriptions and motion data, to learn the correlation between the two modalities. Extensive experiments demonstrate that MotionGPT achieves ofate-of-the-arr performance on text-to-motion, motion-to-texx motion prediction, and motion in-between.
We summarize contributions follows: (1) We propose uniform motion-language generative pre-trained model, MotionGPT, which treats human motion as a foreign language, introduces natural language models into motion-relevant generation, and performs diverse motion tasks with single model. (2) We introduce a motion-language training scheme with instruction tuning, to learn from task feedback and produce promising results through prompts. (3) We propose general motion benchmark for multi-task evaluation, wherein MotionGPT achieves competitive performance across diverse tasks, including tex--to-motion, motion-to-text, motion prediction, and motion in-between, with all available codes and data.
2 Related Work
Human Motion Synthesis involves generating diverse and realistic human-like motion using multi- modal inputs, such as text [10, 30, 58, 46, 11, 1, 17], action [29, 12, 46, 52], and incomplete motion [56, 61, 24, 46]. Text-to-motion is one of the most important motion generation tasks, due to the user-

---

Methods
Text Motion
Motion Text
Motion Prediction
Motion ln-between
Random Motion
Random Description
12M-GPT1577
x
X
MLD [52]
x
X
X
TM2T [1]]
X
X
X
MDM [46]
X
X
MotionDiffuse(58]
X
X
MotionGPT (Ours)
Table I: Comparison of recent state-of-the-ar methods on diverse motion-relevant tasks. Random
Motion and Random Caption represent unconstrained generation motions and motion descriptions.
friendly and convenient language input. MDM [46] proposes a diffusion-based generative model [14] separately trained on several motion tasks. MLD [52] advances the latent diffusion model [43, 38] to generate motions based on different conditional inputs. T2M-GPT [57] investigates generative framework based on VQ- VAE and Generative Pre- rreitrained Transformer (GPT) for motion generation. Motion completion task generates motion conditioning on partial motions, such as classical motion prediction [56, 61, 24] or motion in-between [46], which generates the intermediate motion while the first and last parts are fixed. Although they show promising results in various human motion tasks, most above methods are limited in using a single model to handle multiple tasks. We thus propose uniform approach that treats human motion as foreign language, and leverages the strong language generation and zero-shot transfer abilities of pre-trained language models
Human Motion Captioning. To describe human motion with natural languages, [44] learns the mapping from motions to language relying on two statistical models. Furthermore, recurrent networks have also been used in [54, 32]. More recently, TM2T [11] proposed a new motion representation that compresses motions into short sequence of discrete variables, then a neural translation network to build mappings between two modalities. While previous research like TM2T [11] incorporated captioning modules into their training pipeline for motion generation, these approaches are constrained to bidirectional translation between text and motion within one uniform framework.
Language Models and Multi-Modal. Large-scale language models (LLMs) [7, 36, 3, 59, 47], enabled by extensive datasets and model size, have demonstrated impressive comprehension and generation capabilities, elevating natural language processing to heights. BERT [7] praits deep bidirectional language representations that can support downstream tasks. T5 [36] introduced a unified framework that converts all text-based language problems into text-to-text format. More recent research [51, 2, 27, 5] find that by fine-tuning pre-trained models using input-output pairs consisting of instructions and coupled answers, the performance of pre-trained models can be further improved. FLAN [5] presents an
technique that surpasses the performance of non-tuned models in unseen tasks. Recently, the wave of multi-modal models [20, 15, 19] is intriguing to process text along with other modalities, such as images [20, 15, 8J, audio [13, 8J. and vidcos [53]. CLIP [33] further learns a semantic latent representation that couples images with corresponding language descriptions. Despite the success of language models in various vision- lssguu-la tasks, the development of multi-modal language models that can handle human motion is still limited.
Motion Language Pre-training. Existing text-to-motion generation methods [10, 30, 46, 11, 17] be characterized caption-to-motion, where the models take in pure text description of the desired motion. While these methods can generate motions from textual descriptions, they are often limited in supporting instructions from users like InstructGPT [27]. In other words, they do not allow users to provide context-specific instructions for certain applications. MotionCLIP [45] utilizes the language and visual understanding of CLIP [33] align its latent space with motion auto-encoder. Meanwhile, many language models, such as T5[36] and InstructGPT [27], have been developed to address diverse language processing tasks, including translation, question answering. and classification. These models are typically designed to map given text input to a target output, such as a translation or answer. However, while these models have shown remarkable performance in language tasks, they have not been widely applied to motion tasks. Therefore, we propose MotionGPT to enable the effective integration of natural language models with human motion tasks, providing a unified solution for motion synthesis problems.
3

---

Can you give me motion that A person kneeling on the ground gets up
Text Tokens
Input Mixed Tokens
Text Codebook
<Toloni>>
Motion Codebook
Language Encoders
Language Decoders
Motion Tokens
Encoder
<Tokest>
Input Motion (optonal)
Tolanz
<Tokento
Linear Softmax
Tooents
Motion Decoder
<Tekerio
Vencdec
Output Tokens
Generated Motion
Language Model TS
MOTION TOKENIZER (SEC. 3.1)
MOTION VOCABULARY
MOTION- AWARE LANGUAGE MODEL (SEC. 3.2)
Figure 2: Method overview: MotionGPT consists of a motion tokenizer (Sec. 3.1) and a motion- aware language model (Sec. 3.2). Combining Motion Tokens learned by V and Text Tokens by text tokenizer, we then learn motion and language jointly utilizing language model as backhone.
Method
To involve large language data and models in the motion generation tasks, propose unified motion-language framework named MotionGPT. As illustrated in Fig. 2. MotionGPT consists of motion tokenizer responsible for converting raw motion data into discrete motion tokens (Sec. 3.1), as well as a motion-aweed aware language model that learns to understand the motion tokens from large language pre-training models by corresponding textual descriptions (Sec. 3.2). To address motion- relevant tasks, we introduce a three-stage training scheme (Sec. 3.3) of MotionGPT for the training of motion tokenizer, motion-language pre-training, and instruction tuning.
We first propose the motion tokenizer consisting of a motion encoder & and a motion decoder D, to encode M frame motion 1::M {x" ) == into motion tokens Zا: (= L M/4, and decode 21:.. back into the motion mh l:M D(2 1:)) D(E(m!: (m)):)) where / denotes the temporal downsampling rate on motion length. Then, given an length sentence I:N = {w" Ji=1 describing mnttoon-reltter question demand, MotionGPT aims to generate its answer L length tokens
)
I1:L = { | It could be the human motion tokens #1LL or the text tokens 11:L which results in
motion m 1:M or a sentence 1:L4 like a description of the given motion.
3.1 Motion Tokenizer
To represent motion in discrete tokens. we pre-train 3D human motion tokenizer V based on the Vector Quantized Variational Autoencoders (VQ- VAE) architecture used in [48, 42, 11, 57]. Our motion tokenizer consists of an encoder & and decoder DD The encoder generates discrete motion tokens with high informative density, while the decoder able to reconstruct the motion tokens into motion sequences m I:M This approach enables us to efficiently represent motion as language, facilitating the integration of motion and language for various motion-related tasks.
Specifically, the motion encoder & first applies ID convolutions to given frame-wise motion features - along the time dimension, obtain latent vectors El:L = (mm. Next, we transform 2 into a collection of codebook entries through discrete quantization. The learnable codebook Z { Rd consists of K latent embedding vectors, each of dimension The process of quantization Q--) replaces each row vector b with its nearest codebook entry by in Z, written as
Q(22) == arg min || 2x112-
(1)
After quantization, the motion decoder D project IL { back to raw motion space the motion m I:M with M frames. To train this motion tokenizer, we follow [11, 57] to utilize three distinct loss functions for training and optimizing the motion tokenizer: L, L, L. Le, where reconstruction loss L,, the embedding loss L., and the commitment loss L.. To further improve the generated motion quality, we follow [57] to utilize smooth loss and velocity regularization the reconstruction loss, as well exponential moving average (EMA) and codebook reset techniques [37]

---

to enhance codebook utilization during training. We provide more details about the architecture and the training of our motion tokenizer in the supplement.
3.2 Motion-aware Language Model
Employing this motion tokenizer, a human motion m I:M can be mapped to a sequence of motion tokens 2 allowing for joint representation with similar vocabulary embedding in language mod- els [18, 36, 27J. By combining them in the unified vocabulary, we then learn motion and language jointly. We first represent motion tokens z Z1: as a sequence of indices د² {s")== where s corresoonds to the index number of motion tokens 2!:2. On the other hand, previous language models, such as T5 [36], encode text as WordPiece tokens. They utilized a vocabulary of K; word pieces and trained the SentencePiece [18] model on mixture of language datasets.
Most previous text-to-motion [11, 52, 57] or motion-to-texx [11] approaches employ different modules to handle text and motion individually, while we aim to model text and human motion together and in the same way. To achieve this, we combine the original text vocabulary V22 {wi] with motion K.. vocabulary Vn fur I==1. which is order- peeeerrreseriing to motion codebook Z. Moreover, Vn. includes several special tokens like boundary indicators, for example, </som> and </eom> as the start and end of the motion. Thus, we employ new unified text-motion vocabulary V = (V,, Vm ), and can formulate diverse motion-related tasks in general format, where both input "words" and output "words" from the same V. These "words" can represent natural language, human motion, even a mixture of two, depending on the specific task to be solved. Therefore, our MotionGPT allows for the flexible representation and generation of diverse motion-related outputs within a single model.
K2
To address the conditioned generation task, we employ a transformer-base architecture proposed in [36], which effectively maps the input sequences to the output. Our source input consists a sequence of tokens X, = fxr N Is where 2s E V and N represents the limls
model based on the
input length. Similarly, the target output is X, = { where 2, E V and L denotes the output length. As shown in Fig. 2, the source tokens are fed into the transformer encoder, and the subsequent decoder predicts the probability distribution of the potential next token at each step pa(az v.)) II, pe (zf x process, the objective is to maximize the log- ligeliidd of the data distribution:
22) ,z²,,,)
in autoregressive manner. Therefore, during the training
LLM Σ
log po (x) 2 .x.)
By optimizing this objective, MotionGPT learns to capture the underlying patterns and relationships from the data distribution, facilitating the accurate and meaningful generation of the target "words". During the inference process, the target tokens are sampled recursively from the predicted distribution pe (2t² d .r2) until the end token (i.e., 
). This sampling strategy enables the generation of the target sequence in step-by-step manner, where each token is probabilistically determined based on the previously generated tokens and the given source input.
3.3 Training Strategy
Since T5s have only been exposed to language data, represented within a text vocabulary Vi, we thus bridge motion and language and enable this language model to comprehend human motion concepts, by learning the motion vocabulary Vn. As shown in Fig. 3, our training scheme includes three stages: (1) Training of motion tokenizer, which focuses on learning the motion codebook represent human motion as discrete tokens. (2) Motion-language pre-training stage, which includes unsupervised and supervised objectives to learn the relationship between motion and language. (3) Instruction tuning stage, which tunes the model based on prompt-based instructions for different motion-relevant tasks.
Training of Motion Tokenizer. We first learn the motion tokenizer using the objective defined in Equation 1.1. This training process allows any human motion sequence & I.L. to be represented as a sequence of motion tokens, enabling seamless integration with textual information. Once optimized, the motion tokenizer remains unchanged throughout the subsequent stages of the pipeline.
Motion-language Pre-training Stage. The T5 models [36, 5) are trained and fine-tuned on natural language datasets with instruction-based phrasing [5, 27]. We continue to pre-train this model using a mixture of language and motions data in both unsupervised and supervised manners: 1) To generalize
5

---

Step
Step Motion-languag Pre-training.
Step
Training of Motion Tokenizer.
Instruction Tuning.
motion sequence sampled from 3D motion dataset.
motion and language description are sampled.
A person catches and throws bal.
Can you show me lffel person dies three straight ?
Could to
Motion tokenizer learns motion representation.
This motion mapped to discrete mation indices mixed with words.
Mation
Motion
catches and throws bel.
The sampled from our prompt templates.
seven
Motion codebook used represent human motion discrete tokens.
Mocion Codebook
This data is used to pre-train our motion- language
The prompts are used to fine-tune our model on diverse motion
Motion Tokena
Encoders
Decaders
Encoders
Decaders
Langaaee
Langaage
Figure 3: Training Scheme. We introduce three training steps for our MotionGPT (Sec. 3.3): First V learn a codebook for discrete motion representation. Then we train language using mixture of language and motion data to learn the semantic coupling between text and motion. Finally, we fine-tune the model in multi-task text-motion dataset with instructions.
to various downstream tasks like [7, 35, 36, 27], we follow [36] to design an objective, where certain percentage (15%) of tokens in the input tokens X, are randomly replaced with special sentinel token. On the other side, the corresponding target sequence is constructed by extracting the dropped-out spans of tokens, delimited by the same sentinel tokens used the input sequence, along with an additional sentinel token to indicate the end of the target sequence. 2) We then learn the motion-late languuge relation by the supervision of paired text-motion datasets [10, 31]. We train MotionGPT on the supervised motion-language translation, where the input is either human motion or text description After unsupervised and supervised training processes, we aim to equip our model with the understanding of text and motion relationships.
Instruction Tuning Stage. We construct multi-task text-motion dataset by formulating as instructions. building upon the foundation of existing cext-to-motion datasets such as HumanML3D [10] and KIT [31]. Specifically, define 15 core motion tasks, such motion generation with text, motion captioning. motion prediction, and others. For each task, we compose dozens of different instruction templates, resulting in more than one thousand different tasks, each having a unique instruction prompt. For example, an instruction prompt for motion generation task could be "Can you generate a motion sequence that depicts 'a person emulates the motions of a waltz dance'?". Similarly, for the motion captioning task, the instruction prompt could be "Provide an accurate caption describing the motion of <motion tokens>", <motionntokens>"" where <motion mmotion_tokens>>>> represents sequence motion tokens generated by our motion tokenizer. We have demonstrated the efficacy of instruction tuning in Sec. 4.3, which leads to improvement across various tasks and enhances the model performance for unseen tasks prompts. More examples of prompts are provided in the supplements.
4 Experiments
Extensive comparisons evaluate the performance of our MotionGPTss across multiple motion-relewe tasks and datasets. Details of the dataset settings, evaluation metrics, and implementation specifics (Sec. 4.1) .) are provided. We first present a uniform benchmark by comparing our approach with other SOTAs across various tasks (Sec. 4.2). Then, we evaluate each specific comparison on eex--to-motion (Sec. 4.2), motion-to-ter (Sec. 4.2), motion prediction and motion in-between (Sec. 4.2). The supplements include more qualitative results, user studies, and further implementation details.
4.1 Experimental Setup
Datasets. General motion synthesis can support diverse task settings, and thus previous datasets and modified benchmark are utilized to evaluate MotionGPT. The study primarily focuses on two text-to-motion datasets: HumanML3D [10] and KIT [31]. The KIT dataset provides 6,353 textual descriptions corresponding 3.911 motion sequences, while the HumanMI.3D dataset [10] is more recent dataset that contains 14,616 motion sequences obtained from AMASS [25], along with 44,970

---

Text Motion
Text
Motion Prediction
Motion In between
Methods
R TOPIT
FIDI
DIV-
R TOP31
Bleu@4
Cidert
FIDI
DIV-
FIDI
DIV-
Real
0.511
0.0021.000
9.5034
0.828
0.002
9.503
0.002
9.503
MLDD [52]
0.4734.m0
9.7244.022
T2M [57
0.491 4-00
0.116
9.761
2,056
8.635
TM2T [U]
0.424
15501..00
8.5804-o7s
0.829
7.00
16.8
MDM
0.3204405
0.5 0.5441.00e
9.5594 066
6.031
7,813
2,608
8.420
MotionGPT (Ours)
0.492
0.25g
9.528
0.827
12.47
29.2
0.905
8.972
0.214
9.560
Table 2: Comparison of four motion-related tasks on HumanMI.31 [10] dataset. The evaluation
metrics are computed using the encoder introduced in [10]. The empty columns of previous methods indicate that they can not handle the task. The arrows (-) indicate that closer to Real is desirable. Bold and underline indicate the best and the second best result on text-to-motion task.
sequence-level textual descriptions. To evaluate MotionGPT as a uniform framework on tasks, such as motion prediction and motion completion (in-between). we utilize the motion sequences available in ML33n which is also subset of the larger AMASS dataset. Following the previous works [10, 52, 46], we adopt the same motion representation for fair comparisons, which combines joint velocities, positions, and rotations. By using this consistent representation. MotionGPT enables the availability to support further studies in the field. (ef. supplement for the benchmark details.)
Evaluation Metrics are summarized as four parts. (1) Motion quality: Frechet Inception Distance (FID) is our primary metric based on a feature extractor [10] to evaluate the distance of feature distributions between the generated and real motions. For motion completion, we utilize metrics used in motion prediction studies [56, 61, 24]. such as Average Displacement Error (ADE) and Final Displacement Error (FDE), to evaluate the accuracy of the predicted motion. (2) Generation diversity: We utilize the Diversity (DIV) metric to assess the motions diversity, which calculates the variance through features extracted from the motions [10]. MultiModality (MM) measures the diversity of generated motions within the same text description of motion. (3) Text matching: Based on the feature space from [10], the motion-retrieval precision (R Precision) evaluates the accuracy of matching between texts and motions using Top 1/2/3 retrieval accuracy. Multi-modal Distance (MM Dist) measures the distance between motions and texts. (4) Linguistic quality: We follow [1I] utilizing linguistic metrics from natural language studies, including BLUE [28], Rouge [23], Cider [49], and BertScore [60] to evaluate the quality of generated motion captions.
Implementation Details. We set the codebook of motion tokenizer as K E R512x512 for most comparisons. The motion encoder e incorporates temporal downsampling rate I of 4. We utilize T5 [36] as the underlying architecture for our language model, with baseline model consisting of 12 layers in both the transformer encoder and decoder. The feed-fon ferdffrrwrrd networks have an output dimensionality of da 3072, and the attention mechanisms employ inner dimensionality of du. = 64. The remaining sub-layers and embeddings have a dimensionality of dmnddll = 768. Moreover, all our models employ the Adam W optimizer for training. The motion tokenizers are trained utilizing a 10-4 learning rate and a 256 mini-batch size, while our language models have 2 x 10 learning rate for the pre-train stage, 10 for the instruction tuning stage, and 16 mini-batch size for both stages. The motion tokenizer undergoes 150K iterations of training, while the language model undergoes 300K iterations during the pre-train stage and another 300K iterations during the instruction tuning stage. Small and Base models are trained on 8 Tesla V100 GPUs while Large models are trianed on 64 Tesla V100 GPUs.
4.2 Comparisons on Motion-relevant Tasks
Comparisons on Multiple Tasks. By introducing a uniform framework that treats human motion as a foreign language, we open up the exploration of diverse motion-relevant tasks. We employ a 220M pre-trained (lan-75-Base[33 5] model as our backbone and fine-tune the model through the pre-training and instruction tuning stage (Sec. 3.3) for all following comparisons. As shown in Tab. 2, we evaluate MotionGPT against state-of-the-arr methods on key tasks such as text-conditioned motion generation [52, 57, 11, 46], motion captioning
[11], motion prediction [46], and motion in-between[46) [46]. While we leverage existing results from previous works or benchmarks for text- to-motion and motion-to-text tasks, we re-implement the motion diffusion models [46] for motion
prediction and evaluate it under the same metrics and settings. Please note that some methods are
7

---

RPrecision"
Methods
FID
MMDist
Diversity
MModality
Topl
Top2
Top3
Real
0.511 1
0.703+.008
0.7974:002
0.002=-003
2.974
9.503
TM2T [1]]
0.4224-000
8:-080
0,729
1.501 1.501=-017
3.467
8,58g
2.424
T2M [10]
0.457
0.63945.003
0.740
1.067=
3.340
9.188
008
2.090
MotionDiffuse [58]
0. 0.491 2.001
0.681 001
0.782 2000
0.6 0.630
3.113
9.410 049
1.553
MDM [46]
0.320 2005
0.498
0.611
0.5444=-044
5.506+.027
9.509 066
2.799
MLD [52]
0.481 0.481 #.003
0.673+4.00
0.772
0.473=
3.196**-010
9.724 062
2.413+.079
T2M-GP [57]
0003
0.680 003
0.775
0.116 16
3.1184.011
9.761 061
1.856+.011
MotiouGPT (Pre-trained)
0.455..00
0.007:t-002
0.700
0.100
000
3.700=t-008
9.411 01:1
3.437 2.091
MotionGPT (Fine-tuned) 0.492
0.681 2 #.003
0.778
0,232 2008
3.096
.008
9.528 #.071
2.008 084
Table 3: Comparison of text-to-motion on HumanML3D [10]. The empty MModality indicates Real motion is deterministic. These methods are sorted by FID. Pre-trained and Fine-tuned indicate uniform motion-language pre-training and specific fine-tuning this task. (ef. Tab. 2 for notations.)
RPrecision
Methods
MMDist,
Length
Bleu@ 11
Bleu@4f
Rouget
Cidert
BertScure
Top1
Top3
Real
0.523
0.828
2.901
12.75
TM2T[U]]
0.516
0.823
2.935
10.67
48.9
7.00
38.1
16.8
32.2
MP (Ours)
0.543
0.827
13.04
48.2
12.47
37.4
32.4
Table 4: Comparison of motion captioning HumanMI.3D [10]. The evaluation metrics follow [11], while we use the ground truth texts without pre- pre-procsssing for linguistic metrics calculation.
designed for specific tasks, and thus some metrics are empty for tasks they cannot handle. The results presented in Tab. demonstrate that our MotionGPT achieves competitive performance across all evaluated tasks, highlighting its capability address diverse motion tasks within a single model.
Comparisons on Text-to-Motion. The text-to-motion task involves generating human motion sequences based on given text input. We evaluate the proposed the MotionGPT model as the pre-trained MotionGPT, the same one in Tab. 2, as well as fine-tuned it on text-to-motion task. We compare MotionGPTss with other SOTAs [11, 10, 46, 52, 57] and evaluate the performance both HumanML3D and KIT datasets using suggested metrics [10]. The results are computed with a The majority of the reported results taken directly from their own papers or the benchmark presented in [10]. Tab. 3 summarizes the comparison results, where MotionGPT achieves competitive performance most metrics.
95% confidence interval, obtained from 20 repeated
Comparisons on Motion-to-Text. The motion-to-text task involves generating a text description based on given human motion sequence. We compare the pre-trained MotionGPT with recent work TM2T [1]]. We evaluate the performance on the HumanML3D using the suggested metrics from [[]]. Additionally, measure the average numbers of words Length for further comparisons. Please note that the reported results in [11] are evaluated with pre-processed ground truth text, which ignores the grammatical tense and plural forms of words. In Tab. 4, we directly use the ground truth text descriptions for a more accurate assessment. This comparison shows that MotionGPT overperforms recent work on text descriptions of given motions.
Comparisons on Motion Prediction and In-between. We summarize motion prediction and in-between together as general motion completion. To evaluate the motion completion capability of MotionGPT, we employ part of the AMASS dataset [25], a motion-only dataset. For motion prediction task, we only input around the first 20% of the motion sequence as conditions. For in-between, we mask about 50% motion randomly for completion. We also fine-tune MotionGPT specifically for this task and employ FID, ADE, and FDE as metrics like Sec. 4.1. Furthermore, we evaluate MDM [46] on motion prediction by utilizing their provided model, which also supports motion in-between through masked motion "in-painting". The real motion data used as one of our baselines. Tab. 5 reports that our MotionGPT has the best motion completion quality and diversity.

---

Motion Prediction
Motion In-between
Methods
FID
Diversityt
ADE.
FDEL
FID 1
Diversityt
ADE!
Real
0.002
9.503
0.002
9.503
MDM[46]
6,031
7.813
6.446
8.561
2.698
8.420
3.787
T2M-GPT[57]
2.056
8.635
6.161
8.302
MotionGPT (Ours)
0.905
8.972
4.745
6.040
0.214
9.560
3.762
Table 5: Comparison of motion prediction and motion in-between on part of AMASSS [25] dataset using motion data only. FID indicates motion quality and Diversity (DIV) for motion diversity within each condition. ADE and FDE are joints distance between generation and ground truth.
Text-to-Motion
Motion-to- Text
Motion Prodiction
Motion In-between
Size
Instruction Toning
R TOP3
FID J
DIV
MMDist
Bleu 47
Cidert
FID J
DIV -+
FID
DIV -
Real
0.797
0.002
9.503
2,901
0.002
9.503
0.002
9.503
Small
0.706
0.727
9.261
2.748
12.02
Small
0.663
0.336
9.239
2,931
10.54
21.3
0.951
8.727
0.326
9.618
Base
0.722
0.365
9.407
2.821
12.47
20.2
Base
<
0.700
0.160
9.411
3.019
11.42
28.2
0.905
8.972
0.214
9.560
Large
0.691
0.231
9.310
2,776
12:44
28.3
Large
0.7U8
0,159
9.301
3,011
11.71
29.1
0.556
8.975
0.223
9.358
Table 6: Evaluation of instruction tuning and different model sizes MotionGPTss in four motion tasks on HumanML33 [10] dataset. (ef. Tab. 2 for metrics details)
4.3 Ablation Studies
MotionGPT employs T5 [36] as the motion-awars language backbone model, and we train these models with pre-training and then instruction tuning. Thus, both model size and training strategy influence the performance of MotionGPTs. We here evaluate them on the typical motion tasks. More detailed ablation studies are provided in the supplements.
Model Sizes. We evaluate the performance of models with different sizes across four motion tasks. Besides the base 220M MotionGPT in Sec. 4.1, we now evaluate 60M, 220M, and 770M MotionGPTs. Tab. demonstrates that the 220M base model has achieved remarkable performance compared to the smaller 60M model. However, the larger model size of current Motions does not yield significant improvements and, in few cases, even leads to worse results, as observed in the motion in in-between task. We believe this could be caused by the small amount of current motion datasets. HumanML3D only includes 15k motion sequences, much smaller than even billions of language and image data.
Effectiveness of Instruction Tuning. We evaluate the impact of our instruction tuning strategy different model sizes. The results in Tab. 6 demonstrate that instruction tuning enhances the versatility of MotionGPT, enabling more motion tasks like motion completion and improving the motion performance of the text-to-motion task. However, for pure text-generation tasks, the model performance is downgraded, likely due to the pair amount of textual descriptions and coupled motions.
5 Disscusion
As the first trial, to our best knowledge, exploring human motion generation using language models, the proposed MotionGPT still owns limitations as follows. MotionGPT only utilizes motion on articulated human bodies, while many other works focus on faces [16, 4], hands [39, 22, 21) and even animal [40, 62) motion. Besides, our method is also restricted to multiple humans without modeling human-object, or human-environment interactions [41]. It interesting to model the human interaction scenarios in motion motion-language framework and generate controllable motions [41].
We summarize the proposed MotionGPT as a uniform motion-language framework to generate plau- sible human motion and natural language descriptions through prompt-based instructions. Compared to the compatible motion diffusion methods [52, 46], our MotionGPT produces competitive results on motion generation, motion captioning, motion prediction, and motion in in-between using only one pre- pre-trained generative model. With the advancement of large language data and models [36, 5]. MotionGPT is also capable of addressing natural question-to-answee tasks. Extensive experiments on various human neeevat tasks demonstrate the effectiveness and extendibility of MotionGPT.

---

References
[1] Chaitanya Ahuja and Louis-Philippe Morency. Language2pose: Natural language grounded pose forecast- ing. In 2019 International Conference on 3D Vision (3DV), pages 719-728. IEEE, 2019.
[2] Stephen H Bach, Victor Sanh, Zheng- Xin Yong. Albert Webson, Colin Raffel, Nihal V Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, et al. Promptsource: An integrated development environment and repository for natural language prompts. arXiv preprint Xww::2222.00222772 2022.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot leamerr. Advances in neural information processing systems, 33:1877-1990 2020.
[4] Xuan Cao, Zhang Chen, Anpei Chen, Xin Chen, Shiying Li, and Jingyi Yu. Sparse photometric 3d face reconstruction guided by morphable models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4635-4644, 2018.
[5] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling preprint or Xiv:2210. 10. 1416, 2022.
language models. arXiv
[6] Zihang Dai, Zhilin Yang. Yiming Yang. Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdi- nov. Transformer-x xl: Attentive language models beyond a fixed-length context. arXiv preprint OrXiv:1901.080 2019.
[7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint 180.001810.04805, 2018
[8] Rohit Girdhar, Alaacldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space bind them all. arXiv preprint arXiv:2305.05665 2023.
[9] Yusuke Goutsu and Tetsunari Inamura. Linguistic descriptions of human motion with generative adversarial seq2seq learning. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 4281 4281-428 22817 IEEE, 2021.
[10] Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5152-5161, June 2022.
[11] Chuan Guo, Xinxin ZUo, Sen Wang, and Li Cheng. Tm2t: Stochastic and tokenized modeling for the reciprocal generation 3d human motions and texts. In ECCV, 2022.
[12] Chuan Guo, Xinxin Zuo. Sen Wang. Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. Action2motion: Conditioned generation of 3d human motions. In Proceedings of the 28th ACM International Conference on Multimedia, pages 2021-2029, 2020.
[13] Andrey Guzhov, Federico Raue, Jöm Hees, and Andreas Dengel. Audioclip: Extending clip to image,
text and audio. In ICASSP 2022-20222202 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 976-980, 980, IEEE, 2022.
[14] Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems. 33:6840-688 -6500, 2020.
[15] Shaohan Huang, Li Dong, Wenhui Wang. Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language models. arxiv preprint 2302v:2302. 14045, 2023.
[16] Tero Karras, Timo Aila, Samuli Laine, Antti Herva, and Jaakko Lehtinen. Audio-driven facial animation by joint end-to learning of pose and emotion. ACM Transactions on Graphics (TOG), 36(4):1 12, 2017.
[17] Jihoon Kim, Jiseob Kim, and Sungjoon Choi. Flame: Free-form language-based motion synthesis & editing. arXiv preprint arXiv:2209.003: 00009, 2022.
[18] Taku Kudo and John Richardson. Sentencepiese: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint 1rXiv:1808.06226, 2018
[19] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2lip-2: Bootstrapping language- image pre- ve-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 12597, 2023.
10

---

[20] Jumnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-imann -inge pre-training for unified vision-language understanding and generation. In International Conference Machine Learning, pages 12888-12900. PMLR, 2022.
[21] Yuwei Li, Minye Wu, Yuyao Zhang, Lan Xu, and Jingyi Yu. Piano: A parametric hand bone model from magnetic resonance imaging. arXiv preprint arXiv: 106. 10893, 2021.
[22] Yuwei Li, Longwen Zhang, Zesong Qiu. Yingwenqi Jiang, Nianyi Li, Yuexin Ma, Yuyao Zhang, Lan Xu, and Jingyi Yu, Nimble: non-rigid hand model with bones and muscles, ACM Transactions on Graphics (TOG), 41(4):1-16, 2022.
[23] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarizaatin branches out, pages 74-81, 2004.
[24] Hengbo Ma, Jiachen Li, Ramtin Hosseini, Masayoshi Tomizuka, and Chiho Choi. Multi-objective diverse human motion prediction with knowledge distillation. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8161 8171, 2022.
[25] Naurcen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons- Pons-Moll, and Michael J. Black. Amass: Archive of motion capture as surface shapes. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
[26] Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa. Clip-mesh: Generating textured meshes from text using pretrained image-text models. In SIGGRAPH Asia 2022 Conference Papers, pages 1-8, 2022.
[27] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730 27744, 2022.
[28] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing 7hu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002.
[29] Mathis Petrovich, Michael J. Black, and Güil Varol. Action-condittinnd 3D human motion synthesis with transformer VAE. In International Conference on Computer Vision (ICCV), 2021.
[30] Mathis Petrovich, Michael J. Black, and Gül Varol. TEMOS: Generating diverse human motions from textual descriptions. In European Conference on Computer Vision (ECCV). 2022.
[31] Matthias Plappert, Christian Mandery, and Tamim Asfour. The kit motion language dataset. Big Data, 4(4):236-255) dee 2016.
[32] Matthias Plappert, Christian Mandery, and Tamim Asfour. Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks. Roboticy and Autonomous Systems, (09:13-26, 2018.
[33] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMIR, 2021.
[34] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[35] Alec Radford, Jeffrey Wu, Rewon Child, David I Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 18))9 2019.
[36] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485 5551, 2020.
[37] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
[38] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and BjAJm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
11

---

[39] Javier Romero, Dimitrios Tzionas, and Michael J Black. Embodied hands: Modeling and capturing hands and bodies together. arXiv preprini arXiv:2201. 02610, 2022.
[40] Nadine Rucegg, Silvia Zuffi, Konrad Schindler, and Michael J. Black. BARC: learming to regress 3D dog shape from images by exploiting breed information. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pages 3876-3884, June 2022.
[41] Yonatan Shafir, Guy Tevet, Ray Kapon, and Amit H Bermano. Human motion diffusion as a generative prior. arXiv preprint arXiv:2303.014: 0111181,,,,,, 2023.
[42] Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang. Chen Qian, Chen Change Loy, and Ziwei Liu. Bailando: 3d dance generation by actor-critic gpt with choreographic memory. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10500-1109, 2022.
[43] Jaaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint 2ww:2000002502, 2020.
[44] Wataru Takano and Yoshihiko Nakamura. Statistical mutual conversion between whole body motion primitives and linguistic sentences for human motions. The International Journal of Robotics Research, 34(10))1314-1333 2015.
[45] Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or. Motionclip: Exposing human motion generation to clip space. Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings. Purt XXII, pages 358-374. Springer, 2022.
[46] Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Amit H Bermano, and Daniel Cohen-Or. Human motion diffusion model. arXiv preprint arXiv:2209. 14916, 2022.
[47] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397 2023.
[48] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning, Advances in neural information processing systems, 30, 2017.
[49] Ramakrishna Vedantam, C Lawrence Zätnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566-4575, 2015.
[50] Wenhui Wang, Hangho Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.1444 10442, 2022.
[51] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot leamers. arXiv preprint 11X102200..01652. 2021.
[52] Chen Xin, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, Jingyi Yu, and Gang Yu. Executing your commands via motion diffusion in latent space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), June 2023.
[53] Hu Xu, Gargi Ghosh, Po- Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding, arXiv preprint arXiv:2109. 14084, 2021.
[54] Tatsuro Yamada, Hiroyuki Matsunaga, and Tetsuya Ogata. Paired recurrent autoencoders for bidirectional translation between robot actions and linguistic descriptions. IEEE Robotics and Automation Letters, 3(4):344133445 2018.
[55] Kim Youwang, Kim Ji- Yeon, and Tae-Hyun Oh. Clip-actor: Text-driven recommendation and stylization for animating human meshes. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27 27, 2022, Proceedings, Part III, pages 173-191. Springer, 2022.
[56] Ye Yuan and Kris Kitani. Diow: Diversifying latent flows for diverse human motion prediction. In Computer Vision-ECCV ECCV 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings, Purt IX 16, pages 346-364. Springer, 2020.
[57] Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang. Hongwei Zhao, Hongtao Lu, and Xi Shen. T2m-gpt: Generating human motion from textual descriptions with discrete representations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023.
12

---

[58] Mingyuan Zhang, Zhongang Cai, Liang Pan. Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Mo- tiondifluse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208. 15001, 2022.
[59] Susan Zhang, Stephen Roller. Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068. 2022.
[60] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Weinberger, Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint 1rXiv:1904.09673 2019.
[61] Yan Zhang, Michael Black, and Siyu Tang. are more than our joints: Predicting how bodies move. In Proceedings of the JEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 3372-3382, 2021.
[62] Silvia Zufti, Angjoo Kanazawa, and Michael J. Black. Lions and tigers and bears: Capturing non-rigid, 3D, articulated shape from images. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pages 3955-3963, IEEE Computer Society, 2018.
13


Reviews:
[{"review_id": "Qvsk2m7lkI", "reviewer": "Reviewer_rpBK", "summary": "This paper introduces a motion generation pipeline called MotionGPT, which is based on GPT. MotionGPT utilizes VQ-VAE to discretize human poses into tokens and combines them with language tokens to create a unified codebook. The model is initially pre-trained on motion language data and subsequently fine-tuned on prompt-based tasks to enable it to perform various motion-language tasks.", "questions": "1. Can MotionGPT perform motion editing or motion composition similar to MotionDiffuse and MDM?\n\n2. The supplementary material states that there were only 15 users in the user study for Motion-to-Text. The number of testers for Text-to-Motion is not mentioned. However, 15 users may be considered an insufficient sample size for a reliable evaluation, especially for Motion-to-Text. It is recommended to have a larger sample size, preferably around 50 or more, to provide a more credible assessment.\n\n3. What is the reason behind the significant difference in performance between the KIT-ML and HumanML3D datasets?", "limitations": "Limitations have been well discussed.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This is the first work that explores the application of Large Language Models (LLMs) in the field of text-driven motion generation. The proposed prompt finetuning method further extends the scope of applications by including 10 different tasks. These methods provide inspiration for future research in this area.\n\n2. The performance in the Motion-to-Text task shows a significant improvement  on Bleu@4 and Cider compared to TM2T.\n\n3. The paper is well-written and effectively conveys information in a clear and understandable manner.", "weaknesses": "1. MotionGPT exhibits poorer performance in the crucial text-to-motion task, with a significant gap in FID metrics compared to T2M-GPT. Particularly on the KIT-ML dataset, there is a considerable difference in R Precision compared to T2M-GPT, MLD, and MotionDiffuse.\n\n2. The demo video provides limited comparisons with other examples. Apart from the \"crouch down\" example, the performance of MotionGPT is not noticeably superior to T2M-GPT in the provided examples. These examples do not sufficiently demonstrate an advantage in terms of generation quality.\n\n3. The overall technical contribution is limited. The major distinction from T2M-GPT lies in the combination of motion modality and language modality for modeling, along with subsequent prompt finetuning. However, in the current version, they appear more like shared intermediate layers rather than truly integrated. For instance, a potential approach could involve language-based motion editing, where given a reference motion sequence and a desired text modification, the algorithm produces the edited result. Such task types would better illustrate the advantages of unified modeling. Additionally, considering the significant performance gap between MotionGPT and T2M-GPT in standard text-to-motion tasks, MotionGPT seems to sacrifice accuracy in exchange for additional functionalities. Such technical contribution does not meet the bar set by NeurIPS conference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "📝&#8194;**Q: Motion Quality and Performance Gain** \n\n| Method    | FID $\\downarrow$ |\n|:--|:--|\n| MDM  | $0.544^{\\pm.044}$ |\n| MotionGPT | $0.160^{\\pm.008}$ |\n| T2M-GPT   | $\\boldsymbol{0.116}^{\\pm.004}$ |\n\nComparison of FID in text-to-motion task on HumanML3D dataset.\n\n| Method    | FID $\\downarrow$  |\n|:--|:--|\n| T2M-GPT   | $0.514^{\\pm.029}$ |\n| MotionGPT | $0.510^{\\pm.016}$ |\n| MDM       | $\\boldsymbol{0.497}^{\\pm.021}$ |\n\nComparison of FID in text-to-motion task on KIT-ML dataset.\n\n💡&#8194;**A:**  The FID metrics primarily focuses on the motion quality rather than the correlation between motion and text. While MDM serves as a successful benchmark for motion generation, both MotionGPT and T2M-GPT outperform MDM by a margin of 0.38~0.43 on the FID scale. However, the difference in motion quality among these three works is not significant in video supply. Additionally, MDM outperforms two vector quantized methods, MotionGPT and T2M-GPT, in terms of FID on the KIT dataset. This can be attributed to the limited number of 3,911 motion sequences, which makes it challenging to construct a comprehensive motion codebook. More importantly, MotionGPT contributes to multiple motion tasks with LLM, particularly in generating both text and motion within a single model, rather than aiming to improve the FID metric.\n\n\n📝&#8194;**Q: Performance Gain on R-Precision in KIT**  \n💡&#8194;**A:** The evaluation of R-Precision in the KIT dataset relies on the text encoder, which is built using a limited set of 6,353 textual descriptions. In contrast, MotionGPTs benefit from LLM and large language data, enabling them to generate longer and more nature language descriptions for motion. However, this leads to a discrepancy between the generated descriptions and the GT descriptions, resulting in a lower R-Precision.\n\n📝&#8194;**Q: MotionGPT seems to sacrifice accuracy in exchange for additional functionalities.** \n\n💡&#8194;**A:** As shown in Fig. 10, MotionGPT achieves SOTA on 18 out of 23 metrics across four motion-related tasks. Additionally, as mentioned by R3, both HumanML3D and KIT are limited in overall dataset size, particularly when compared to billion-level language datasets. This affects the efficacy of large-scale models. We will further employ a larger motion-text dataset to evaluate MotionGPT. Besides, MotionGPTs introduce motion-language pre-training, as well as its zero-shot ability, which is a promising direction worth exploring and could stimulate self-training procedures for further research.\n\n📝&#8194;**Q: Can MotionGPT perform motion editing or motion composition similar to MotionDiffuse and MDM?** \n\n  |Method  | FID $\\downarrow$   |DIV  $\\rightarrow$   | ADE $\\downarrow$  | FDE  $\\downarrow$|\n|:--|:--|:--|:--|:--|\n  |Real   |0.002   |9.503  |  -  | - |\n  |MDM   |6.031   |7.813  |5.446  |8.561  |\n  |T2M-GPT  | 2.056 | 8.635|6.161| 8.302 |\n  |**MotionGPT (Ours)**   | **0.905** | **8.972** | **4.745** | **6.040** |\n\nComparison of motion prediction on HumanML3D dataset using motion data only. \n\n💡&#8194;**A:**  Referring to MDM, motion editing has two categories: body part editing and motion completion in the temporal domain. MotionGPT is capable of the latter, which includes motion prediction and motion in-between. It outperforms both MDM and T2M-GPT in table above. However, when it comes to body part editing, the vector quantization(VQ)-based methods, like MotionGPT and T2M-GPT, are not as suitable as diffusion-based models that utilize diffusion inpainting on raw motion data. We agree that editing body parts with LLM and prompts is a promising direction but still needs exploration.\n\n📝&#8194;**Q: Technical contribution.**\n\n💡&#8194;**A:**  Thanks for pointing out that MotionGPT is the first work that explores motion generation with LLM. Firstly, we propose this motion-language pre-training on LLM rather than using CLIP models like previous methods. It is not a trivial combination since it needs to model and generate two distinct modes from scratch. To achieve this on MotionGPTs, we thus introduce a new training procedure: a motion-language pre-training stage and an instruction tunning stage. Secondly, we propose MotionGPT as a uniform motion-language generative pre-trained model to address various motion tasks, particularly in text-to-motion and motion-to-text. To our best knowledge, it is the first exploration of achieving such large models in the motion domain. We have developed new instruction templates and multi-task evaluation protocols, which could also contribute to the motion domain. \n\n📝&#8194;**Q: User-study**\n\n💡&#8194;**A:**  We achieve a more detailed user study to evaluate our model's performance. For text-to-motion assessment, we generated motions for 80 HumanML3D test set descriptions, comparing MotionGPTs with MDM and T2M-GPT, alongside GT. Semantic and realism studies presented text-video pairs to participants, asking which motion **corresponded better** or was **more realistic**, respectively. In the motion-to-text study, we visualized 50 GT motions with GT descriptions and generated corresponding textual descriptions using TM2T and our method. Each participant addressed a batch of questions randomly from all questions, and 19 unqualified participants among a total of 110 samples were identified and excluded by 2 'catch trials' questions. Each video pairs were reviewed by multiple participants, with a majority vote determining superior methods. Equal scores were assigned for tied results.\n\n|  Question | MotionGPT vs MDM | MotionGPT vs T2M-GPT | MotionGPT  vs GT |\n|:--|:--:|:--:|:--:|\n| Which of the two motions is more realistic? | 54% | 53% | 48% |\n| Which of the two motions corresponds better to the text prompt? | 57% | 56% | 49% |\n\n| Question | MotionGPT vs GT | MotionGPT vs TM2T |\n|:--|:--:|:--:|\n| Which description can better describe the motion? | 48% | 55% |\n\nThe results above indicate improved action quality alignment with text and motion, similar to ground truth.\n"}, {"review_id": "GnLWqw0bBd", "reviewer": "Reviewer_gizA", "summary": "This paper presents a motion-language model via a shared vocabulary, where the texts are represented by original tokens, and the motions are encoded by a trained discrete tokenizer. Based on a pre-trained encoder-decoder framework, i.e., T5, the authors fine-tune the T5 with masked modeling on motion-language paired data. Finally, the obtained model is finetuned with specific instructions for the target job (text-to-motion, motion-to-text, motion prediction, motion in-between in the paper). The experiments demonstrate the MotionGPT does such tasks well.", "questions": "- Why do you choose T5 as the base model? which is an encoder-decoder architecture. Have you tried a decoder-only model like LLaMA since it is a more straightforward solution for instruction tuning?\n- How do you implement the MDM on the motion prediction and in-between tasks? the numbers of MDM exist a large gap with text-to-motion. I mean, for diffusion models, there are lots of details in implementation. For example, reset the prefix 20% motion at each diffusion step. Lacking the details makes the numbers not fully convincing.\n- How do you merge the text vocab and motion vocab in detail? concatenating them together?\n- For tuning on each task, do you tune the entire model or just part of it?\nTable 2 is redundant since all the information is duplicated in the latter tables.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- this paper has a clear and interesting motivation, i.e., jointly learn motion-language on a token-to-token model, enabling the trained model to be aligned with the text instructions.\n- the main paper, along with the supp., provides solid and comprehensive experimental results.", "weaknesses": "- It seems the work is finished in a rush. For example, the instruction tuning is performed on each task independently, acting like a simple finetuning (the experiments also give some pieces of evidence, where it depends on the task-specific tuning). In this way, is it just a pretrain-finetune scheme? It is interesting to see any difference with the previous pretrain-finetune pipeline, such as enabling more abilities like reasoning, and zero/few-shot learning.\n- The motion prediction lacks comparison with border models like T2M-GPT, since the auto-regressive models are better at prediction.\n- Although the authors use a strong pre-trained language model, and finetune it on each task independently, it shows a limited performance gain compared with previous works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your approval and insightful comments. We will address your concerns in the following comments, re-organize this redundant Tab. 2, and update our paper accordingly.\n\n📝&#8194;**Q: Instruction tuning, Reasoning, and Zero-shot learning**\n\n💡&#8194;**A:** We propose instruction tuning to train a single MotionGPT across all motion-related tasks, while task-specific tuning is to train and evaluate MotionGPTs on a single task. We employ these two training schemes to study the ability of MotionGPT across multi-tasks. As shown in Fig. 12, we provide zero-shot cases. Benefitting from strong language models, MotionGPTs can understand unseen works in the text-to-motion training set, like \"scuttling\" and \"barriers\", and generate correct motions based on the meaning of sentences. However, it still struggles to generate unseen motions, like gymnastics, even if MotionGPTs understand the text inputs. Moreover, this reasoning provides inspired insight for our future research. We will explore this direction and provide more detailed zero-shot learning evaluations. \n\n📝&#8194;**Q: Comparison of motion prediction with T2M-GPT**: \n\n  |Method  | FID $\\downarrow$   |Diversity  $\\rightarrow$   | ADE $\\downarrow$  | FDE  $\\downarrow$|\n|:--:|:--:|:--:|:--:|:--:|\n  |Real   |0.002   |9.503  |  -  | - |\n  |MDM   |6.031   |7.813  |5.446  |8.561  |\n  |T2M-GPT  | 2.056 | 8.635|6.161| 8.302 |\n  |MotionGPT (Ours)   | **0.905** | **8.972** | **4.745** | **6.040** |\n\n💡&#8194;**A:**  We have added T2M-GPT to this comparison of motion prediction in Tab. 5. As shown in the above table, all three methods, MDM, T2M-GPT, and MotionGPTs, follow the same setting with the first 20% and generate the remaining, as mentioned in the implementation details, appendix B.5. Benefitting from larger model size, MotionGPT outperforms two other methods in this task.\n\n📝&#8194;**Q: Limited performance gain with strong language models.**\n\n💡&#8194;**A:** We thought MotionGPT, using a significantly larger language model, would surpass all existing methods in all tasks. However, the evaluation shows MotionGPT achieves SOTA results in 18 out of 23 metrics, where many improvements are only small gains. This can be attributed to the limited size of the dataset. As mentioned in R3, both HumanML3D (14,616 motions) and KIT (3,911 motions) are limited in vocabulary size and overall dataset size, particularly when compared to billion-level language datasets, which affects the efficacy of large-scale models. Benefitting from recent dataset works, like Motion-X, we will evaluate the performance gain of MotionGPT in larger datasets once they become available.\n\n\n📝&#8194;**Q: Why choose T5 as the base model? an encoder-decoder architecture. Have you tried a decoder-only model like LLaMA?** \n\n💡&#8194;**A:**  The first language model that we used to build MotionGPTs is LLaMA-13B. However, it shows insufficient performance and low training efficiency. We assume the reason is the limited dataset size compared to the large parameters and language data of LLaMA. **As shown in Tab. 15**, we tried a smaller size decoder-only backbone GPT2-Medium and provide the results. Then, we thus choose T5-770M, a small but common language model, as our final backbone, because many previous vision-language multimodal works, like Unified-IO and BLIP, have chosen T5, this encoder-decoder architecture. It shows a strong power to address multi-modal tasks. In addition, the decoder-only model has the advantage for self-supervised without pair data while we have paired data which this advance is greatly weakened. We are still working on collecting a large motion dataset for larger motion-language models.\n\n📝&#8194;**Q: How do you implement the MDM on the motion prediction and in-between tasks?** \n\n💡&#8194;**A:**  Thank you for your inquiry. We follow the approach outlined in Appendix B.4 and Line-296 of our paper, where we highlight that MDM achieves the motion in-between task using a masked motion \"in-painting\" technique. Specifically, this involves fixing the initial and final portions of the motion and allowing the model to generate the central portion. To adapt this concept for motion prediction, we similarly fix a portion of the motion – in our case, the first 20% – and generate the subsequent sequence. \n\n📝&#8194;**Q: How do you merge the text vocab and motion vocab in detail? concatenating them together?** \n\n💡&#8194;**A:**  To ensure a shared distribution between language and motion, we initialize the Motion tokens separately and concatenate them alongside the language tokens. This step ensures a balanced representation that encompasses both modalities. Besides the token embeddings are actively trained during the entirety of stages 2 and 3, ensuring a comprehensive fusion of language and motion knowledge. We will also elaborate on this concatenation in the final version.\n\n📝&#8194;**Q: For tuning on each task, do you tune the entire model or just part of it?** \n\n💡&#8194;**A:**  To address individual tasks, we adopt a focused approach where the entire model is fine-tuned. Our rationale lies in the fact that, for each specific task, our emphasis is on optimizing task-specific performance, without retaining an excessive amount of intelligence learned from other tasks. Besides, we only exclusively fine-tune the Text-to-Motion task, while other tasks are reported without specific tuning.\n"}, {"review_id": "pWov1ihDBh", "reviewer": "Reviewer_symu", "summary": "In view of the idea that motion could be perceived as a form of body language, the authors propose to fuse motion and language to perform a unified motion-language pre-training.\nIn detail, motion is quantized into discrete tokens in the same form as natural languages.\nThen, language modelling is performed on both motion and text in a unified manner.\nFurthermore, prompt tuning is adopted to fine-tune the pre-trained model.\nExperiments demonstrate the impressive performance on multiple motion tasks.", "questions": "I'm interested in the vocabulary that VQ-VAE learned. Is it possible to visualize some of the tokens? Or directly generate description on each single token?\n\nHow is the down-sample rate chosen? It is a fundamental hyper-parameter that decides the overall granularity of the model.", "limitations": "The authors provide their discussion on the limitation of the paper.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The idea of unifying motion and language into tokens for uniform pre-training is interesting and novel.\n\nThe uniform motion-language model manages to provide a solution for a wide range of tasks.\n\nThe provided demo is rather impressive and convincing.\n\nExtensive ablation studies provide a detailed analysis on the effectiveness of different design choices.\n", "weaknesses": "Though the quantized representation provides the ability to unify motion and text, it also imposes constraints on the motion representation due to the sequence-level encoding. In other words, when the operation granularity is smaller than the down-sample rate, I'm not sure whether the method could provide satisfying performance. For example, motion in-between seems to be designed as a token-level in-between. If only given a start frame and an end frame as the in-between input (which could be a more practical application scenario), would the model perform well?\n\nBoth HumanML3D and KIT are limited in the vocabulary size and the overall dataset size compared to language datasets as I know. Therefore, I understand the limited performance on KIT and when increasing the scale of the model. While in view of the recent success of LLMs, I think the authors should pay attention to unifying current available datasets to exploit the scalable potential of language models when processing large scale data besides increasing model size.\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "📝&#8194;**Q:  Motion Down-sample, if only given a start frame and an end frame as the in-between input, would the model perform well?** \n\n💡&#8194;**A:**  VQ-based methods, such as MotionGPT and T2M-GPT, employ downsampling tricky to enhance the density of the codebook or tokens and reduce computing costs. This indeed becomes a constraint when the operation granularity is smaller than the down-sample rate. However, to address this issue, only the start and end frames are provided as in-between inputs. Some technical tricks can be used, such as repeating a single start or end frame up to the window size as inputs and removing the redundant parts in outputs. This does not significantly impact the effectiveness of the model, as there are often static beginnings or endings in the GT motion data.\n\n📝&#8194;**Q: While in view of the recent success of LLMs, the authors should pay attention to unifying current available datasets to exploit the scalable potential of language models when processing large-scale data besides increasing model size.** \n\n💡&#8194;**A:**  We appreciate your insight and totally agree with this suggestion. We have faced this limited dataset issue while implementing MotionGPT and in our further research. It is a hard but valuable work to unify and collect a larger motion dataset. Foruthertaly, some researchers are working on this problem, as seen in recent work like Motion-X and other datasets, which hold promise for advancing large-scale motion models. We intend to further evaluate MotionGPT on these larger datasets once they become available. \n\n📝&#8194;**Q: Visualize some of the tokens in the vocabulary that VQ-VAE learned.**\n\n💡&#8194;**A:** As shown in Fig.13, we visualize these motion tokens in motion vocabulary $V_m$ and their corresponding localized spatial-temporal contexts, depicted within 4-frame motion segments. However, MotionGPT falls short in generating descriptions for each individual token, as the training is conducted on token sequences.\n\n📝&#8194;**Q: How is the down-sample rate chosen? It is a fundamental hyper-parameter that decides the overall granularity of the model.**\n| Downsampling | MPJPE $\\downarrow$ | MPJPE $\\downarrow$ | ACCL $\\downarrow$ | FID $\\downarrow$ | DIV $\\rightarrow$ |\n|---|---|---|---|---|---|\n| $l=1$ | 76.2 | 49.5 | 19.5 | 0.421 | 9.613 |\n| $l=2$ | **52.6** | **37.7** | **9.5** | 0.135 | 9.722 |\n| $l=4$ | 55.8 | 40.1 | 7.5 | **0.067** | 9.675 |\n| $l=8$ | 62.7 | 45.3 | 8.7 | 0.223 | **9.584** |\n\n💡&#8194;**A:** We selected the down-sample rate based on the frames-per-second (FPS) of the HumanML3D and KIT-ML datasets, which is 20 fps. Therefore, down-sampling by a factor of 4 to achieve 5 fps can ensure distinctiveness in motion frames, and prevents redundancy, and acceleration training. This choice was also made to ensure a fair comparison, as we utilized the same down-sample rate as T2M-GPT. As shown in the above table, we provide an ablation study on these parameters, where a factor of 4 achieves the best FID in motion reconstructions.\n"}, {"review_id": "0gyFF15Wud", "reviewer": "Reviewer_hFS6", "summary": "This paper proposes MotionGPT, an approach that unifies language modeling with human motion modeling by treating each new motion token the same as a language token. To achieve this, a motion tokenizer based on VQ-VAE is first learned. Then, a pretrained language model is fine-tuned to learn from a unified vocabulary of motion and language and then instruction-finetuned to perform tasks such as text-to-motion, motion completion, and motion captioning. Experiments show that the proposed model achieves state-of-the-art performance for the proposed tasks.", "questions": "Do TM2T, T2M, and poseGPT capture all human motion in their training dataset's discrete latent code? How is the reconstruction loss on ALL the training data?\n\n\n--- \n\nAfter rebuttal, my main concerns about failure analysis and text complexity are addressed and I would like to maintain a positive rating of this work. \n\n---", "limitations": "More failure cases should be discussed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of treating motion as discrete tokens for language models is novel and sound, and can enable a large number of possible applications. The showcased tasks are also comprehensive and demonstrate the flexibility of the proposed model.\n- The provided quantitative and qualitative results show state-of-the-art performance and generate natural and good-looking human motion. The multi-task ability is impressive and shows that the model can handle multiple different tasks and natural language instructions.\n- Extensive ablation is provided.", "weaknesses": "- I do find the lack of failure analysis a bit concerning. It is difficult to gauge how well the model learns the relationship between motion and language.\n- Similarly, I find the evaluated motion in the provided video and supplement to be relatively simple and clear instructions, which do not really conform to the objective of \"generalizing effectively to unseen tasks or data\". For instance, in the text-to-text demonstration, the word \"praying\" is showcased, but I wonder how well the motion generator part can handle words that have semantic meaning but could be unseen.\n\n\n\nMinor: Error T2M-GPT and MDM’s citation are mixed in Table 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your approval of our idea, human motion as a foreign language, as well as what it could enable on applications. We will fix the mixed citations, add more failure cases, and analyze the zero-shot ability of MotionGPT in the paper.\n\n📝&#8194;**Q:  Failure analysis. Zero-shot ability on handling words that have semantic meaning but could be unseen.**\n\n💡&#8194;**A:**  As shown in Fig. 12, we provide both zero-shot cases and failure cases. Benefitting from strong language models, MotionGPTs can understand unseen works in the text-to-motion training set, like \"scuttling\" and \"barriers\", and generate correct motions based on the meaning of sentences. However, it still struggles to generate unseen motions, like gymnastics, even if MotionGPTs understand the text inputs.\n\n📝&#8194;**Q: How well MotionGPT learns the relationship between motion and language?**\n\n💡&#8194;**A:** Unlike the previous motion generators using the text encoder of CLIP for conditions, please note that MotionGPTs leverage language models to learn the motion-language relationship, instead of relying on text features from CLIP. According to our zero-shot results (cf. Fig. 12) and performances on multi-tasks (cf. Fig. 10), MotionGPTs establish robust connections between simple/complex texts and simple motions in evaluations, but they fall short when it comes to complex-text to complex motion translation. \n\n\n📝&#8194;**Q:  Do TM2T, T2M, and poseGPT capture all human motion in their training dataset's discrete latent code?** \n\n| Method | MPJPE$\\downarrow$ | MPJPE $\\downarrow$ | ACCL $\\downarrow$ | FID $\\downarrow$ | DIV $\\rightarrow$ |\n|---|---|---|---|---|---|\n| VPoser-t | 75.6 | 48.6 | 9.3 | 1.430 | 8.336 |\n| ACTOR | 65.3 | 41.0 | **7.0** | 0.341 | **9.569** |\n| MLD-1 | **54.4** | 41.6 | 8.3 | 0.247 | 9.630 |\n| MotionGPT (Ours) | 55.8 | **40.1** | 7.5 | **0.067** | 9.675 |\n\n**Motion reconstruciton comparision.**\n\n| Method    | FID $\\downarrow$ |\n|-----------|-------------------|\n| MotionGPT (Ours) | $0.510^{\\pm.016}$ |\n| T2M-GPT   | $0.514^{\\pm.029}$ |\n| MLD       | $\\boldsymbol{0.404}^{\\pm.027}$ |\n\n**Comparison of FID in text-to-motion task on KIT-ML dataset.**\n\n\n💡&#8194;**A:**  Given sufficient training or testing data from the same dataset, motion reconstruction is not a challenging task for both VAE and VQ-VAE. We have provided the evaluation on motion reconstruction in Tab.8. However, when dealing with a limited amount of motion data, like the KIT dataset, the VAE model shows better ability in motion interpolation, surpassing VQ-VAE. \nA relevant evaluation is shown above (also in Tab.7), where MLD (VAE) outperforms MotionGPT and T2M-GPT (VQ-VAEs) on FID. \nThe real challenge lies in reconstructing complex motions, such as diving or gymnastics sports. Existing motion generators struggle to accurately reconstruct complex motions using a codebook extracted from daily motion datasets. Collecting these complex yet valuable motions is still a significant challenge to the motion research community.\n"}]

Author rebuttal:
We thank all the reviewers for approvals: The idea of **unifying motion and language into tokens for uniform pre-training** is **novel and sound** (R3, R4), and this motivation is **clear and interesting** (R2). This paper provides **inspiration for future research** (R1) and **impressive demo** (R3); has **comprehensive experimental results** (R2), **extensive ablation studies** (R3, R4), and **impressive multi-task ability** (R4). We will address the concerns and fix the mixed citations.
(Reviewer rpBK - R1, Reviewer gizA - R2, Reviewer symu - R3, Reviewer hFS6 - R4)

**Motivation and Novelty** :
We present MotionGPT to address various human motion-related tasks within one single unified model, by unifying motion modeling with language through a shared vocabulary. To train this unified model, we propose an instructional training scheme under the protocols for multiple motion-language tasks, which further reveals the potential of Large Language Models (LLMs) in motion tasks beyond the success of language generation. However, it is non-trivial for this combination since it needs to model and generate two distinct modes from scratch. Contrary to the previous work leveraging CLIP to extract text embedding as motion generation conditions, like T2M-GPT, MotionGPT introduces the motion-language pre-training on LLM so it can leverage the strong language generation and zero-shot transfer abilities (See Fig.12) of pre-trained language models, as well as generates human language and motion in a unified model.

**Limited Datasets and Evaluation Metrics** : Both HumanML3D (14,616 motions) and KIT (3,911 motions) are limited in the vocabulary size and the overall dataset size, also mentioned by Reviewer symu, particularly when compared to billion-level language datasets. This hampers the efficacy of large-scale models within the motion domain. The KIT dataset, with only 3,911 motion sequences, falls short in training large models with billions of parameters, as the extracted motion vocabulary struggles to represent all potential motions. Fortunately, during this review, recent works such as Motion-X, propose significantly larger motion datasets with multi-modal annotations, which hold promise for advancing large-scale motion models. We intend to further evaluate MotionGPT on these larger datasets once they become available. 

Furthermore, the metrics employed for motion-to-text evaluation, such as R-Precision in KIT, are dependent on the text encoder, which is constructed from a limited pool of 6,353 textual descriptions. In contrast, MotionGPTs benefit from LLM and large language data, enabling them to generate longer and more nature language descriptions for motion. However, this leads to a discrepancy between the generated descriptions and the ground truth (GT) descriptions, resulting in a lower R-Precision. We also notice a recent work, Text-to-Motion Retrieval (TMR), a contrastive model on text and motion, which could assist MotionGPT in more accurate text-motion evaluations.


Meta-review: This paper was reviewed by four experts in the field. Based on the reviewers' feedback, the decision is to recommend the paper for acceptance to NeurIPS 2023. The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes to the best of their ability. We congratulate the authors on the acceptance of their paper!
Decision: Accept (poster)