SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Biao Jiang $^{1,2*}$  Xin Chen $^{2*}$  Wen Liu $^{2}$  Jingyi Yu $^{3}$  Gang Yu $^{2}$  Tao Chen $^{1\dagger}$ $^{1}$ Fudan University  
 $^{2}$ Tencent  
 $^{3}$ ShanghaiTech University  
https://github.com/OpenMotionLab/MotionGPT
Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multimodal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models,motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.
Recent years have witnessed a significant breakthrough in pre-trained large language models such as GPT [34, 35, 3, 27], BERT [7], and T5 [36, 5], which lead to the convergence of language [59, 47], image [33, 50, 20], mesh [55, 26] and multimodal [8] modeling. Nevertheless, a general pre-trained model for human motion and language has yet to emerge. This pre-trained motion-language model, capable of supporting numerous motion-relevant tasks through prompts, should benefit diverse fields like gaming, robotics, virtual assistant, and human behavior analysis.
Previous research on human motion has explored various tasks, including motion generation [29, 10, 46, 52, 57], motion captioning [9, 11], and motion prediction [56, 61, 24]. Recent text-to-motion works[46, 58, 30, 52] have attempted to employ pre-trained language-relevant models [7, 33]. For instance, MDM [46] learns a motion diffusion model with conditional text tokens from CLIP [33], while MLD [52] integrates motion latent space to improve the efficiency of motion diffusion process. On the other hand, MotionCLIP [45] and TM2T [11] concentrate on modeling the coupled relationship between motion and text description. However, the above approaches treat motion and language as separate modalities, which often require strictly paired motion and text data. Moreover, since the supervisions are task-specific, they can hardly generalize effectively to unseen tasks or data, as they lack a comprehensive understanding of the relationship between motion and language. We thus focus on building a pre-trained motion-language model, which can generalize to various tasks and learn in-depth motion-language correlation knowledge from more feasible motion and language data.
Two challenges are crucial and need to be solved for pre-training a promising motion-language model. The first is modeling the relation between language and motion, and the second is building a uniform multi-task framework that can generalize to new tasks. Fortunately, human motion exhibits a semantic
*Contributed equally and work done while Biao Jiang was a Research Intern with Tencent PCG
Figure 1: MotionGPT can address diverse motion-relevant tasks uniformly given different instructions. We provide the results on text-to-motion (the upper left), motion captioning (the bottom left), motion completion (the upper right), and the language question-to-answer (the bottom right). The left to right of motion represents the time order. Blue motion denotes the input, and yellow is the generation.
coupling similar to human language, often interpreted as a form of body language. Building upon this observation, we follow vision-language pre-training from BEiT-3 [50] to treat human motion as a specific foreign language. By integrating motion and language data together and encoding them within a single vocabulary, the relationship between motion and language becomes more apparent. Therefore, with recent significantly larger-scale language data and models, the motion-language pretraining has great potential to improve the performance on motion tasks. Meanwhile, this pre-training on language enables textual instructions like prompts in InstructGPT [27] and makes the model more versatile and user-friendly for various motion tasks.
In this work, we propose a uniform motion-language framework, namely MotionGPT, that leverages the strong language generation and zero-shot transfer abilities of pre-trained language models for doing human motion-related tasks. To enable MotionGPT to comprehend and generate human-like motions, we first learn a motion-specific vector quantized variational autoencoder (VQ-VAE) model to construct "motion vocabulary", akin to English vocabulary and then convert raw motion data into a sequence of motion tokens. These tokens are then processed by a pre-trained language model [36, 5] that learns the underlying grammar and syntax of the motion language, as well as its relationship with the corresponding textual descriptions. To effectively integrate language and motion in MotionGPT, we design a two-stage training scheme. We first pre-train the language model on the raw motion dataset to learn the basic grammar and syntax of the motion language. For prompt tuning, we fine-tune the language model on an instruction dataset, which contains both textual descriptions and motion data, to learn the correlation between the two modalities. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performance on text-to-motion, motion-to-text, motion prediction, and motion in-between.
We summarize our contributions as follows: (1) We propose a uniform motion-language generative pre-trained model, MotionGPT, which treats human motion as a foreign language, introduces natural language models into motion-relevant generation, and performs diverse motion tasks with a single model. (2) We introduce a motion-language training scheme with instruction tuning, to learn from task feedback and produce promising results through prompts. (3) We propose a general motion benchmark for multi-task evaluation, wherein MotionGPT achieves competitive performance across diverse tasks, including text-to-motion, motion-to-text, motion prediction, and motion in-between, with all available codes and data.
Human Motion Synthesis involves generating diverse and realistic human-like motion using multimodal inputs, such as text [10, 30, 58, 46, 11, 1, 17], action [29, 12, 46, 52], and incomplete motion [56, 61, 24, 46]. Text-to-motion is one of the most important motion generation tasks, due to the user
Methods	Text-to-Motion	Motion-to-Text	Motion Prediction	Motion In-between	Random Motion	Random Description
T2M-GPT [57]	✓	✗	✓	✗	✓	✗
MLD [52]	✓	✗	✗	✗	✓	✗
TM2T [11]	✓	✓	✗	✗	✗	✗
MDM [46]	✓	✗	✓	✓	✓	✗
MotionDiffuse[58]	✓	✗	✓	✓	✓	✗
MotionGPT (Ours)	✓	✓	✓	✓	✓	✓
Table 1: Comparison of recent state-of-the-art methods on diverse motion-relevant tasks. Random Motion and Random Caption represent unconstrained generation of motions and motion descriptions.
friendly and convenient language input. MDM [46] proposes a diffusion-based generative model [14] separately trained on several motion tasks. MLD [52] advances the latent diffusion model [43, 38] to generate motions based on different conditional inputs. T2M-GPT [57] investigates a generative framework based on VQ-VAE and Generative Pre-trained Transformer (GPT) for motion generation. Motion completion task generates motion conditioning on partial motions, such as classical motion prediction [56, 61, 24] or motion in-between [46], which generates the intermediate motion while the first and last parts are fixed. Although they show promising results in various human motion tasks, most above methods are limited in using a single model to handle multiple tasks. We thus propose a uniform approach that treats human motion as a foreign language, and leverages the strong language generation and zero-shot transfer abilities of pre-trained language models
Human Motion Captioning. To describe human motion with natural languages, [44] learns the mapping from motions to language relying on two statistical models. Furthermore, recurrent networks have also been used in [54, 32]. More recently, TM2T [11] proposed a new motion representation that compresses motions into a short sequence of discrete variables, then uses a neural translation network to build mappings between two modalities. While previous research like TM2T [11] incorporated captioning modules into their training pipeline for motion generation, these approaches are constrained to bidirectional translation between text and motion within one uniform framework.
Language Models and Multi-Modal. Large-scale language models (LLMs) [7, 6, 36, 3, 59, 47], enabled by extensive datasets and model size, have demonstrated impressive comprehension and generation capabilities, elevating natural language processing to new heights. BERT [7] pre-trains deep bidirectional language representations that can support downstream tasks. T5 [36] introduced a unified framework that converts all text-based language problems into a text-to-text format. More recent research [51, 2, 27, 5] find that by fine-tuning pre-trained models using input-output pairs consisting of instructions and coupled answers, the performance of pre-trained models can be further improved. FLAN [5] presents an instruction-tuning technique that surpasses the performance of non-tuned models in unseen tasks. Recently, the wave of multi-modal models [20, 15, 19] is intriguing to process text along with other modalities, such as images [20, 15, 8], audio [13, 8], and videos [53]. CLIP [33] further learns a semantic latent representation that couples images with corresponding language descriptions. Despite the success of language models in various vision-language tasks, the development of multi-modal language models that can handle human motion is still limited.
Motion Language Pre-training. Existing text-to-motion generation methods [10, 30, 46, 11, 1, 17] can be characterized as caption-to-motion, where the models take in a pure text description of the desired motion. While these methods can generate motions from textual descriptions, they are often limited in supporting instructions from users like InstructGPT [27]. In other words, they do not allow users to provide context-specific instructions for certain applications. MotionCLIP [45] utilizes the language and visual understanding of CLIP [33] to align its latent space with a motion auto-encoder. Meanwhile, many language models, such as T5[36] and InstructGPT [27], have been developed to address diverse language processing tasks, including translation, question answering, and classification. These models are typically designed to map a given text input to a target output, such as a translation or answer. However, while these models have shown remarkable performance in language tasks, they have not been widely applied to motion tasks. Therefore, we propose MotionGPT to enable the effective integration of natural language models with human motion tasks, providing a unified solution for motion synthesis problems.
Figure 2: Method overview: MotionGPT consists of a motion tokenizer  $\mathcal{V}$  (Sec. 3.1) and a motion-aware language model (Sec. 3.2). Combining Motion Tokens learned by  $\mathcal{V}$  and Text Tokens by text tokenizer, we then learn motion and language jointly utilizing language model as backbone.
To involve large language data and models in the motion generation tasks, we propose a unified motion-language framework named MotionGPT. As illustrated in Fig. 2, MotionGPT consists of a motion tokenizer responsible for converting raw motion data into discrete motion tokens (Sec. 3.1), as well as a motion-aware language model that learns to understand the motion tokens from large language pre-training models by corresponding textual descriptions (Sec. 3.2). To address motion-relevant tasks, we introduce a three-stage training scheme (Sec. 3.3) of MotionGPT for the training of motion tokenizer, motion-language pre-training, and instruction tuning.
We first propose the motion tokenizer consisting of a motion encoder  $\mathcal{E}$  and a motion decoder  $\mathcal{D}$ , to encode a  $M$  frame motion  $m^{1:M} = \{x^i\}_{i=1}^M$  into  $L$  motion tokens  $z^{1:L} = \{z^i\}_{i=1}^L$ ,  $L = M / l$ , and decode  $z^{1:L}$  back into the motion  $\hat{m}^{1:M} = \mathcal{D}(z^{1:L}) = \mathcal{D}(\mathcal{E}(m^{1:M}))$ , where  $l$  denotes the temporal downsampling rate on motion length. Then, given an  $N$  length sentence  $w^{1:N} = \{w^i\}_{i=1}^N$  describing a motion-related question or demand, MotionGPT aims to generate its answer as  $L$  length tokens  $\hat{x}^{1:L} = \{\hat{x}^i\}_{i=1}^L$ . It could be the human motion tokens  $\hat{x}_m^{1:L}$  or the text tokens  $\hat{x}_t^{1:L}$ , which results in a motion  $\hat{m}^{1:M}$  or a sentence  $\hat{w}^{1:L}$  like a description of the given motion.
3.1 Motion Tokenizer
To represent motion in discrete tokens, we pre-train a 3D human motion tokenizer  $\mathcal{V}$  based on the Vector Quantized Variational Autoencoders (VQ-VAE) architecture used in [48, 42, 11, 57]. Our motion tokenizer consists of an encoder  $\mathcal{E}$  and a decoder  $\mathcal{D}$ . The encoder generates discrete motion tokens with high informative density, while the decoder is able to reconstruct the motion tokens into motion sequences  $\hat{m}^{1:M}$ . This approach enables us to efficiently represent motion as a language, facilitating the integration of motion and language for various motion-related tasks.
Specifically, the motion encoder  $\mathcal{E}$  first applies 1D convolutions to given frame-wise motion features  $m^{1:M}$  along the time dimension, to obtain latent vectors  $\hat{z}^{1:L} = \mathcal{E}(m^{1:M})$ . Next, we transform  $\hat{z}$  into a collection of codebook entries  $z$  through discrete quantization. The learnable codebook  $Z = \{z^i\}_{i=1}^K \subset \mathbb{R}^d$  consists of  $K$  latent embedding vectors, each of dimension  $d$ . The process of quantization  $Q(\cdot)$  replaces each row vector  $b$  with its nearest codebook entry  $b_k$  in  $Z$ , written as
$$
z _ {i} = Q \left(\hat {z} ^ {i}\right) := \arg \min  _ {z _ {k} \in Z} \| \hat {z} _ {i} - z _ {k} \| _ {2}. \tag {1}
$$
After quantization, the motion decoder  $D$  project  $z^{1:L} = \{z^i\}_{i=1}^L$  back to raw motion space as the motion  $\hat{m}^{1:M}$  with  $M$  frames. To train this motion tokenizer, we follow [11, 57] to utilize three distinct loss functions for training and optimizing the motion tokenizer:  $\mathcal{L}_{\mathcal{V}} = \mathcal{L}_r + \mathcal{L}_e + \mathcal{L}_c$ , where the reconstruction loss  $\mathcal{L}_r$ , the embedding loss  $\mathcal{L}_e$ , and the commitment loss  $\mathcal{L}_c$ . To further improve the generated motion quality, we follow [57] to utilize L1 smooth loss and velocity regularization in the reconstruction loss, as well as exponential moving average (EMA) and codebook reset techniques [37]
to enhance codebook utilization during training. We provide more details about the architecture and the training of our motion tokenizer in the supplement.
3.2 Motion-aware Language Model
Employing this motion tokenizer, a human motion  $m^{1:M}$  can be mapped to a sequence of motion tokens  $z^{1:L}$ , allowing for joint representation with similar vocabulary embedding in language models [18, 36, 27]. By combining them in the unified vocabulary, we then learn motion and language jointly. We first represent motion tokens  $z^{1:L}$  as a sequence of indices  $s^{1:L} = \{s^i\}_{i=1}^L$ , where  $s^i$  corresponds to the index number of motion tokens  $z^{1:L}$ . On the other hand, previous language models, such as T5 [36], encode text as WordPiece tokens. They utilized a vocabulary of  $K_t$  word pieces and trained the SentencePiece [18] model on a mixture of language datasets.
Most previous text-to-motion [11, 52, 57] or motion-to-text [11] approaches employ different modules to handle text and motion individually, while we aim to model text and human motion together and in the same way. To achieve this, we combine the original text vocabulary  $V_{t} = \{v_{t}^{i}\}_{i=1}^{K_{t}}$  with motion vocabulary  $V_{m} = \{v_{m}^{i}\}_{i=1}^{K_{m}}$ , which is order-preserving to our motion codebook  $Z$ . Moreover,  $V_{m}$  includes several special tokens like boundary indicators, for example,  $\langle \text{som} \rangle$  and  $\langle \text{eom} \rangle$  as the start and end of the motion. Thus, we employ a new unified text-motion vocabulary  $V = \{V_{t}, V_{m}\}$ , and can formulate diverse motion-related tasks in a general format, where both input "words" and output "words" are from the same  $V$ . These "words" can represent natural language, human motion, or even a mixture of two, depending on the specific task to be solved. Therefore, our MotionGPT allows for the flexible representation and generation of diverse motion-related outputs within a single model.
To address the conditioned generation task, we employ a transformer-based model based on the architecture proposed in [36], which effectively maps the input sequences to the output. Our source input consists of a sequence of tokens  $X_{s} = \{x_{s}^{i}\}_{i = 1}^{N}$ , where  $x_{s} \in V$  and  $N$  represents the input length. Similarly, the target output is  $X_{t} = \{x_{t}^{i}\}_{i = 1}^{L}$ , where  $x_{t} \in V$  and  $L$  denotes the output length. As shown in Fig. 2, the source tokens are fed into the transformer encoder, and the subsequent decoder predicts the probability distribution of the potential next token at each step  $p_{\theta}(x_t \mid x_s) = \prod_i p_{\theta}(x_t^i \mid x_t^{<i}, x_s)$  in an autoregressive manner. Therefore, during the training process, the objective is to maximize the log-likelihood of the data distribution:
$$
\mathcal {L} _ {L M} = - \sum_ {i = 0} ^ {L _ {t} - 1} \log p _ {\theta} \left(x _ {t} ^ {i} \mid x _ {t} ^ {<   i}, x _ {s}\right). \tag {2}
$$
By optimizing this objective, MotionGPT learns to capture the underlying patterns and relationships from the data distribution, facilitating the accurate and meaningful generation of the target "words". During the inference process, the target tokens are sampled recursively from the predicted distribution  $p_{\theta}\left(\hat{x}_t^i \mid \hat{x}_t^{<i}, x_s\right)$  until the end token (i.e.,  $
$ ). This sampling strategy enables the generation of the target sequence in a step-by-step manner, where each token is probabilistically determined based on the previously generated tokens and the given source input.
3.3 Training Strategy
Since T5s have only been exposed to language data, represented within a text vocabulary  $V_{t}$ , we thus bridge motion and language and enable this language model to comprehend human motion concepts, by learning the motion vocabulary  $V_{m}$ . As shown in Fig. 3, our training scheme includes three stages: (1) Training of motion tokenizer, which focuses on learning the motion codebook to represent human motion as discrete tokens. (2) Motion-language pre-training stage, which includes unsupervised and supervised objectives to learn the relationship between motion and language. (3) Instruction tuning stage, which tunes the model based on prompt-based instructions for different motion-relevant tasks.
Training of Motion Tokenizer. We first learn the motion tokenizer using the objective defined in Equation 3.1. This training process allows any human motion sequence  $\hat{x}^{1:L}$  to be represented as a sequence of motion tokens, enabling seamless integration with textual information. Once optimized, the motion tokenizer remains unchanged throughout the subsequent stages of the pipeline.
Motion-language Pre-training Stage. The T5 models [36, 5] are trained and fine-tuned on natural language datasets with instruction-based phrasing [5, 27]. We continue to pre-train this model using a mixture of language and motions data in both unsupervised and supervised manners: 1) To generalize
Figure 3: Training Scheme. We introduce three training steps for our MotionGPT (Sec. 3.3): First  $\nu$  learn a codebook for discrete motion representation. Then we train language using a mixture of language and motion data to learn the semantic coupling between text and motion. Finally, we fine-tune the model in a multi-task text-motion dataset with instructions.
to various downstream tasks like [7, 35, 36, 27], we follow [36] to design an objective, where a certain percentage (15%) of tokens in the input tokens  $X_{s}$  are randomly replaced with a special sentinel token. On the other side, the corresponding target sequence is constructed by extracting the dropped-out spans of tokens, delimited by the same sentinel tokens used in the input sequence, along with an additional sentinel token to indicate the end of the target sequence. 2) We then learn the motion-language relation by the supervision of paired text-motion datasets [10, 31]. We train MotionGPT on the supervised motion-language translation, where the input is either a human motion or a text description. After unsupervised and supervised training processes, we aim to equip our model with the understanding of text and motion relationships.
Instruction Tuning Stage. We construct a multi-task text-motion dataset by formulating it as instructions, building upon the foundation of existing text-to-motion datasets such as HumanML3D [10] and KIT [31]. Specifically, we define 15 core motion tasks, such as motion generation with text, motion captioning, motion prediction, and others. For each task, we compose dozens of different instruction templates, resulting in more than one thousand different tasks, each having a unique instruction prompt. For example, an instruction prompt for motion generation task could be "Can you generate a motion sequence that depicts 'a person emulates the motions of a waltz dance'?" Similarly, for the motion captioning task, the instruction prompt could be "Provide an accurate caption describing the motion of <motion_tokens> ", where <motion_tokens> represents a sequence of motion tokens generated by our motion tokenizer. We have demonstrated the efficacy of instruction tuning in Sec. 4.3, which leads to improvement across various tasks and enhances the model performance for unseen tasks or prompts. More examples of prompts are provided in the supplements.
Extensive comparisons evaluate the performance of our MotionGPTs across multiple motion-relevant tasks and datasets. Details of the dataset settings, evaluation metrics, and implementation specifics (Sec. 4.1) are provided. We first present a uniform benchmark by comparing our approach with other SOTAs across various tasks (Sec. 4.2). Then, we evaluate each specific comparison on text-to-motion (Sec. 4.2), motion-to-text (Sec. 4.2), motion prediction and motion in-between (Sec. 4.2). The supplements include more qualitative results, user studies, and further implementation details.
Datasets. General motion synthesis can support diverse task settings, and thus previous datasets and a modified benchmark are utilized to evaluate MotionGPT. The study primarily focuses on two text-to-motion datasets: HumanML3D [10] and KIT [31]. The KIT dataset provides 6,353 textual descriptions corresponding to 3,911 motion sequences, while the HumanML3D dataset [10] is a more recent dataset that contains 14,616 motion sequences obtained from AMASS [25], along with 44,970