SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Methods	Text-to-Motion	Motion-to-Text	Motion Prediction	Motion In-between
R TOP1↑	FID↓	DIV→	R TOP3↑	Bleu@4↑	Cider↑	FID↓	DIV→	FID↓	DIV→
Real	0.511±.003	0.002±.000	9.503±.065	0.828	-	-	0.002	9.503	0.002	9.503
MLD [52]	0.481±.003	0.473±.013	9.724±.082	-	-	-	-	-	-	-
T2M-GPT [57]	0.491±.003	0.116±.004	9.761±.081	-	-	-	2.056	8.635	-	-
TM2T [11]	0.424±.017	1.501±.003	8.589±.076	0.823	7.00	16.8	-	-	-	-
MDM [46]	0.320±.005	0.544±.044	9.559±.086	-	-	-	6.031	7.813	2.698	8.420
MotionGPT (Ours)	0.492±.003	0.232±.008	9.528±.071	0.827	12.47	29.2	0.905	8.972	0.214	9.560
Table 2: Comparison of four motion-related tasks on HumanML3D [10] dataset. The evaluation metrics are computed using the encoder introduced in [10]. The empty columns of previous methods indicate that they can not handle the task. The arrows  $(\rightarrow)$  indicate that closer to Real is desirable. Bold and underline indicate the best and the second best result on text-to-motion task.
sequence-level textual descriptions. To evaluate MotionGPT as a uniform framework on tasks, such as motion prediction and motion completion (in-between), we utilize the motion sequences available in HumanML3D, which is also a subset of the larger AMASS dataset. Following the previous works [10, 52, 46], we adopt the same motion representation for fair comparisons, which combines joint velocities, positions, and rotations. By using this consistent representation, MotionGPT enables the availability to support further studies in the field. (cf. supplement for the benchmark details.)
Evaluation Metrics are summarized as four parts. (1) Motion quality: Frechet Inception Distance (FID) is our primary metric based on a feature extractor [10] to evaluate the distance of feature distributions between the generated and real motions. For motion completion, we utilize metrics used in motion prediction studies [56, 61, 24], such as Average Displacement Error (ADE) and Final Displacement Error (FDE), to evaluate the accuracy of the predicted motion. (2) Generation diversity: We utilize the Diversity (DIV) metric to assess the motions diversity, which calculates the variance through features extracted from the motions [10]. MultiModality (MM) measures the diversity of generated motions within the same text description of motion. (3) Text matching: Based on the feature space from [10], the motion-retrieval precision (R Precision) evaluates the accuracy of matching between texts and motions using Top 1/2/3 retrieval accuracy. Multi-modal Distance (MM Dist) measures the distance between motions and texts. (4) Linguistic quality: We follow [11] utilizing linguistic metrics from natural language studies, including BLUE [28], Rouge [23], Cider [49], and BertScore [60] to evaluate the quality of generated motion captions.
Implementation Details. We set the codebook of motion tokenizer as  $K \in \mathbb{R}^{512 \times 512}$  for most comparisons. The motion encoder  $\mathcal{E}$  incorporates a temporal downsampling rate  $l$  of 4. We utilize T5 [36] as the underlying architecture for our language model, with a baseline model consisting of 12 layers in both the transformer encoder and decoder. The feed-forward networks have an output dimensionality of  $d_{\mathrm{ff}} = 3072$ , and the attention mechanisms employ an inner dimensionality of  $d_{\mathrm{kv}} = 64$ . The remaining sub-layers and embeddings have a dimensionality of  $d_{\mathrm{model}} = 768$ . Moreover, all our models employ the AdamW optimizer for training. The motion tokenizers are trained utilizing a  $10^{-4}$  learning rate and a 256 mini-batch size, while our language models have a  $2 \times 10^{-4}$  learning rate for the pre-train stage,  $10^{-4}$  for the instruction tuning stage, and a 16 mini-batch size for both stages. The motion tokenizer undergoes 150K iterations of training, while the language model undergoes 300K iterations during the pre-train stage and another 300K iterations during the instruction tuning stage. Small and Base models are trained on 8 Tesla V100 GPUs while Large models are trianed on 64 Tesla V100 GPUs.
4.2 Comparisons on Motion-relevant Tasks
Comparisons on Multiple Tasks. By introducing a uniform framework that treats human motion as a foreign language, we open up the exploration of diverse motion-relevant tasks. We employ a 220M pre-trained *Flan-T5-Base* [36, 5] model as our backbone and fine-tune the model through the pre-training and instruction tuning stage (Sec. 3.3) for all following comparisons. As shown in Tab. 2, we evaluate MotionGPT against state-of-the-art methods on key tasks such as text-conditioned motion generation [52, 57, 11, 46], motion captioning [11], motion prediction [46], and motion in-between [46]. While we leverage existing results from previous works or benchmarks for text-to-motion and motion-to-text tasks, we re-implement the motion diffusion models [46] for motion prediction and evaluate it under the same metrics and settings. Please note that some methods are
Methods	RPrecision↑	FID↓	MMDist↓	Diversity→	MModality↑
Top1	Top2	Top3
Real	0.511±.003	0.703±.003	0.797±.002	0.002±.000	2.974±.008	9.503±.065	-
TM2T [11]	0.424±.003	0.618±.003	0.729±.002	1.501±.017	3.467±.011	8.589±.076	2.424±.093
T2M [10]	0.457±.002	0.639±.003	0.740±.003	1.067±.002	3.340±.008	9.188±.002	2.090±.083
MotionDiffuse [58]	0.491±.001	0.681±.001	0.782±.001	0.630±.001	3.113±.001	9.410±.049	1.553±.042
MDM [46]	0.320±.005	0.498±.004	0.611±.007	0.544±.044	5.566±.027	9.559±.086	2.799±.072
MLD [52]	0.481±.003	0.673±.003	0.772±.002	0.473±.013	3.196±.010	9.724±.082	2.413±.079
T2M-GPT [57]	0.491±.003	0.680±.003	0.775±.002	0.116±.004	3.118±.011	9.761±.081	1.856±.011
MotionGPT (Pre-trained)	0.435±.003	0.607±.002	0.700±.002	0.160±.008	3.700±.009	9.411±.081	3.437±.091
MotionGPT (Fine-tuned)	0.492±.003	0.681±.003	0.778±.002	0.232±.008	3.096±.008	9.528±.071	2.008±.084
Table 3: Comparison of text-to-motion on HumanML3D [10]. The empty MModality indicates Real motion is deterministic. These methods are sorted by FID. Pre-trained and Fine-tuned indicate uniform motion-language pre-training and specific fine-tuning on this task. (cf. Tab. 2 for notations.)
Methods	RPrecision↑	MMDist↓	Lengthavg↑	Bleu@1↑	Bleu@4↑	Rouge↑	Cider↑	BertScore↑
Top1	Top3
Real	0.523	0.828	2.901	12.75	-	-	-	-	-
TM2T[11]	0.516	0.823	2.935	10.67	48.9	7.00	38.1	16.8	32.2
MotionGPT (Ours)	0.543	0.827	2.821	13.04	48.2	12.47	37.4	29.2	32.4
Table 4: Comparison of motion captioning on HumanML3D [10]. The evaluation metrics follow [11], while we use the ground truth texts without pre-processing for linguistic metrics calculation.
designed for specific tasks, and thus some metrics are empty for tasks they cannot handle. The results presented in Tab. 2 demonstrate that our MotionGPT achieves competitive performance across all evaluated tasks, highlighting its capability to address diverse motion tasks within a single model.
Comparisons on Text-to-Motion. The text-to-motion task involves generating human motion sequences based on a given text input. We evaluate the proposed the MotionGPT model as the pre-trained MotionGPT, the same one in Tab. 2, as well as fine-tuned it on text-to-motion task. We compare our MotionGPTs with other SOTAs [11, 10, 46, 52, 57] and evaluate the performance on both HumanML3D and KIT datasets using suggested metrics [10]. The results are computed with a  $95\%$  confidence interval, obtained from 20 repeated runs. The majority of the reported results are taken directly from their own papers or the benchmark presented in [10]. Tab. 3 summarizes the comparison results, where MotionGPT achieves competitive performance on most metrics.
Comparisons on Motion-to-Text. The motion-to-text task involves generating a text description based on a given human motion sequence. We compare the pre-trained MotionGPT with recent work TM2T [11]. We evaluate the performance on the HumanML3D using the suggested metrics from [11]. Additionally, we measure the average numbers of words  $\mathrm{Length}_{\mathrm{avg}}$  for further comparisons. Please note that the reported results in [11] are evaluated with pre-processed ground truth text, which ignores the grammatical tense and plural forms of words. In Tab. 4, we directly use the ground truth text descriptions for a more accurate assessment. This comparison shows that MotionGPT overperforms recent work on text descriptions of given motions.
Comparisons on Motion Prediction and In-between. We summarize motion prediction and in-between together as general motion completion. To evaluate the motion completion capability of MotionGPT, we employ part of the AMASS dataset [25], a motion-only dataset. For motion prediction task, we only input around the first  $20\%$  of the motion sequence as conditions. For in-between, we mask about  $50\%$  motion randomly for completion. We also fine-tune MotionGPT specifically for this task and employ FID, ADE, and FDE as metrics like Sec. 4.1. Furthermore, we evaluate MDM [46] on motion prediction by utilizing their provided model, which also supports motion in-between through masked motion "in-painting". The real motion data is used as one of our baselines. Tab. 5 reports that our MotionGPT has the best motion completion quality and diversity.
Methods	Motion Prediction	Motion In-between
FID ↓	Diversity↑	ADE↓	FDE↓	FID ↓	Diversity↑	ADE↓
Real	0.002	9.503	-	-	0.002	9.503	-
MDM[46]	6.031	7.813	5.446	8.561	2.698	8.420	3.787
T2M-GPT[57]	2.056	8.635	6.161	8.302	-	-	-
MotionGPT (Ours)	0.905	8.972	4.745	6.040	0.214	9.560	3.762
Table 5: Comparison of motion prediction and motion in-between on part of AMASSS [25] dataset using motion data only. FID indicates motion quality and Diversity (DIV) for motion diversity within each condition. ADE and FDE are joints distance between generation and ground truth.
Size	Instruction Tuning	Text-to-Motion	Motion-to-Text	Motion Prediction	Motion In-between
R TOP3 ↑	FID ↓	DIV →	MMDist↓	Bleu@4↑	Cider↑	FID ↓	DIV →	FID ↓	DIV →
Real	-	0.797	0.002	9.503	2.901	-	-	0.002	9.503	0.002	9.503
Small	0.706	0.727	9.264	2.748	12.02	24.9	-	-	-	-
Small	✓	0.663	0.336	9.239	2.931	10.54	24.3	0.954	8.727	0.326	9.618
Base	0.722	0.365	9.407	2.821	12.47	29.2	-	-	-	-
Base	✓	0.700	0.160	9.411	3.019	11.42	28.2	0.905	8.972	0.214	9.560
Large	0.694	0.234	9.310	2.776	12.44	28.5	-	-	-	-
Large	✓	0.708	0.159	9.301	3.011	11.71	29.1	0.556	8.975	0.223	9.358
Table 6: Evaluation of instruction tuning and different model sizes of MotionGPTs in four motion tasks on HumanML3D [10] dataset. (cf. Tab. 2 for metrics details)
MotionGPT employs T5 [36] as the motion-aware language backbone model, and we train these models with pre-training and then instruction tuning. Thus, both model size and training strategy influence the performance of MotionGPTs. We here evaluate them on the typical motion tasks. More detailed ablation studies are provided in the supplements.
Model Sizes. We evaluate the performance of models with different sizes across four motion tasks. Besides the base 220M MotionGPT in Sec. 4.1, we now evaluate 60M, 220M, and 770M MotionGPTs. Tab. 6 demonstrates that the 220M base model has achieved remarkable performance compared to the smaller 60M model. However, the larger model size of current Motions does not yield significant improvements and, in few cases, even leads to worse results, as observed in the motion in-between task. We believe this could be caused by the small amount of current motion datasets. HumanML3D only includes 15k motion sequences, much smaller than even billions of language and image data.
Effectiveness of Instruction Tuning. We evaluate the impact of our instruction tuning strategy on different model sizes. The results in Tab. 6 demonstrate that instruction tuning enhances the versatility of MotionGPT, enabling more motion tasks like motion completion and improving the motion performance of the text-to-motion task. However, for pure text-generation tasks, the model performance is downgraded, likely due to the pair amount of textual descriptions and coupled motions.
5 Disscusion
As the first trial, to our best knowledge, exploring human motion generation using language models, the proposed MotionGPT still owns limitations as follows. MotionGPT only utilizes motion on articulated human bodies, while many other works focus on faces [16, 4], hands [39, 22, 21] and even animal [40, 62] motion. Besides, our method is also restricted to multiple humans without modeling human-object, or human-environment interactions [41]. It is interesting to model the human interaction scenarios in a motion-language framework and generate controllable motions [41].
We summarize the proposed MotionGPT as a uniform motion-language framework to generate plausible human motion and natural language descriptions through prompt-based instructions. Compared to the compatible motion diffusion methods [52, 46], our MotionGPT produces competitive results on motion generation, motion captioning, motion prediction, and motion in-between using only one pre-trained generative model. With the advancement of large language data and models [36, 5], MotionGPT is also capable of addressing natural question-to-answer tasks. Extensive experiments on various human motion-relevant tasks demonstrate the effectiveness and extendibility of MotionGPT.
[58] Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208.15001, 2022.
[59] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
[60] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019.
[61] Yan Zhang, Michael J Black, and Siyu Tang. We are more than our joints: Predicting how 3d bodies move. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3372-3382, 2021.
[62] Silvia Zuffi, Angjoo Kanazawa, and Michael J. Black. Lions and tigers and bears: Capturing non-rigid, 3D, articulated shape from images. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3955-3963. IEEE Computer Society, 2018.