M$^3$GPT: An Advanced Multimodal, Multitask Framework  for Motion Comprehension and Generation
Authors: Mingshuang Luo
RuiBing Hou
Zhuo Li
Hong Chang
Zimo Liu
Yaowei Wang
Shiguang Shan
Venue: NeurIPS 2024 poster (2024)
Keywords: motion comprehension; motion generation; multitask; multimodal; large language model
Primary area: machine_vision

Abstract: This paper presents M$^3$GPT, an advanced $\textbf{M}$ultimodal, $\textbf{M}$ultitask  framework for $\textbf{M}$otion comprehension and generation.  M$^3$GPT operates on three fundamental principles. The first focuses on creating a unified representation space for various motion-relevant modalities. We employ discrete vector quantization for multimodal conditional signals, such as text, music and motion/dance, enabling seamless integration into a large language model (LLM) with a single vocabulary.
The second involves modeling motion generation directly in the raw motion space. This strategy circumvents the information loss associated with a discrete tokenizer, resulting in more detailed and comprehensive motion generation. 
Third, M$^3$GPT learns to model the connections and synergies among various motion-relevant tasks. Text, the most familiar and well-understood modality for LLMs, is utilized as a bridge to establish connections between different motion tasks, facilitating mutual 
reinforcement. To our knowledge, M$^3$GPT is the first model capable of comprehending and generating motions based on multiple signals.
Extensive experiments highlight M$^3$GPT's superior performance across various motion-relevant tasks and its powerful zero-shot generalization capabilities for extremely challenging tasks. Project page: \url{https://github.com/luomingshuang/M3GPT}.

Gº An Advanced Multimodal, Multitask Framework for Motion Comprehension and Generation
Mingshuang Luo 1.2.3 Luo Ruibing Hou' Zhuo Li", Hong Chang' 1.3
Zimo Liu², Yaowei Wang*., Shiguang Shan Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology. CAS. China
Peng Cheng Laboratory. China, University of Chinese Academy of Sciences, China "WeChat, Tencent Inc, Harbin Institute of Technology, Shenzhen
mingshuang- luo@vipl. ict. ac. cn, (houruibing, changhong, sgshan) @ict. ac cn albertzli@tencent com, liuzm@pel. ac. en, wangyaowei@hit edu. en
Abstract
This paper presents M&GPT, an advanced Multimodal, Multitask framework for Motion comprehension and generation. M operates three fundamental principles. The first focuses on creating a unified representation space for various motiont modalities. We employ discrete vector quantization for multimodal conditional signals, such as text, music and motion/dance, enabling seamless inte- gration into large language model (LLM) with a single vocabulary. The second involves modeling motion generation directly in the raw motion space. This strat- egy circumvents the information loss associated with discrete tokenizer, resulting in more detailed and comprehensive motion generation. Third, M learns to model the connections and synergies among various motion-relevant tasks. Text, the most familiar and well-understood modality for LLMs, is utilized as a bridge to establish connections between different motion tasks, facilitating mutual reinforce- ment. To our knowledge, GPP is the first model capable of comprehending and generating motions based on multiple signals. Extensive experiments highlight MSGPT's superior performance across various motion-relevant tasks and its power- ful zero-shot generalization capabilities for extremely challenging tasks. Project page: ptttps://Luominghhhee github. 1o/M3GPT/
Can
person is perfierting Hip Rerns Wrie Clicle
CAN you CORAIAN dase muii?
t
McGT
Anncy?
AAAAAAtE
Figure 1: GPT handle motion comprehension and generation tasks, including text-to-motion, motion- to-text, music-to-dance, danco-to-musit to-music, motion prediction, and motion in-between. The motion sequences within the dashed-line line arcas are masked the input.
"Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Motion comprehension and generation in multimodality are crucial for diverse applications, including AR/VR creation, video games, and virtual reality. Numerous studies [15, 51, 64, 52] focus on motion comprehension, including captioning 3D human motions and generating music from 3D human dances'. Recent advancements in AI [14, 44, 42, 48] have paved the way for motion generation, allowing for various control signals including textual descriptions, music pieces, and human poses. A significant shortcoming of most existing works is their focus on single-modality control signals, overlooking the potential for multimodal information integration. More importantly, the comprehension and generation of motions are predominantly studied in isolation. In reality, human motion cognition and communication indispensably require seamless transitions between any motion-relevare modalities. Therefore, it is vital to develop a unified framework for motion comprehension and generation that can efficiently utilize multiple signals simultaneously.
Recent works [12, 62, 63, 58] have shown success in developing a unified multitask motion frame- work which integrates text-driven and audio-driven motion generation through a single architecture. Employing a large language model (LLM), [60] adeptly handles multimodal control signals, such as text and single-frame pose, to generate consecutive motions. Despite their promising performance in motion generation, these approaches often fall short in comprehending motion. MotionGPT [21], a recent innovation, constructs unified motion-langued and natural language descriptions through prompt instructions. However, MotionGPT focuses solely on single non-motion modality, i.e., text. While aligning motion with one additional modality is relatively straightforward, integrating three or more modalities within a single framework and achieving bidirectional alignment among them to cover broad range of modalities for motion comprehension and generation presents a formidable challenge.
model generate plausible human motions
Two main challenges need to be solved for building unified multimodal framework for motion comprehension and generation. The first is how to create a unified representation space across different motion-relevani modalities. MotionGPT [21] and SpeechGPT [54] separately treat motion and speech as specific language for seamlessly integrating with text. Inspired by these efforts [21, 54], view both motion and music distinct forms of language, facilitating better associations with text via LLMs. Specifically, akin to language. we compress raw motion and music into a sequence of discrete semantic tokens. By encoding motion, music, and language within single vocabulary, we can build a unified representation space across these different modalities. The second is how to model the connections and synergies among various motion tasks. Different meliont-reevvan tasks are interconnected and can mutually enhance each other. Since text is the most familiar and well-understood modality for LLMs, we propose employing text bridge to establish connections between different motion tasks. Specifically, to better learn the complex music-to-dance task where both input and output modalities are unfamiliar to LLMs, we introduce two auxiliary tasks: music- to-text and text-to-dance aimed at aligning music and dance modalities with the structured text embedding space. This strategy enables us establish connections and synergies between music- to-dance and text-to-motion tasks, facilitating the alignment and collaboration of text, music, and motion/dance modalities across different tasks.
In this work, we propose a uniform Multimodal, Multitask framework for Motion comprehension and generation, namely GPP that leverages the strong language generation capability of LLMs for unifying various motion-relevant tasks, as depicted in Fig. 1. MCGPT comprises three tires. Firstly, M*GPT is equipped with multimodal tokenizers capable of compressing raw multimodal data, including motion, music, and text, into a sequence of discrete semantic tokens. These discrete representations allow the core LLM to unify motion comprehension and generation in an autoregres- sive manner, operating at the discrete semantic representation space. Secondly, different from [21, 60] that solely optimize LLM in discrete semantic space, we jointly train LLM and motion de-tokenizer, optimizing LLM in both discrete semantic space and raw continuous motion space. This operation enables the motion-space error signals from de-tokenizer to backpropagate to LLM, enhancing LLM's ability to generate the details of motion. Thirdly, we construct paired text descriptions for music, and design two auxiliary music-to-tex and text-to-dance tasks, which aid in aligning music and dance with the text embedding space. Also, we build up a shared tokenizer for motion and dance data
'In this paper, the term "motion" generally includes "dance." We distinguish them when referring to specific tasks or scenes, such text-to cext-to-motin mo-moionn and music-to-dant -ondce..
2

---

Methods
T2M
M2T
A2D
D2A
M2M
Random M
Random T
Random A
TM2D[12]
X
X
X
UDE[62]
XX
KK
EXX
Mo6
X
MotionGPT[21]
X
X
GP (Ours)
Table I: Comparison recent multimodal, multitask methods across various motion comprehension and generation tasks. T2M: text-to-motion; M2T: motion-to-text; A2D: music-to-dance; D2A: dance-to-music M2M: motion-to-motion that includes motion prediction and motion in-between. Random M. Random T, and Random A represent the unconstrained generation of motion, text, and music", respectively.
to project them into a shared semantic space. These auxiliary tasks and shared tokenizer establish connections between music-to-daned and text-to mottio, enabling mutual reinforcement.
pipeline to train GPT enhancing inter-modal alignment and effectively aligning them with human intent. To our knowledge, M is the first approach to integrate six core tasks of motion comprehension and generation- text-to- motion, notion-to-text, music-to-dance, dance-to-music, motion prediction, and motion in-between into a uniform framework. Extensive experiments demonstrate that M G achieves competitive performance across multiple motion-relevant tasks. Additionally, through qualitative results, we demonstrate that GPT possesses powerful zero-shot generalization capabilities, e.g., long-term dance generation and music-text conditioned dance generation.
We employ multimodal pre-training + instruction-tumi
2 Related Work
Motion comprehension and Generation. Many existing works focus studying human ap- pearance, pose, detection, attribute, part parsing and so on [61, 19, 45, 40, 23, 17]. This work focuses on studying human motion, including motion comprehension and motion generation. Motion comprehension involves two core tasks: motion-to-text and dance-to-music. Motion-lo-text aims to describe human motion with natural language [37]. For example, recurrent networks have been used in [37] to accomplish this task. Danse-to-mussic involves creating a piece of music from given dance [20, 27, 64]. For example, Zhun al. [64] utilizes generative adversarial network to generate music from dance videos. On the other hand, motion generation involves generating diverse human motions using multimodal inputs, such as text [44, 56, 15, 57], music [27, 18, 52, 42) and incomplete motion [31, 1, 3.. Text-to-motion is one of the most important motion generation tasks. Recent works typically map text to motion using different architectures: diffusion model [57] and autoregressive transformer model [15]. mausicto-dance focuses on generating dance movements from music. For example, [42] predicts discrete token sequences conditioned on music, which are then used to regenerate the dance sequence. Motion Completion generates motion conditioning on partial motions, such as motion prediction [31, 1] and motion-in-betwent [34, 43]. Although these methods have shown promising results in various human motion tasks, most are limited to handling a single task. Until recently, some works [12, 21, 60, 62] attempt to integrate two or more tasks into unified model, shown in Tab. 1. However, these works either lack the ability of motion comprehension [62, 60] or fail to handle music modality [12, 21J. In this work, we propose a unified motion comprehension and generation framework that can handle multiple control signals simultaneously.
Language Models and Multimodal. Large language models (LLMs) enabled by extensive datasets and model size, such as T5 [39], Flan-T5 [7], LLaMA [46], LLaMA-2 [47] and Vicuna [6], have demonstrated impressive comprehension and generation capabilities. Researchers have leveraged the capabilities of LLMs to handle multimodal tasks, expanding them to multimodal large language models (MLLMs). For example. AnyGPT [53] employs LLaMA-2 [47] to construct an any-to-any multimodal language model. NEXT-GPT [50] employs Vicuna [6] with multimodal adaptors and diffusion decoders to perform tasks across arbitrary combinations of text, images, videos, and audio. Recently, the works [21, 60] attempt to use LLMs for motion-related tasks. [60] uses LLaMA [46] to build general- purposs motion generator, which, however, lacks the ability to comprehend motion. [21] leverages T5 to construct a unified motion- llnguage model, but cannot deal with music modality.
in this paper, the term "audio" specifically refers to "music". This designation is adopted to avoid confusion between the initial letter "M" shared by both "music" and "motion," which could lead to ambiguity when these modalities represented by their initials.
*Note:
3

---

yon
copricn that person perfoems linee
Toxt Cadkbakk
Music Codibon
Moute
Medi Takcc
poloo Music
Mtuki
Music De-tobenizer
Generated
LLM
LLM
Masica Cadkbokk
Pre traisina LLM
Instrncioon Turing
Nation Token
shando
Medion De-tokenlarr
Tal
Telender
Hode
Madic
1
Stare 1: Truining Muttimodal Tokenieers
Muliindaal Vicabuaary
Triiiigg aoong Muliinodaliics and Multinak
Figure 2: An overview of the GPT framework. GP consists of multimodal tokenizers and a motion- aware language model. The training process of GPT consists of three stages: multimodal tokenizers training. modality-alignment pre-training, and instruction tuning.
Method
To enhance the comprehension and generation of motion-relevant modalities, propose unified multimodal framework named M As illustrated in Fig. 2, M consists of multimodal tokenizers responsible for compressing a motion-aware language model that learns to understand and generate motion tokens from LLMS by corresponding text and music (Sec. 3.2). To address motion-relevant tasks, we employ three-stage training scheme encompassing multimodal tokenizers training, modality-alignment pre-training, and instruction tuning (Sec. 3.3). During the inference process, multimodal tokens decoded back into their original representations by associated de-tokenizers (decoders of multimodal tokenizers), enabling various motion-relevant tasks be executed via instructions (Sec. 3.4).
motion and music data into discrete tokens (Sec. 3.1), and
3.1 Multimodal tokenizers
As shown in Fig. 2, Multimodal tokenizers aim to discretize continuous human motion and music into language-like tokens, allowing the three modalities to be unified within a single language model.
3D Human Motion Tokenizer. To represent motion in discrete semantic tokens, we build 3D human motion tokenizer based on Vector Quantized Variational Autoencoders (VQ-VAE) following [12, 62, 21, 60]. The motion tokenizer consists of motion encoder Em and motion decoder Dm along with codebook B,, = {b', .b" .......b'. b.... } containing N,, discrete semantic vectors. Notably, to facilitate mutual enhancement between motion and dance data, we employ a shared tokenizer for both motions and dances, projecting them into consistent and shared semantic space. Formally, given a 3D motion sequence m E TRImxd xdm, where Tm the time length and dm the dimensionality of each frame's 's pose, the motion encoder Em, that consists of several I-D convolutional layers projects m to a latent embeddings z E R. Xd Here, Lm is the time interval after downsampling and is the latent dimension. Next, we transform into a collection of codebook entries through discrete quantization. Specifically, the quantization process replaces each item of z with its nearest embedding in the codebook B.,. obtaining the quantized latent vectors e I from xd follows:
e = arg min 2 - 112 bkgB...
The motion decoder D.., which consists several IDD deconvolutional layers, projects the quantized embeddings back to raw motion space m Dm (e). Following [21, 60), the motion tokenizer can be trained by the reconstruction loss, embedding loss and commitment loss as follows:
Log = m ml| + |sg [x] e1 + 3 sg[e] 112
(2)
where sg [ is the stop gradient, and is the factor that adjusts the weight of the commitment loss.

---

After training the motion tokenizer, a motion sequence m can be represented as sequence of discrete codebook-indices of quantized embedding vector, namely motion tokens 9m E R as follows:
4m = arg min ArE Nm.
||Em (m) - 1l11
Music Tokenizer. For the music data, we adopt the VQ- VAE in Jukebox [9] as the music tokenizer, which consists of music encoder &a, a music decoder D, and music codebook B.. Notably, the limited number of music samples in dance datasets makes it inadequate for training an effective music tokenizer. To leverage the strong representation ability of the tokenizer trained on the large-scale musical dataset, dataset of 1.2 million songs. Specifically, we first segment each input music sample into 5-second music segments. Then, for each 5 seconds segment E Rd we use the pre-trained music tokenizer {Ea, ] to encode a into a sequence of discrete aadebook-indices qa E Rio (namely music tokens) following Eq. 3.
use the pre-trained VA- from Jukebox [9], which has been trained
3.2 Language Model Backhone
Expanding Vocabulary. To incorporate multimodal discrete representations into a pre-trained LLM, we expand the original text vocabulary V, in LLM with motion vocabulary B,, and music vocabulary B, forming new unified vocabulary V Bv, Ba }. To accommodate the expanded vocabulary, we extend the corresponding embedding and prediction layer of L.M,, where the newly incorporated parameters are initialized randomly.
Unified Multimodal Language Model. Equipped with multimodal tokenizers, we can compress multimodal data into discrete token sequences. To be specific, employing the trained motion tokenizer and music tokenizer, the input motion m E R R Xdm and music a E R R Xda can be mapped into a sequence of discrete motion tokens 9m E TT and music tokens qa E R Then equipped with unified vocabulary V, we can formulate various motion-relevan tasks in general format, where both input and output tokens come from the same vocabulary. These tokens can represent natural language, human motion, music, or any combination, depending on the specific task to be solved. This naturally enables the core LLM to unify motion comprehension and generation tasks in an autoregressive manner.
Following [21], we employ T5 [39] the language model backbone, which is pre-trained on 750 GB of text tokens. By leveraging this pre-trained large language model, we can harness its powerful modeling capabilities and generalizability to develop more user-friendly, motion-related human-computer interaction model.
3.3 Training Strategy
The training process is divided into three stages. The first stage is Multimodal Tokenizers Training, which focuses on learning the motion/music tokenizer to represent motion/music as discrete tokens. The second stage is Modality-Alignmenn Pre-training, which aims align motion, music, and text modalities, and facilitate collaboration across different motion-relevant tasks. The third stage is Instruction Fine- Tuning, aimed enhancing the model's 's instruction-followiei capability.
Stagel: Multimodal Tokenizers Training. We first train a motion tokenizer using the objective defined in Eq. 2. As for the music tokenizer, due to the limited music samples in existing dance datasets, we directly use the pre-trained VQ-VAE model from Jukebox [9]. This process allows any motion sequence and music to be represented with text within LLM. To ensure the stability of LLM training, the encoder of motion tokenizer and whole music tokenizer remain unchanged. Notably, continue to optimize the decoder of motion tokenizer in subsequent training stages to further enhance the quality of generated motions.
sequence of tokens, enabling seamless integration
Stage2: Modality Aligmment Pre-training, To enable LLM to handle discrete modalities, we utilize paired motion corpus train LLM next-token prediction task. This process aims to align the text, music, and motion modalities for unified reasoning in LLM.
Joint optimization of LLM and motion de-tokenizer. Human motion (especially dance) en- compasses intricate details. Previous works [21, 60] keep the motion de-tokenizer fixed during training I.M,, which hinders LLM's ability to perceive the distribution and details of motions. Specifically, in the output space of L.MM, different motion tokens are treated as independent classes; therefore, the cost of classifying a motion token semantic-simile token and semantic-distani
5

---

token is the same. Apparently, relying solely on LLM's autoregressive loss is insufficient for capturing the details of motion. To address this problem, we jointly optimize LLM and motion de-tokenizer in stage2 and stage3. This strategy enables the reconstruction error signals in raw motion space to backpropagate to L.IM, enhancing L.M's ability to generate the details of motion. With the goal of minimizing I.1 loss between the predicted and real motion, we search for the motion's token sequence that could minimize this LI loss in original motion space. As the motion de-tokenizer continuously optimizes, the target motion's 's token sequence, which supervises LLM training, dynamically changes. This dynamic adjustment reduces LI loss progressively, achieving joint optimization.
Synergy learning of multitasks. Although aligning text with one additional modality is relatively straightforward, integrating multiple modalities (e.g.. motion, text, and music) within a single framework poses significant challenge. Additionally, as noted in [4], multitask joint training usually achieves inferior performance on each individual task compared to single-task training. This phenomenon also observed our texi-to-motion task, as shown in Tab. 2. We argue that the large modality difference among different motion-relevan tasks (e.g., music-to-dance and text-to-motte (ot-too-moion) prevents the model from effectively establishing connections between these tasks. Thus it is difficult for the model to identify a common optimization direction that benefits all tasks.
As 'text' serves as highly semantic descriptor for other modalities and is the most familiar and well-modeled modality to LLM, we use 'text' as a bridge to align motion, text, and music data, thereby mitigating confficts in aligning multiple modalities. Initially, we construct paired textual descriptions for music samples in the dance datasets. Specifically, we use the style annotations of the music to create paired texts, such as 'a person is dancing Jazz'. Then, we construct two auxiliary tasks using the generated pairs of music and text, i.e., music-to-text and text-to text-to-dance. Through these two auxiliary tasks, GPT implicitly learns to decompose the complex music-to-dan -ance task into two simpler tasks music-to-teed to-text and text-to-dance Additionally, with a shared tokenizer for motion and dance, text-to-dance and lext-lo-motion tasks occupy the same matching space, and thus can mutually reinforce each other. In this way, M builds the synergies between the two primary motion generation tasks, music-to-dance and eext-to-motion, facilitating mutual reinforcement, as shown in Tab. 2.
Combining the above analysis, we jointly train LLM and motion de-tokenizer using a mixture of motion comprehension and generation tasks, along with two auxiliary music-to-text and text-to- dance tasks. Besides the auxiliary tasks, we consider 2 basic motion comprehension tasks, i.e., motion-to-text and dance-to-music, and 4 basic motion generation tasks, i.e., text-to-motion, music- to-dance, motion prediction and motion in- between. Formally, for a specific task, we denote the source input consisting of sequence of tokens as q, = { f.. ) the target output as q: = { Tu
LLM predicts the probability distribution of potential next token at each step pe (q'lg,' <i q.) in an autoregressive manner. For motion generation tasks, add a reconstruction loss. Specifically, when the output tokens are motion tokens, we pass them to motion de-tokenizer to generate a motion sequence (denoted as f where a reconstruction loss is then employed for guidance. Overall, during this training process, the objective is to maximize the log-likelihood of the data distribution and minimize the reconstruction error within raw motion space:
L log po (qilg,", .q.) + A +llmm ml11
(4)
where m denotes the ground-truth for m generated by motion de-tokenizer, and À is a hyper- parameter to adjust the weight for reconstruction loss.
Stage3: Instruction Fine-Tuning. To enhance the generalization and cestruction-folloiing ca- pability of MDPT, we construct multimodal instruction dataset with resort to GPT-4, building upon existing motion datasets. Specifically, we define 11 core tasks, each comprising 200/50/50 training/validation/ee instruction templates. For example, an instruction for text-to-motion task could be "Create a motion that complements the poetic elements Ciapttoo__laccholdeer standing for any text sequence; an instruction for music-to-dance could be "Script a dance that adapts to the tempo shifts in <Audio _Audio_Placcholdeeee Placeholder>", ", with <Audio <Audii_PPaceholder Placeholder> standing for any music sequence. Further details are available in Appendix B.4.
"laaacooolcee"""" with

---

3.4 Inference GPP
During inference, we evaluate GP performance across multiple motion-relevant tasks and datasets (Sec. 4.2 and Appendix (C, D). Also, we consider two challenging dance generation tasks to evaluate the zero-shot generalization ability of GPT
(1) Generating long-duration dances from long music. Long-duration dance generation involves creating uninterrupted, coherent dance sequences based on a single piece of music. Due to the limitation of computational cost and memory overload, music-to-dame tosicett-danceeeee generation. Conversely, during inference, we can combine this training task music-to- dance to generate initial 5-second dance segment, and an unseen zero-shot task dance that recursively generates subsequent dance segments conditioned on both music and previously generated dance segments, to perform long-duration and coherent dance generation.
train GPT on the task of 5-second
(2) Generating dance controlled by both music and text. Integrating music and text as control augments music-to-dance task with text modality,
signals in dance generation prussic+exx-to-ddee This process guides the generated dances to synchronize with particular actions described input texts. Thanks to the tokenizer mechanism, GPT can seamlessly combine music and text in LLM's input, enabling the integration of text instructions to produce a wide variety of dance sequences.
Experiments
4.1 Experimental setup
Datasets and Preprocessing. We use a large-scale text-to-motion dataset: Motion-X [29], and two music-to-dance datasets: +IST+ [24] and FineDance [25]. Notably, the 3D pose annotations differ among these datasets, therefore, we standardize their processing for uniform usage. Specifically, we select 22 joints common to these datasets and preprocess the motion samples following [13], resulting in motion sequences with identical representation. Further details on datasets and preprocessing are provided in Appendix B.1.
Different tasks employ distinct evaluation metrics. We use the most common evaluation metrics to assess the performance of MSG for each task. (1) Text-to-Motion. Following [2]. we use Frechet Inception Distance (FID), Diversity (Div), R- R-Precition that calculates the top-1 motion-to-text retrieval accuracy (R TOPI). (2) Motion-to-Tes
Evaluation Metries.
Following [21]. we use linguistic metrics like BLEU, CIDEr, along with R-Precision for evaluating motion-to-text alignment. (3) Music-to-Dance. Following [26, 52], use FID, Diversity and Beat Align Score (BAS) on kinetic features [22] (denoted as "k") to evaluate the dance generation quality. Notably, as noted measure of dance generation quality. So only use the kinetic features for evaluation. (4) Dance-to-Music. Following [52], we use Beats Coverage Scores (BCS), Beats Hit Scores (BHS), and Fl score to evaluate music generation quality. (5) Motion Prediction and In-Between. Following [21], we use FID and Diversity to measure the consistency between the provided pose conditions and generated motion. More details and results on other evaluation metrics are provided in Appendix B.2 and C.
in [48], the geometric features [33] are unreliable
Implementation Details. For motion tokenizer, set the codebook size to 512. As for music tokenizer, we use the pre-trained VQ-VAE from Jukebox with a codebook size of 2048. In term of temporal downsampling rate, the motion encoder uses a rate of 4, while the music encoder uses rate of 128. We utilize T5 base [39] as our language model backbone. For training the motion tokenizer, we use Adam as the optimizer with batch size of 1000 and initial learning rate of 10-4, To train the language model backbone, we employ the Adafactor_de _de optimizer and use CosineAnnealin as the learning rate scheduler. The learning rate is set to 2 10 for pre-training stage, and 10-1 for instruction fine-tuning stage. For hyperparameter settings, in Eq. is set to 0.2, and 3 in Eq. is set to 0.02. All our experiments are conducted on 8 NVIDIA A40 GPUs. To evaluate the model's performance across different platforms, we also test our trained GGP with TS-base on Ascend 910B NPUs. Further details on implementation and hyperparameter analysis are provided in Appendix E.
7

---

Table 2: Evaluation of synergy learning and joint optimization of LLM and motion de-tokenizer on Text-to- Motion (Motion-X [29]) and Music-to-Dance (AIST+ [24]). T2M: Text-to-Motion A2D: Music-to-Dance T2D: Text-to-Dance. A2T: Music-to-Text. Trained single task refers to a model trained and tested on a single task. Pre-trained and Instruction -unod indicate the model after pre-training (stage2) and instruction tuning
(stage3), followed by direct testing each task. The arrows (1) indicate that higher values better. The
arrows (4) indicate that smaller values are better. Bold indicates the best result.
Re- De-Optimizing
Text to Motion
Music to Do-ancce
Methods
motion de-tokenizer
TOP1
FID
Divt
FID; 4
Dive
BAS
Ground Truth
0.675
0.009
2.316
17.10
8.19
0,2374
Trained single task
0.645
0.081
2.124
83.33
5.18
0,1835
Trained single task
0.656
0.078
2.133
75.47
5:57
0.1884
T2M+A2D
0,564
0.094
2,080
51.26
6.73
0,2037
T2M+A2D
0.578
0.092
2.106
47.71
7.47
0.1958
T2M+A2D-12D-A2T
0.617
0.093
2.110
42.70
7.54
0,2084
T2M-A2D-T2D-AZT
/
0.626
0,088
2.197
25.24
7.63
0.2217
G (Pretrained without T2D and A2T)
0,526
0.105
2.058
40.71
7.47
0,2030
M (Pretrained without T2D and A2T)
/
0,547
0.104
2.099
37.14
7.61
0,2005
MPGPT (Pre-truined)
0.598
0.089
2.218
32.71
7.43
0.2090
G (Pre-tminedd)
/
0.601
0.092
2.251
27.65
7.52
0.2105
M (Instruction-tunee)
0.606
0.091
2.251
28.46
7.49
0,2052
M (Instraction-tuned)
0.615
0.093
2.253
24.34
7.50
0,2056
4.2 Ablation Studies
In this section, we conduct ablation studies to validate the effectiveness of our method. We use the same model architecture throughout the experiments. The ablation results are shown in Tab. 2.
Effectiveness of joint optimization of LLM and motion de-tokenizer. Different from previous works [21, 60) that fix motion de-tokenizer during training LLM, we jointly optimize LLM and motion de-tokenizer in stage2 and stage3, as detailed Sec. 3.3. As shown in Tab. 2. the joint optimization consistently brings performance gains across various metrics and most settings. Specifically, it largely enhances the fidelity of generated dances, reflected in a notable decrease in FID, score. We also notice minor increase (less than 0.003) in FID of text-to-motion task in GPT The possible reason is that the motion patterns controlled by text are relatively simple, making LLM optimized solely in discrete semantic space adequate for to-ttooooooo.... Conversely, dances involve greater complexity, necessitating the joint optimization of motion decoder to accurately capture intricate dance movements without compromising information.
Effectiveness of synergy learning of multitasks. During the training of M&GPT, we introduce synergy multitask learning strategy by constructing two auxiliary tasks: Text-to-Dance (T2D) and Ausic-to-Text (A2T), as detailed in Sec. 3.3. As shown in Tab. 2, the inclusion of T2D and A2T consistently brings performance gains across various metrics on both text-to-motion and music-to-dance tasks. Specifically, for music-to-dance, the FID; score is decreased by nearly 10 points, indicating that the synergy learning helps generate more realistic dances. We argue that by incorporating these two auxilary tasks, G implicitly learns to decompose the complex music-to-dance into two simpler tasks, music-to-text and text-to-dance. This way, the rexi-to-motion task can assist in learning the music-to-dance task, thereby enhancing its performance.
4.3 Comparisons with State-of-the-arts
In this section, we compare our M with state-of-the-arts on multiple core motion-relevant tasks. We respectively report the comparison results on text-to-motion dataset, Motion-X [29], and music-to-dance datasets, +IST+ [24] and FineDance [25]. More quantitative and qualitative results are provided in Appendix C and D. Also, in the supplementary material's 's zip file, we provide the render videos of generated motions/dances and generated music files by our MPPPT
Main results on text-to-motion dataset. On the rext-to-motion dataset, Motion-X, we evaluate MPG on 4 tasks, i.e., text-to- tottoonn, to-tet, motion prediction, and motion in-between. The comparison results are shown Tab. 3. As shown, M achieves competitive performance across all evaluated tasks, highlighting its capability to address diverse motion tasks in single model. only T2M), which combines multitask pre- training and instruction fine-tuning solely on T2M task, yields better performance than Trained single
Also, for text-to-motion task, M GPT (instruction-
8

---

Table 3: Comparison results on Motion-X [29] dataset. The evaluation metrics computed using the encoder introduced in Appendix A. Empty columns of previous methods indicate that they can not handle the task. Instruction-tunel only T2M indicates the model that is initially pre-trained on multiple tasks, followed by instruction tuning solely on text-to-motion task.
Methods
RexHoo-Mttio
Motion to
Motion Prediction
Motion between
TOPI"
FID]
DirT
TOP31
Bleus4t
CIDRM
FID,
Div?
FID)
Div?
Real
0,675
0.00g
2.316
0.881
0.009
2,316
0.009
2.366
0.61240000
0. 0.12241044
2.267
TPM [55]
0,647
0.101 stuee
2.200ttwer
0.814
1.755
MelionDiTTsse [57]
0.639
0.075
2.220
TM2T [15]]
0.58]
0.14g
2,0054mma
0.806
12.13
20.16
MDM [44]
0
0.07g
2.12g
1.028
1.746
0.831
1.768
McMask [16]
0,668 Sou0
0.004
2.24] Anons
0.626
1.884
Mc [8]
0.658-000s
0.07g-40000
2.206
211
0669
0 0.07gg41000
2.16
11.21
0.701
1,818
0.648
1.875
Tainnd single
0,656
0.07g 1ooes
2.133
0.767
22,92
0.774
1.778
0.692
1,810
McGPT (Pre-trainod)
0.60] 10000
4.002:40000
2.25] A0000
24.12
1,874
0.604
1.579
McGTT (Instractioo-mmedd)
0.615
0.0gg
2.25g
0.845
11:50
42,93
0.682
1.838
0.612
1,900
MRGPT (Ietractionstmedd only T2M)
0.661
4.076
2.273 2.277
Table 4: Comparison results on Motion-X [29] dataset based on Ascend 910B NPUs.
Methods
Textion-ottion
Motion-to-Test Bleuir
Metion Prediction
Motion In-between
TOPI
PID
Dart
TOP31
CIDbrt
FID.
Div?
FIDI
Dar
Trained single task
0.654
0.08 stoues
2.30g
0.763
10.16
22.89
0.776
1.818
0.712
1,80
MPGPT (Pre-trinee))
0.596 40402
0.096 souce
2.241 Aooms
0,831
11.03
21.02
0.710
18882
0.608
1.874
MPGPT (Instruction: tuned)
0,612
0.094
2.276
0.846
11.52
42.64
0.684
1.841
0.621
1.903
MºGPT (lostructive-tmeed only T2M)
0.659
0.078 tou0s
2.314 t0003
Table 5: Comparison results on +IST+ [24] and FineDance [25] datasets.
Music-to-Danse Dan-c on AIST++
Music-to-Dance on FineDance
Dance-to-Music on AIST+
Mothods
FDD;
Div,
BAS 1
FID,
Div, 1
BAS
BCS1
BHSt
FI
Real
17.10
10.60
0,2374
0,2120
FACT [24]
35.35
5.94
0,2209
113.38
3.36
0.1831
Bailando [42]
28.16
7.83
0.2332
82.81
7.74
0,2029
EDGE [48]
42.16
3.96
0,2334
94,34
8.13
0.2116
Lodge [26]
37,09
5,58
0,2423
45.56
6.75
0.2397
Foley [1]]
96.4
41,0
57.5
CMT [10]
97.1
46.2
62.6
D2MGAN 164]
95.6
88.7
93.1
CDCD [65]
96.5
89,3
92,7
LORIS [52]
98.6
90.8
94,5
Trained single task
75.47
5.57
0.1884
128.37
6.48
0.2036
93.9
93.6
92.8
MºGPT (Pre-trainod)
27.65
7.52
0.2105
92.35
7.67
0,2134
93.4
93.8
94.2
MCPT (Instruction-tuned)
24.34
7,50
0.2056
86.47
7.75
0,2158
93.6
94.0
94.9
MºGPT (Instruction-tumn) for single task)
23.01
7.85
0.2261
42.66
8.24
0.2231
94.3
94.0
95.0
Table 6: Comparison results on + [24] and FineD [25] datasets based on Ascend 910B NPUs.
Methods
Music-to- Dance on AIST+
Musie-to- on FineDance
Dance-to-Minn on AIST++
FID;
Div f
BAS
FID,
Div 1
BAS
BCS1
BHSt
F1f
Trained single task
77.32
5.61
0.1860
134.66
6.52
0.2088
93.8
93.6
92.7
McGPT (Pre-trainee trained))
27.99
7.61
0.2102
91.39
7.71
93.4
93.6
94.2
McGP (Instruction tuned)
25.05
7.48
0,2072
88.25
7.76
93.5
93.8
94.7
MPGP (Instruction-tume single task)
23.68
7.83
0.2264
43.78
8.39
94.3
94.0
94.9
task that only trains the model on T2M task. Tab. 4 presents the results of testing the Ascend 910B NPUs, where MDGPT also achieves comparably good performance. These results demonstrate that multitask pre-training platforms. Further results of GP trained on NPUs will be presented later.
enhance the performance of individual tasks across different computation
Main results on music-to-dance datasets. On the music-to-dance datasets, AIST+ and FineDance, we evaluate M on tasks, i.e., music-to-dance and dance-to-musie. As shown in Tab. 5, in general, the performance of multitask pre-training and instruction fine-tuning in MºGPT outperforms single-task training, underscoring the effectiveness of multitask training for dance-relevant tasks. Also, GP achieves competitive performance on most metrics. For music-to-danee -a-ce,, the best-performing

---

Transidd
Tunel4?
Furalel
Framai44
Fraale3
PaxelS:
TuralSt
<Masietto
<ussic Dunce-to-Damcpe
Generiic
<Long Danee
Hefoii]oooooo "A person does cartaheel."
"
Generate
Dancee
Figure 3:
Qualitative results for long long-term dance and music-text conditioned dance generation of M GPP...
method on the FineDance dataset is Lodge [26]. This approach features specialized two-stage architecture for generating long-duration dance sequences, progressively refining movements from coarse fine granularity using diffusion model. On AIST++ dataset, M reports the best FID; of 24.34 for music-to-dance task, the best BHS and F1 94.0 and 94.9 for dance-to-music task. The results in Tab. 6, tested the Ascend 910B NPUs, also demonstrate that multitask training can enhance the performance of both music-to-dance and dance-to-music tasks.
4.4 Evaluation on Zero-Shot Tasks
In this section, explore MDGPT's capabilities in handling zero-shot tasks, mentioned in Sec. 3.4. Fig. 3 (a) shows the long-term dance generation. As seen, M$GPT can generate a coherent dance sequence by seamlessly integrating the music-to-dance and zero-shot music=dance-to-dune tasks. Fig. 3 (b) shows the generated 3D dance with both music and text instruction. We can see that M maintains plausible visual results in accordance with input text instructions (cartweel), underscoring its remarkable zero-shot generalization capability.
Conclusion
In this paper, we present G a unified framework for comprehending and generating motion aligned with both text and music modalities. By employing text bridge, we build connections and synergies between different motion- relevann tasks, facilitating mutual reinforcement. Additionally, the joint optimization of L.LM and motion de-tokenizer further enriches the details of generated motion, enhancing overall motion generation quality. Our extensive evaluations across various motion-relevant tasks demonstrate the effectiveness of MDGPT both motion comprehension and generation. Besides, M exhibits strong zero-shot generalization abilities, enabling it to handle previously unseen and challenging motion-relevant tasks.
Limitations and Broader Impacts. Although our G has successfully processed signals from motion, text, and music modalities for motion comprehension and generation, focuses on modeling human body movements, excluding hands and faces details. Future research can extend the scope of M&GPT to include hands and facial modeling-
Acknowledgments. This work is sponsored by the National Natural Science Foundation China (NSFC) under Grant 62306301, the National Postdoctoral Program for Innovative Talents under Grant BX20220310 10, and the CAAI-CANN Open Fund developed on the OpenI Community. It is also supported by the project of Peng Cheng Laboratory (PCL2023A08).
References
[1] Judith Butepage, Michael J Black, Danica Kragic, and Hedvig Kjellstrom. Deep representation learning human motion prediction and classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6158 6166, 2017.
[2] Ke Chen, Yusong Wu, Haohe Liu, Marianna Nezhurina, Taylor Kerg-Kirkpatrick, and Shlomo Dubnov. Musicldm: Enhancing novelty text-to-music generation using heat-synchronous mixup strategies. ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1206-1210. IEEE, 2024.
[3] Ling-Har Hao Chen, Jiawei Zhang, Yewen Li, Yiren Pang. Xiaobo Xia, and Tongliang Liu. Humanmac: Masked motion completion for human motion prediction. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 9544-9555, 2023.
10

---

[4] Ting Chen, Saurabh Saxena, Lala Li. Tsung- Yi Lin. David Fleet, and Geoffrey E Hinton. A unified sequence interface for vision tasks, Advances Neural Information Processing Systems, 35:31333-31346, 2022.
[5] Xin Chen, Biao Jiang. Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu. Executing your commands via motion diffusion in latent space. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 18000-18010, 2023.
[6] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, march 2023. URL https://llmys.
3(5), 2023.
[7] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetune language models. Journal of Machine Learning Research, 25(70): 53, 2024.
[8] Wenxun Dai, Ling-Hao Chen, Jingbo Wang. Jinpeng Liu, Bo Dai, and Yansong Tang. Motionlem: Real- time controllable motion generation via latent consistency model. arXiv preprint 1rX1v:2404.19759 2024.
[9] Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint arXiv:2005.0034) 2020.
[10] Shangzhe Di, Zeren Jiang, Si Liu, Zhaokai Wang, Leyan Zhu, Zexin He, Hongming Liu, and Shuicheng Yan. Video background music generation with controllable music transformer. In Proceedings of the 29th ACM International Conference on Multimedia, pages 2037-2045, 2021.
[11] Chuang Gan, Deng Huang, Peihao Chen, Joshua B Tenenbaum, and Antonio Torralba. Foley music: Learning generate music from videos. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pages 758-775. Springer, 2020.
[12] Kehong Gong, Dongze Lian, Heng Chang, Chuan Guo, Zihang Jiang, Xinxin Zuo, Michael Bi Mi, and Xinchao Wang- Tm2d: Bimodality driven 3d dance generation via music-text integration. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 9942-9952, 2023.
[13] Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5152 5161, 2022.
[14] Chuan Guo, Shihao Zou, Xinxin Zuo Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5152-5161, 2022.
[15] Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng. Tm21: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts. In European Conference on Computer Vision, pages 580-597. Springer, 2022.
[16] Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. Momask: Generative masked modeling of 3d human motions. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition. pages 1900-1910, 2024.
[17] Hao Guo, Xiaochuan Fan, and Song Wang. Visual attention consistency for human attribute recognition. International Journal of Computer Vision, 130(4):1088 1106, 2022.
[18] Bo Han, Yi Ren, and Yuheng Li. Dance2midi: Dance-driven multi-instrummets music generation. arXiv preprint 13X1::3301.09880 2:3, 2023.
[19] Ruihing Hou, Bingpeng Ma, Hong Chang, Xingian Gu, Shiguang Shan, and Xilin Chen. Feature completion for occluded person re-identification. IEFE Transactions on Pattern Analysis and Machine Intelligence, 44 (9)44844-912,, 2021.
[20] Ruozi Huang, Huang Hu, Wei Wu, Kei Sawada, Mi Zhang, and Daxin Jiang. Dance revolution: Long-term dance generation with music via curriculum learning. arXiv preprint arXiv:2006.06/ /9, 2020.
[21] Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as foreign language. Advances in Neural Information Processing Systems, 36, 2024.
[22] and Jessica K Hodgins Kensuke Onuma, Christos Faloutsos. Fidistance: A fast and effective distance function for motion capture data. In Eurographics, pages 83- 2008
11

---

[23] Peike Li, Yunqiu Xu, Yunchao Wei, and Yi Yang. Self-correction for human parsing. IEEE Transactions Pattern Analysis and Machine Intelligence, 44(6):3200-3271, 2020.
[24] Ruilong Li, Shan Yang. David A Ross, and Angjoo Kanazawa. Ai choreographer: Music conditioned 3d dance generation with aist++. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 13401-13412, 2021.
[25] Ronghui Li, Junfan Zhao, Yachao Zhang, Mingyang Su, Zeping Ren, Han Zhang, Yansong Tang, and Xiu Li. Finedance: A ane-grained choreography dataset for 3d full body dance generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10234-10244 2023.
[26] Ronghui Li, YuXiang Zhang, Yachao Zhang, Hongwen Zhang, Jie Guo, Yan Zhang, Yebin Liu, and Xiu Li. Lodge: A coarse to fine diffusion network for long dance generation guided by the characteristic dance primitives. arXiv preprint arXiv:2403. 10518, 2024.
[27] Sifei Li, Weiming Dong. Yuxin Zhang, Fan Tang, Chongyang Ma, Oliver Deussen, Tong- Yee Lee, and Changsheng Xu. Dance-to-music generation with encoder-based textual inversion of diffusion models. arXiv preprint arXiv:2401.17800 2024.
[28] Chin- Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
[29] Jing Lin, Ailing Zeng. Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang. Motion-x: A large-scale 3d expressive whole-body human motion dataset. Advances in Neural Information Processing Systems, 2023.
[30] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pages 2023.
[31] Julieta Martinez, Michael J Black, and Javier Romero. On human motion prediction using recurrent neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2891-2900, 2017.
[32] Brian McFee, Colin Raffel, Dawen Liang, Daniel PW Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto, librosa: Audio and music signal analysis in python. In SciPy, pages 18-24, 2015.
[33] Meinard Müller, Tido Röder, and Michael Clausen. Efficient content-based retrieval motion capture data In ACM SIGGRAPH, pages 677-685. 2005.
[34] Boris N Oreshkin, Antonios Valkanas, Félix G Harvey, Louis-Simon Ménard, Florent Bocquelet, and Mark Coates. Motion in-betweening via deep delta-interpolator. IEEE Transactions on Visualization and Computer Graphics, 2023.
[35] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 31 3-3118 2002.
[36] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael Black. Expressive body capture: hands, face, and body from single image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10975-10985, 2019.
[37] Matthias Plappert, Christian Mandery, and Tamim Asfour. Learning a bidirectional mapping between human bodye-body motion and natural language using deep recurrent neural networks. Robotics and Autonomous Systems, 109:13-24 26. 2018.
[38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. International conference on machine learning, pages 8748-8763, PMLR, 2021.
[39] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1667 2020.
[40] Mikel Rodriguez. Ivan Laptev, Josef Sivic, and Jean-Yves Audibert. Density- aware person detection and tracking in crowds. In 2011 International Conference Computer Vision, pages 4423-2430. IEEE, 2011.
12

---

[41] Javier Romero, Dimitrios Tzionas, and Michael J Black. Embodied hands: Modeling and capturing hands and bodies together. arXiv preprint urXiv:2201.0266 2022.
[42] Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang, Chen Qian, Chen Change I. Loy, and Ziwei Liu. Bailando: 3d dance generation by actor-critic gpt with choreographic memory. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 11050-11059, 2022.
[43] Paul Starke, Sebastian Starke, Taku Komura, and Frank Steinicke. Motion in-hetweening with phase manifolds. Proceedings of the ACM on Computer Graphics and Interactive Techniques, 6(3):1-17, 11.1-17,.50 2023.
[44] Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffusion model. In The Eleventh International Conference on Learning Representations, 2022.
[45] Alexander Toshev and Christian Szegedy. Deeppose: Human pose estimation via deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1653-1660, 2014.
[46] Hugo Touvron, Thibaut I Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Erie Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 1371,, 2023.
[47] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Ilama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 09288, 2023.
[48] Jonathan Tseng, Rodrigo Castellon, and Karen Liu. Edge: Editable dance generation from music. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 448-458, 2023.
[49] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 0666-455,.20 2015.
[50] Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal Ilm. arXiv preprint (tX.:i333909011999 2023.
[51] Tatsuro Yamada, Hiroyuki Matsunaga, and Tetsuya Ogata. Paired recurrent autoencoders for bidirectional translation between robot actions and linguistic descriptions. IEEE Robotics and Automation Letters, 3(4): 3441-3448, 2018.
[52] Jiashuo Yu, Yaohui Wang, Xinyuan Chen, Xiao Sun, and Yu Qiao. Long-term rhythmic video soundtracker. In International Conference on Machine Learning. pages 00339-40353. PMLR, 2023.
[53] Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, et al. Anygpt: Unified multimodal Ilm with discrete sequence modeling. arXiv preprint arXiv:2400.12220
2024.
[54] Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities. arXiv preprint arXiv:2305. 11000 2023.
[55] Jianrong Zhang. Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, and Xi Shen. T2m-gpt: Generating human motion from textual descriptions with discrete representations. arXiv preprint arXiv:2301.0660 2023.
[56] Jannong Zhang, Yangsong Zhang, Xiaodong Cun, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xi Shen, and Ying Shan. Generating human motion from textual descriptions with discrete representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14730- 14740, 2023.
[57] Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong. Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. IEEE Trunsuctions on Puttern Analysis and Machine Intelligence, 2024.
[58] Mingyuan Zhang, Daisheng Jin, Chenyang Gu, Fangzhou Hong. Zhongang Cai, Jingfang Huang, Chongzhi Zhang, Xinying Guo. Lei Yang. Ying He, et al. Large motion model for unified multi-modal motion generation. arXiv preprint arXiv: 204.01244, 2024.
[59] Yan Zhang, Michael Black, and Siyu Tang. We are more than our joints: Predicting how 3d bodies move. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3382, 2021.
13

---

[60] Yaqi Zhang, Di Huang. Bin Liu, Shixiang Tang, Yan Lu, Lu Chen, Lei Bai, Qi Chu, Nenghai Yu, and Wanli Ouyang. Motiongpt: Finetuned Ilms are general- purpose motion generators. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7368-7376, 2024.
[61] Jiahe Zhao, Ruibing Hou, Hong Chang, Xinqian Gu. Bingpeng Ma. Shiguang Shan, and Xilin Chen. Clonhes- person re-identification with feasibility-awe aware intermediary matching, arXiv preprint arXiv:2404009507, 2024.
[62] Zixiang Zhou and Baoyuan Wang. Ude: unified driving engine for human motion generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5632-56600 5641, 2023.
[63] Zixiang Zhou, Yu Wan, and Baoyuan Wang. A unified framework for multimodal, multi-part human motion synthesis. arXiv preprint arXiv:2311. 16447,,, 2023.
[64] Ye Zhu, Kyle Olszewski, Yu Wu, Panos Achlioptas, Menglei Chai, Yan Yan, and Sergey Tulyakov. Quantized gan for complex music generation from dance videos. In European Conference Computer Vision, pages 182-199. Springer, 2022.
[65] Ye Zhu, Yu Wu, Kyle Olszewski, Jian Ren, Sergey Tulyakov, and Yan Yan. Discrete contrastive diffusion for cross modal music and image generation. arXiv preprint arXiv:2206.07771, 2022.
14

---

Appendix
A Text-Motion Aligmment Model
Due the lack of powerful and publicly available text-motion alignment model, we independently leverage existing datasets to develop a functional text-motion alignment model, which used to evaluate tasks such as text-to-motion, motion-to-text, motion prediction, and motion in-hetween. We adopt the motion encoder architecture Em from [14] and use the pretrained CLIP [38] ViT-B/32 model for the text encoder E,. As depicted in Fig. 4, the training of the lext-motion alignment model is split into two phases: pre-training the motion auto-encoder and texi-motion contrastive learning. During the pre-training phase, we use motion data from the eexl-to-motion and dance data from the music-to-dance tasks. This stage employs reconstruction loss to ensure the model achieves a robust initial state capable of extracting an expressive motion representation. In the text-motion contrastive learning phase, utilize text-motion pair data from the text- to-motion task. To train a more robust text- motion alignment model, we employ text eext-motion training dataset data along with motion data reconstructed by Motion VQ-VAE for text-motion contrastive training. We incorporate an adapter MI.P layer into both the motion and text encoders to align the dimensions of zm and x; at 512. This setup facilitates the alignment of text and motion in the representational space. The motion reconstruction loss Tracon motion for the pre-traie pre-training stage and the contrastive learning loss LinfeNCE for the aligning stage are used optimize this model, as follows.
Lrecon_ motion = || all
(5)
exp((zE, 2/1/T)
LiniONCE
log
(6)
N 1m1
Σラ= exp((z/, 251/T)
G
Li--i AAAAAAAA
E.
E.
Inper Mation
Fra parko
peront perforts bree nol tock
E
Learning
Figure 4: Pipeline of Text-Motion Aligument Model, The training of the text- text-motion alignment model includes two stages: pre- training motion auto-encoder and text-motion contrastive learning.
B Details for Training and Evaluating
B.1 Data Introductions and Preprocessing
We leverage the largest available text-to-motion dataset, Motion-X [29], along with widely-used music-to- dance datasets, AIST++ [24] and FineDance [25], for multitask training regimen. Motion-X is used for text-to-motion, motion-to-text, motion prediction, and motion in-hetween tasks, while AIST+ and Finelance datasets support both music-to-dance and dance-to-music tasks, and are also adapted for motion prediction and in-between tasks to enhance our training resources.
Motion-X includes 15.6 million precise 3D whole-body SMPL-X [36] pose annotations across 81. motion sequences with sequence- semantic text descriptions. AIST+ contains 1,409 dance motion sequences across 10 dance genres with SMPL [30] pose annotations, and FineDance provides 7.7 hours dance, totaling 331,600 frames with SMPL-H [41] poses at 30 fps across 16 dance genres. Tab. shows the training datascts and their corresponding sample numbers that we use to train our model.
We standardize data across these datasets by selecting the 22 common joints and normalizing each motion sequence to face the same direction and to run at 30 fps. We use processing technique consistent with prior research [14, 5, 21] that integrates joint velocities, positions, and rotations for consistent motion representation, facilitating effective utilization across tasks. For the music data, we use the Librosa toolkit [32] to load raw wwvv data at sampling rate of 22050 Hz, processed into features by the Jukebox encoder [9]. To optimize the use of these datasets, we strategically employ data from specific datasets for different tasks. During training for music and dance tasks, we randomly select 5-second segments from complete music tracks and corresponding dance
15

---

segments as training samples, setting the sample length to 6.25 seconds for motion prediction and in-between tasks with + and FineDance. When assessing all 5-second samples directly. Instead, we generate continuous long dance sequences using music-to-dance and motion prediction, then segment these into 30-second samples for evaluation to align with Lodge's 's testing methodology.
on the FineDance dataset, we don't evaluate
Table 7: The training datasets and sample numbers for different tasks.
Tisks
T2M
Mation Predictionnla-eeteer
Training donaee Training samples number
Metion X
Metion 64867
Motion NAIST
AlST==//neeDance 952/177
AlSt=//lineDance 952/777
AIST Aiseeelinee AlS---/inee 9521177
64867
6686 18621177
952477
B.2 Comprehensive Evaluation Metrics
Different tasks utilize specific evaluation metrics. We use the consistent evaluation metrics following prior research [14, 21, 12, 26, 64, 52].
Text-to-Mation. We measure the discrepancy between generated and actual motion features using Frechet Inception Distance (FID), assess diversity (Div) among all generated motion sequences, and evaluate motion-text semantic correlation with R-precision. Multimodal Distance (MM Dist) quantifies the disparity between motions and texts. specialized model developed for evaluating the text-to-motion task on the Motion-X dataset with 22 joints detailed in Appendix A.
Motion-to Textx. We use linguistic metrics including BLEU [35], ROUGE [28], CIDEr [49], and BertScore [59], along with R- Preriiiio and MM Dist to assess alignment between generated text and motion.
Mluicctto-Dance We employ the evaluation framework recommended by FACT [24] and Bailando [42], utilizing FID, Diversity, and Beat Align Score (BAS) gauge dance quality. our paper, we use kinetic features to compute FID and Diversity.
Dance-to- tusii.. We use metrics from [52, 64] such as Beats Coverage Scores (BCS), Beats Hit Scores (BHS). F1 scores, Beats Coverage Std (CSD), and Beats Hit Std (HSD) to evaluate music generation quality.
Motion Prediction and Motion In-between. We use Average Displacement Error (ADE) and Final Displacement Error (FDE) to assess the quality of predicted motion. The text-motion alignment model aids in evaluating motion prediction performance.
B.3 Distributed Training for Multitasks
We employ single-node multi-GPU distributed training strategy for M GPG distributing each task across separate GPUs to facilitate multitask training through shared model parameters. This method allows us to tailor the maximum token length of the LLM for each task, based on the longest sample sequence typical for that task. Specifically, set the maximum LLM token length to 192 for tasks such text-to motion, motion-to -o-t-tt--tttttttte motion prediction, and motion in- ne-eeween. For tasks involving the music modality, such as music-to-dance and Lance-to-music the maximum length is set at 980. This task-specific configuration enables us to optimize batch sizes effectively, thus maximizing GPU utilization.
In our experimental setup, the hatch size is set to 40 for the text-to-motion, text-to-dance, and motion-to-text tasks, and 4 for the music-to-dance, dance- fance-to-music, and music-to-text tasks. For motion prediction and motion in-between tasks, the batch size set to 45. We also establish the number of 1,000,000. instruction fine-tuning at 200,000. This structured approach ensures that each task is optimally processed, enhancing the efficiency and effectiveness of our training regimen.
iterations for pre-training
B.4 Tasks for Pre-training and Instruction tuning
As shown in Fig. 5, we define 11 core notion-related tasks for the instruction tuning of GPT, including text-to-motion, random--o-mootion. motion-to-text, random-to-lext, motion prediction, motion in-between, music- to-dance, dance-to-mund to-music, random-to-music. texi-to-dance, and music-to-text. The tasks of text-to-dance and music-to-text were specifically constructed based on the music-to-dance datasets. Random-to-X represents the unconstrained generation of motion, text and music. In Tab. 8, we present a selection of command templates for each task, where MMMotion_laachholl// represent the motion sequence (including dance sequence), textual description, and music segment from the training data.
<Caption Plachoolleaaccc.ere...... and <Music CattusicPaaccooderrrrr respectively
C Quantitative Results and Comparisons with SOTA Methods
In this section, we compare the performance of our G' with existing SOTA methods on a broader set of metrics for each task across three datasets: Motion-X [29]. AIST+ [24], and Fine [25]. Tab. shows
16

---

M&GPT
Music-to-Dance
Text-to-Motion
Motion-to-Tee Text
Motion-to-Monn Mootio
Dance-to-Min
Text-to-Dance
text-to-motion
mation-to-text
motion prediction
dance to music
motioo
random-to-test
in-between
random-lo-music
Music-to- Text
Figure Tasks M pre-training and instruction tuning. Random represents the unconstrained generation of motion/texu/musi in the corresponding task.
Table 8: Examples of instruction templates for each task when instruction tuning G
Tisk
Instruction template
Design motion that illustrates the emotions conveyed by <Caption Placebolder>. mentioned <Cuption_ Placeholder> through motion? Can you develop a motion that captures the existential debate in <Caption Placeholders?
exxt-40-mie
could you express
Can you generati a motion randomly?
mandom-to-nnn moniooo-----iiinn
Please generate random motion.
Placho_P
Display motion me.
What themes are explored through the motion in <Motion Placeholderz?
mation-to-tee
Can you describe the motion eMation Placeholders with texts?
Cuptionn_Paachohleee
How would you interpeet the actions depicted in <Motion Placeholdas??
description Give me cuption hich describes action.
you generate
randomly?
rundom- to text
<Caption Placeholders
How can we describe motion with texts?
What movements would suitably follow the thematic climax of <Motion, Placcholders?
motion prediction
What might deepen the emotional expreasion
movements could follow resalve susperse <Motion Placcbolders?
Alstioa_Paucchodeer
<Motion, Placcholders?
What new character dynamics could the middle section <Motion
Caccion_PPaccchlee Placcholder> explore?
nn-beween
type drumatic climax
section
Placeholder> might
Motion_PPacchooderr
What potential themes ascent or deseent could he explored in the midkle of <Motion Placeholders?
Script dance that adapts to the tempo shifls <Music Placeholdor>.
musác-to tosii--to-aanc
Create a dance that would visually mimic the lyrical journey in <MMssi_Plucchlldee Placeholder>.
Plachhoee
Compose dance that explores the
charateristics <Music. Placeholders.
Can you design music for this dance <Mecion Placeholders?
dasce-to-music
music
Placeholders.
<Music Plactoller>
Araagg a symphony that captures the shifts <Mation Placeholders.
Please generate
manomnl.
rundom to music
Can you generate a music with dance style?
<Music_ Piaceholder>
Crest a music for any style dance.
Please generate dince based on the caption cCaption Plachhoddrrs
lox--to-aacc
Create dance the text <Catiion_ Piaccholdrr..
Phachokea
a that comesponds
description eCaption Placeholderz.
Describe the dance movements that correspond the given masic <Music Placeholder>.
music-to-lext
Deserite dance actions that match the provided music <Music Placeholdar>.
<Caption_Pluucholle Placeholder>
Detail the dance sleps asocciated with the specified music <Music <Muuss_Pascchlddrr Placcheldar>.
the comparison of text-to-motinn on Motion-X X dataset. Tab. 10 shows the comparison motion-to-tex. on Motion-X dataset. Tah. 11 shows the comparison of music-to-dance on AIST+ and Dance datasets. Tah. 12 shows the comparison of dance-to-musie comparison of motion prediction and motion in- in-hetween on Motion-X dataset. Tab. 15 shows the comparison of music-to-text, text- to-dcee and text-to-music on AIST+ dataset. Tab. and Tab. 17 show comparison of motion-related tasks among different size of TS. As shown in these tables, GP can achieve competitive performance with SOTAs across all evaluated tasks.
AIST+ + and FineDance datasets. Tab. 13 and Tab. 14 shows the
D Qualitative Results and Comparison with SOTA Methods
Fig. presents visualizations for varicty of tasks, including text-to-mation, motion-to-text, motion prediction, motion in-between. music-to-dance long-term dance generation, and music-texi conditioned dance generation. The visualization results show that method can generate realistic results various motion-relevanl tasks. Fig. presents the qualitative results between different methods for text- to-motion and music-to-dance
E Additional Experiments
The performance of text-to-motion based on different Tab. performance of GPT on Motion-X dataset is analyzed across different values of A for the text-to-motin task. The results indicate that as À increases, the model's 's recall precision (Topl, Top2, Top3) initially improves but subsequently declines. The optimal performance is achieved at À = 0.2, evidenced by the highest scores in RPrecision and modality metrics.
17

---

Further increases in À to 0.3 and 0.4 lead to deteriorating performance, particularly in FID and RPrecision, suggesting that excessive À values may result in over-regularization reduced adaptability of the model.
Table 9; Comparison of Text-to-Motion Motion-X dataset [29]. The arrows (1) indicate that higher values better. The arrows (L) indicate that smaller values better. Bold and underline indicate the best and the second best result.
RPrecision
Methods
FID
MM Dist
MModality
Top1
Top2
Top3
Real
0.675
0.821 #50000
0.878=0002
+00000
2.938 20007
2.316
MDM [44)
0.402
4616
0.704
0.161 +00000
-4044
2.234
2.241 -0045
MLD [5]
0.612#4000
0. 0.743#9002
0,808 0,80g=0004
0.122
3.117
2.267
2.210=0005
T2M [5S]
0.647#40008
20005
0.101
3.046
2.270
2.226 -0206
MotionDiffuse [55]
0.659:20000
0. 1.802 t:0004
0.865 760000002
0.005
2.944 25.004
2.220
2,102=0056
Trained single task
2.656
0. 79755
0,843
0.078
2.942 20001
2.13328002
2,046-8082
McGPT (Pre-trained)
0.601 0.601
0.751 #50000
0.803 0.803=0.002
0.092
2.94520000
2.251
2.188=0074
M (Fine-tuned)
0.615
0. 0.75740004
0.8 5
0.093
2.944 20,00
2.253
2.204
MFGPT(Fine-tuncl only T2M)
0.661 20,00
0.804 +00004
45004
0.861 =0003
0.076
2.940 20,000
2.273=0025
2.131 =0,52
Table 10: Comparison of Motion-to- Text Motion X [29].
RPrecision f
Methods
MM Dist
Bleu@
Bleu@4
Rouge f
CIDEr t
BertScore
Top1
Top2
Top3
Real
0.681
0.824
0.881
2.897
TM2T [15]
0.574
0.726
0.806
2.988
30.54
12.13
32.52
20.16
25.37
Trained single task
0.565
0.706
0.767
3.011
31.07
10.14
31.65
22.92
28.19
G (Pre-trained)
0.627
0.773
0.834
2.946
33.31
11.00
34.10
24.12
30,96
M (Fine-tuned)
0.631
0.783
0.845
2.950
34.27
11.50
34.55
42.93
31.49
Table 11: Comparison of Music-to-Dance AIST+ [24] and FineDance [25].
Music-to- Donce on +IST+
Music-to-Dancee FineDance
Methods
FID;
Div. f
BASt
FID,
Div, f
BAST
Real
17.10
8.19
0.2374
9.73
0.2120
FACT [24]
35.35
5.94
0.2209
113.38
3.36
0.1831
Bailando [42]
28.16
7,83
0.2332
82.81
7.74
0.2029
EDGE [48]
42.16
3.96
0.2334
94.34
8.13
0,2116
Lodge [26]
37.09
5.58
0.2423
45.56
6.75
0.2397
Trained single task
75.47
5.57
0.1884
128.37
6.48
0,2036
M GPT (Pro-trained)
27.65
7.52
0.2105
92.35
7.67
0.2134
CCC (Fine-tuned)
24.34
7,50
0.2056
86.47
7.75
0.2158
Table 12: Comparison of Dance-to-Music on AIST++ [24] and FineDance [25].
Dance-to Music on AIST+
Monsecc
Dance-te to Mussici on FineDance
Methods
BCS t
CSD
BHS
HSD
FI
BCS
CSD
BHS 1
HSD 1
F1
Foley [11]
96.4
6.9
41.0
15.0
57.5
CMT [10]
97.1
6.4
46.2
18.6
62.6
D2MGAN [64]
95.6
9.4
88.7
19.0
93.1
CDCD [65]
96.5
9.1
89.3
18.1
92.7
LORIS [52]
98.6
6.1
90.8
13.9
94.5
Trained single task
93.9
9.2
93.6
12.8
92.8
84,84
21.61
51.35
27.13
63.97
MGGP (Pre-trained)
93.4
10.9
93.8
11.5
94.2
83.16
19.95
73.65
23.90
78.12
GGPT (Fine-tuned)
93.6
10.1
94.0
10.6
94.9
84.10
18.36
74.66
23.45
78.23
18

---

Table 13: Comparison of Motion Prediction and Motion In-between on Motion-X [29].
Motion Prediction
Motion In-between
Methods
FID
Diversity
ADE 4
FDE
FID
Diversity t
ADE 4
Ground Truth
0,009
2.316
0.009
2.316
MDM [44]
1.028
1.746
8.057
11,266
0.831
1.768
6.542
Trained single task
0.774
1.778
7.840
9.575
0.692
1.810
6.690
GPT-prrra 0.707
1.874
6.954
8.684
0.604
1.897
5.692
GPT-finneeune 0.682
1.838
6.898
8.091
0.612
1.900
5.649
Table 14: Comparison of MPJPE on Motion Prediction and Motion In in-between on Motion-X X [29].
Methods on Motion-X
Motion Prediction (MPJPE L) Motion In-between (MPJPE,
T2M-GPT [55]
80.2
63.7
MoMask [16]
67.9
55,2
MotionGPT [21]
71.3
59.9
M (Instruction- tunndd)
54.2
51.0
Table 15: Comparison of Music-to-Text (A2T), Text-to-Dance (T2D) and Text-to-Music (T2A) +
Music-to Music-to-Tens Text
Text-to-Dance
Text-to Musie
Methods on AIST+
Bleu@4 f
CIDEr f
R-TOPI 1 FID
BCS 1 BHS
GP (Single task training for A2T)
9.24
24.55
M (Single task training for T2D)
0.541
0.095
MusicLDM [2]
74.5
73.8
Mubert
73.3
73.0
GP (Instruction-uun tuned)
11.95
28.88
0.588
0.077
74.5
74.7
Table 16: Comparison of Text-to-Motion and Motion-to- Text with different size of T5.
Text-to-Motion
Motion-to-Text
Methods Motion-X
LLM
Training time
R-TOP1 1
FID
Div 1
R-TOP3
Bleu4
CIDEr 1
M
T5-small (60M)
days
0.598
0.096
2.202
0.822
10.43
38.22
'
T5-base (220M)
days
0.615
0.093
2.253
0.845
11.50
42.93
GT
T5-Jarge (770M)
days
0.619
0.090
2.256
0,848
11.64
43.05
Table 17: Comparison of Music-to Mesiictt---oeee So-cacce and Dance-to Moaceett-MMsicc with different size of TS.
Music-to-Dance
Dance-to-Muss
Methods AIST+
I.LM
Training time
FID; |
DIV, 1
BAS
BCS t
BHS f
M
T5-small (60M)
days
28.05
5.96
0.2091
89.1
91.2
Gº
ba-e (220M)
days
24.34
7.50
0.2056
93.6
94.0
Ta-large (770M)
days
23.26
7.54
0.2061
93.8
94.1
Table 18: Hyper-parameter analysis of X. Comparison of Text-to-Mooteee on Motion - [29] with different values of A. For this ablation study, M is trained solely on the text-to-motion task to examine the impact of X. This study is conducted during the pre-training stage.
RPrecision
Methods
FID
MM Dist
Diversity
MModality
Topl
Top2
Top3
Real
0.605
0.020
0.821 10.003
0.878
100000 10.000
0.009 10.000
2.938 +0.007
2.316 100011
A=0.0
0.645
0.778 10.000
0.826 400000
0.08
1:00002
2,944
400002
4.004
2.025 1:0.21
A=0.1
0.000
40.002
0.838
+0,002 +0.002
0.078 +0.002
2.944
+00000
2.1282 #:0.022
2.128
2.039 +:0.49
A=0.2
0.606
0.795 0.795-40.001
10.001
0.843 10.001
0.078 00.000
2,942
11.000
2.133 2.133 400012
2.046 #00052
A=0.3
0.629
0.716 t0,002
0.784 t0.00: tt.000
0.095
0.0002
2.941
#000001
2.113 +:0.077
2.036
2.036 +:0004
A=0.4
0.573
0.025
0.793 +0000
0.1022
2.942
2.090
2.030
19

---

Narke-to Text
dodking eegge te Arm log in Creac ouerice
Aasca Sade High
Envondaed wwwdddiiii wibiaaggeaee
A patanaing Seawe Wiss
Maatente-Sattle Tacks
pudticiun
Leg: Term Dass Gueeratee Task
Mutte Test Candibeed Coneration Task
Figure 6: The qualitative results for different motion comprehension and generation tasks.
20

---

"a person appears have pain holding and slouching his loft shoulder."
person walks forward and picks things and puts them down with their hands..
person slowly walking forward and them jogs some before coming to stop.
Real
MDM
MoMask
MPGPT
(a) Qualitutive comparison on text-lo-motion task.
2
(Break style, 5s)
Bailando
MºGPT
(b) Qualitative comparison on music-to-dance task.
Figure 7: Qualitative comparisons for text-to-motion task and music-to-dance task. (a) refers to the qualitative comparison between Real, MDM, MoMask and M3GPT on texi-to-motion task. The red words and boxes highlight the misaligned motions. The results demonstrate that M3GPT shows good text understanding for motion generation. (b) refers to the qualitative comparison between Bailando M3GPT on music-to-dance task. The input is an e-sccond-long piece of music in the Break style.
21

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
A summary of the paper's contributions is provided in the introduction.
Guidelines:
The answer NA means that abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match thcorctical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed the paper.
The authors are encouraged create separate "Limitations" scction their paper.
The paper should point out any strong assumptions and how robust the results these assumptions (e.g.. independence assumptions, noiseless settings, model well rell-specifications asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
violations of
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should he articulated.
The authors should reflect the factors that inffuence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Not Applicable
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
22

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided in appendix or supplemental material.
Theorems and Lemmas the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: See Section 4 and Appendix for implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case of a large language model), releasing of model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. a large language model), then there should either be a way access this model for reproducing the results way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct
dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the of cosed-source models, it may be that access to the model is limited some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open faithfully reproduce the main experimental results, described supplemental material?
the data and code, with sufficient instructions
Answer: [Yes]
Justification: See http://ghhab..ccm/oooninnaaaa//a
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines futtps://nigee cttplllipppsccpppiii///// Trides/CodeSubee ssionPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (e.g.. for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: :/ //nips....///////// cc/public/ Saa.ddes/Coceeuummisssoooooo
for more details.
23

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommeneed, but including URLs to data and code is permitted.
6. Experimental Setting/Detan Details
Question: Does the paper specify all the training and test details (c.g,, data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Section 4 and Appendix for implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [NA]
Justification: Not Applicable.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should given (e-g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error than state that they not verified.
CL, if the hypothesis of Normality of errors
asymmetric distributions, authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). crror bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, docs the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
See Section 4 and Appendix for implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https::/ httpss////uriss..... cc/ wc/ppblic//eeeeeecceeccc Ethi httpa///eeehiiouieeeeeeeeeeeeeee cotuidelineeeeeeeeeoeeeee lnne??
Answer: [Yes]
Justification:
The research conducted in the paper conform, in every respect, with the NeurIPS Code
Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., if there is special considcration duc laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Section 5
Guidelines:
The answer NA means that there is societal impact of the work performed.
the authors answer NA No. they should explain why their work has societal impact why the paper does address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point out. For example, the quality generative models could used to generate deepfakes for disinformation. On other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
legitimate to point out that an improvement in
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that been put in place responsible release data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Not Applicable.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk misuse or dual-use should he released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
25

---

We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: See References.
Guidelines:
The answer means that the paper does not use existing assets.
authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, possible, include URL.
The name of the license (e.g.. g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g.. website), the copyright and terms of service of that source should he provided.
If assets released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Not Applicable.
Guidelines:
The NA means that the paper does release Researchers should communicate the details of the dataset/code/model part of their sub- missions structured templates. This includes details about training. license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Not Applicable.
Guidelines:
answer means that paper does not involve crowdsourcing nor research with human subjects.
Including this information in supplemental material is fine, but main contribution of paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Beard (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants. whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
26

---

Justification:
Not Applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review.
27


Reviews:
[{"review_id": "4K9pTxZcm2", "reviewer": "Reviewer_ZwnC", "summary": "In this paper, the authors introduce $M^3$GPT, a multimodal multitask framework designed for both motion comprehension and generation. Utilizing discrete vector quantization, $M^3$GPT establishes a discrete semantic representation space for various modalities. To avoid information loss during discrete de-tokenization, $M^3GPT$ jointly trains the large language model (LLM) and motion de-tokenizer, thereby optimizing the LLM within both the discrete semantic space and the continuous raw motion space. Furthermore, the authors construct paired text descriptions for music and devise two supplementary tasks music-to-text and text-to-dance, which facilitate the alignment of music and dance within the text embedding space. The efficacy of this approach is demonstrated through experiments across a range of motion-related tasks, and showcases zero-shot generalization capabilities for highly challenging tasks.", "questions": "1. I am curious about the long-duration dance generation mentioned in Page 6, especially if the LLM can maintain coherence and consistency in generating long sequences or this long-duration generation is achieved by the consistency during de-tokenization.\n\n2. In Table 4, the authors evaluate music-to-dance on two datasets AIST++ and FineDance, $M^3$GPT achieves state-of-the-art performance on AIST++, but the performance on FineDance is not that competitive. Could the authors provide more insights into the reasons behind this performance gap between the two datasets?\n\n3. Have authors evaluate different sizes of the $M^3$GPT model? Given the added modalities and tasks, it would be interesting to understand how the model scales with respect to performance and computational requirements.", "limitations": "The authors have discussed limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation behind investigating the integration of text, music, and motion for motion comprehension and generation is convincing.\n2. The proposed joint optimization of the large language model (LLM) and motion de-tokenizer, along with the synergy learning strategy incorporating auxiliary tasks, demonstrates potential in enhancing multimodal motion generation.", "weaknesses": "1. The effectiveness of combining text, music, and motion modalities in the $M^3$GPT framework is not thoroughly evaluated. Although the authors present the $M^3$GPT capabilities in handling zero-shot tasks. However, there is still doubt about how can text-to-motion or music-to-dance tasks benefit from the multimodal framework.\n\n2. Some key experiments are missing.\n\n   a. In Table 3, the authors evaluate proposed $M^3$GPT on text-to-motion dataset Motion-X on 4 tasks. However, MotionGPT[1] can also perform the tasks while the comparison between MotionGPT[1] is missing.\n\n   b. Besides motion prediction lacks comparison with models like T2M-GPT[2], since the auto-regressive models are better at prediction.\n\n   c. Motion inbetweening lacks comparison with the state-of-the-art models like MoMask[3] capable of temporal inpainting which can handle motion inbetweening tasks.\n\n3. The qualitative comparison of generated motions by the proposed $M^3$GPT framework from text or music is absent. The inclusion of detailed qualitative results is crucial for providing a comprehensive understanding of the naturalness and overall realism of the generated gestures.\n\n\nMinor:\n\n1. In Figure 2, there is an error in drawing graphics in the Motion Codebook part.\n\n[1] Jiang, Biao, et al. \"Motiongpt: Human motion as a foreign language.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[2] Zhang, Jianrong, et al. \"Generating human motion from textual descriptions with discrete representations.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.\n\n[3] Guo, Chuan, et al. \"Momask: Generative masked modeling of 3d human motions.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: There is still doubt how can text-to-motion or music-to-dance tasks benefits from multimodal framework.**\n\nWe argue that text-to-motion and music-to-dance tasks benefits from multimodal framework in two main aspects: \n1. **A shared tokenizer for motion and dance data:** The shared tokenizer inherently broadens the scale and diversity of the training data, allowing this tokenizer to learn more generalizable representations for both motion and dance data. \n2. **Constructing two auxiliary tasks, music-to-text (A2T) and text-to-dance (T2D):** Through auxiliary tasks, $M^3$GPT implicitly learns to decompose the complex music-to-dance task into simpler A2T and T2D tasks. Also, with a shared motion tokenizer, text-to-dance and text-to-motion tasks reinforce each other within the same matching space. In this way, $M^3$GPT builds the synergies between  music-to-dance and text-to-motion, facilitating mutual reinforcement. \n   \nAs shown in the table below, the shared tokenizers and auxiliary tasks generally lead to gains in both tasks, validating their effectiveness.\n\n|Methods  | Text-to-Motion on Motion-X (R TOP1$\\uparrow$ \\| FID$\\downarrow$ \\| Div$\\uparrow$) | Music-to-Dance on AIST++ (FID_k$\\downarrow$ \\| Div_k$\\uparrow$ \\| BAS$\\uparrow$)|\n| ---- | ----- | ----- |\n|Single Task | 0.638 \\| 0.095 \\| 2.101 | 80.41 \\| 5.51 \\| 0.1882\n|Single Task with shared tokenizer| 0.656 \\| 0.078 \\| 2.133 | 75.47 \\| 5.57 \\| 0.1884\n|$M^3$GPT (without T2D and A2T)| 0.547 \\| 0.104 \\| 2.099 | 37.14 \\| 7.61 \\| 0.2005\n|$M^3$GPT (Pre-trained) | 0.601 \\| 0.092 \\| 2.251 | 27.65 \\| 7.52 \\| 0.2105\n|$M^3$GPT (Instruction-tuned only single task) | **0.661** \\| **0.076** \\| **2.273** |  **23.01** \\| **7.85** \\| **0.2261**\n\n\n**W2-a: Comparison with MotionGPT on Motion-X is missing.**\n\nThe table below presents the results of MotionGPT on motion-X. Our method outperforms MotionGPT across all 4 tasks on motion-X. \n\n|Methods on Motion-X| Text-to-Motion (R TOP1$\\uparrow$   \\| FID$\\downarrow$ \\| Div$\\uparrow$)| Motion-to-Text (R TOP3$\\uparrow$ \\| Bleu@4$\\uparrow$ \\| CIDEr$\\uparrow$) | Motion Prediction (FID$\\downarrow$ \\|Div$\\uparrow$ \\| MPJPE $\\downarrow$)| Motion In-between (FID \\| Div \\| MPJPE)\n| ---- | ----- | ----- | ----- |----- |\n|MotionGPT| 0.659 \\| 0.078 \\| 2.166 | 0.840 \\| 11.21 \\| 31.36 | 0.701 \\| 1.818 \\| 71.3 | 0.648 \\| 1.875 \\| 59.9\n|$M^3$GPT| **0.661** \\| **0.076** \\| **2.273** | **0.845** \\| **11.50** \\| **42.93** | **0.682** \\| **1.838** \\| **54.2** | **0.612** \\| **1.900** \\| **51.0** \n\n\n**W2-b: Lacking comparison with T2M-GPT on motion prediction task.**\n\nThe table below presents the results of T2M-GPT on motion prediction task. Our method largely outperforms T2M-GPT.\n\n|Methods (Motion Prediction) Motion-X|  FID $\\downarrow$   | Div $\\uparrow$ | MPJPE $\\downarrow$\n| ---- | ----- | ----- | ----- |\n|T2M-GPT| 0.814 | 1.755 | 80.2\n|$M^3$GPT| **0.682** | **1.838** | **54.2**\n\n**W2-c: Lacking comparison with MoMask on motion In-between task.**\n\nThe table below presents the results of MoMask on motion In-between  task. Our method outperforms MoMask in motion In-between task.\n\n|Methods (Motion In-between) Motion-X|  FID $\\downarrow$   | Div $\\uparrow$ | MPJPE $\\downarrow$\n| ---- | ----- | ----- | ----- |\n|MoMask| 0.626 | 1.884 | 55.2\n|$M^3$GPT| **0.612** | **1.900** | **51.0**\n\n**W3: The qualitative comparison of generated motions by the proposed $M^3$GPT framework from text or music is absent.**\n\n**(1) Figure 1 of uploaded PDF file** visualizes generated motions from text. We compare our generations with MDM and MoMask. As seen, our model generates motions of higher quality, with unrealistic motions from MDM and MoMask highlighted in red.\n\n**(2) Figure 2 of uploaded PDF file** visualizes generated dances from music. We compare our generations with Bailando. As seen, the dances generated by our $M^3$GPT are more danceable.\n\n**Q1: Curious about the long-duration dance generation.**\n\nWe maintain coherence and consistency in generating long dance sequences during **LLM prediction phase**. For long-duration dance generation, we sequentially generate 5-second dance segments. Each dance segment is generated using corresponding music and **previously generated dance segments as control conditions**. This strategy ensures that each newly generated dance segment aligns seamlessly with the preceding ones, maintaining overall coherence.\n\n**Q2: Could the authors provide more insights into the reasons behind the performance gap between AIST++ and FineDance datasets?**\n\n The main reason is that **the training and testing settings on FineDance are inconsistent  in our work, whereas they are consistent in other works**. \n\nExisting works typically train and test **separately** on  AIST++ and FineDance—using  5-second clips for AIST++ and 30-second clips for FineDance. In contrast, we develop a **unified multitask model across datasets**. To maintain consistency, we slice  AIST++ and FineDance data into 5-second clips for unified training, aligning with the 5 to 10-second range of motion-X data. During testing, we generate longer 30-second clips on FineDance to compare with existing methods. This inconsistency on FineDance could lead to  performance drop compared to existing models.\n\nTo further validate our method, we finetune $M^3$GPT on FineDance using 30-second clips. As shown in the table below, $M^3$GPT can achieve competitive, even superior performance compared to existing methods, demonstrating its potential on FineDance.  \n\n|Methods (Music-to-Dance) FineDance|  FID$_k$$\\downarrow$   | Div$_k$$\\uparrow$ | BAS$\\uparrow$\n| ---- | ----- | ----- | ----- |\n|EDGE | 94.34 | 8.13 | 0.2116\n|Lodge | 45.56 | 6.75 | **0.2397**\n|$M^3$GPT |86.47 | 7.75 | 0.2158\n|$M^3$GPT(Instruction-tuned with 30-second clips)| **42.66** | **8.24** | 0.2231\n\n\n**Q3: Should evaluate different sizes of $M^3$GPT.**\n\nPlease refer to our **reply to Q1 in Global Response**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uTz8lMV82B", "reviewer": "Reviewer_Wkeh", "summary": "This paper presents an advanced Multimodal, Multitask framework for Motion comprehension and generation. It aims to create a unified representation space for various motion-relevant modalities and model the connections and synergies among different motion tasks. M3GPT consists of multimodal tokenizers and a motion-aware language model, and is trained through a multimodal pre-training + instruction-tuning pipeline. The model is evaluated on multiple motion-relevant tasks and datasets, and shows competitive performance and strong zero-shot generalization capabilities.", "questions": "(1) Did authors try different LLM architecture or different size of T5, what is the conclusion and the training time for T5 base?\n\n(2) Did you evalute two auxiliary tasks music2text or text2dance quantitatively or qualitatively?\n\n(3) For zero-shot tasks, can they combined together for more application? For instance, generating a long dance while we specify text prompt for one segment in the middle of sequence. Also, how about zero-shot text2music?\n\n(4) For tab.4, what is the result of Instruction-tuned only on AIST++ or FineDance of which the setting is similar to the last row in tab.3.\n\n(5) How did you implement motion in-between in pretrained stage use the equation(4)? More specifically, whether your model is conditioned on future target frame, if so, the equation is not precise enough.  \n\n(6) Is there any special token for motion and music in the unified token vocabulary, like start/end of motion/music token? If not, how to ensure the pretrained model generate desired modality? For example, I wanna directly generate dance conditioned on music, but actually the model generate text token.\n\n(7) The statistics of music2text and text2dance paired dataset are missing.\n\n   \nI would like to raise the score if authors can well address my concerns questions.", "limitations": "N.A", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) Unifies multiple motion-related tasks in a single model, achieving bidirectional alignment.\n\n(2) Achieves competitive performance across multiple tasks and datasets and strong zero-shot generalization capabilities.\n\n(3) Explores motion related music2text and text2dance as auxiliary tasks.", "weaknesses": "(1) For supplementary videos, it's hard to judge the quality of music-motion bidirectional generations without combining them in one video; also, long-term dance generation on aist++ dataset is only a little longer than 5s.\n\n(2) The method of Joint optimization of LLM and motion de-tokenizer lacks important details. In line 194-195, I wonder how the gradient from de-tokenizer backprogate to LLM if author use softmax to choose top-1 token.\n\n(3) For motion prediction and In-between tasks, feature-level metrics like FID are not enough, pose-level metrics like MPJPE are needed for comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: For supplementary videos, it's hard to judge the quality of music-motion generations without combining them in one video; Long-term dance generation on aist++ is only a little longer than 5s.**\n\n(1) Due to the rebuttal period restrictions on video uploads, we will present music and dance in one video for clearer evaluation in revised version. \n\n(2) Regarding long-term dance generation, AIST++ comprises musics that are slightly longer than 5s. So the generated dances on AIST++ are relatively short. To showcase our model’s capability for long-term dance generation, we have included a dance generation of over 90s on FineDance in supplementary video.\n\n**W2: How the gradient from de-tokenizer backpropagate to LLM?**\n\nIn implementation, we do not directly backpropagate de-tokenizer gradient to LLM. Instead, with the goal of minimizing L1 loss between the predicted and real motion, we search for the motion's token sequence that could minimize this L1 loss in original motion space. As the motion de-tokenizer continuously optimizes, the target motion's token sequence, which supervises LLM training,  dynamically changes. This dynamic adjustment reduces L1 loss progressively, achieving joint optimization. \n\n**W3: For motion prediction and in-between tasks, pose-level metrics like MPJPE are needed for comparison.**\n\nIn the tables below, we add MPJPE as a metric. Our method still outperforms existing works on MPJPE metric in both tasks.\n\n\n|Methods on Motion-X| Motion Prediction (MPJPE$\\downarrow$)| Motion In-between ( MPJPE) |  \n| ---- | ----- | ----- |\n|T2M-GPT | 80.2| 63.7 | \n|MoMask |67.9 | 55.2|\n|MotionGPT | 71.3| 59.9|\n| M3GPT (Instruction-tuned)| **54.2**|**51.0** |\n\n\n\n**Q1: Did authors try different LLM architecture or different size of T5, what is conclusion and training time for T5?**\n\n Yes, Please refer to our **reply to Q1 in Global Response**.\n\n**Q2:  Did you evalute two auxiliary tasks music2text (A2T) or text2dance (T2D) quantitatively or qualitatively?**\n\n(1) For quantitative evaluation, we report the *Bleu@4* and *CIDEr* for A2T, and *R Top1* and *FID* for T2D. As shown in the tables below, our method consistently outperforms single task training. \n\n(2) For qualitative evaluation, **Figure 4 of uploaded PDF** visualizes the generation of text2dance. Compared to the singe-task model, $M^3$GPT generates more realistic dance aligned with input music.\n\n|Methods (Music-to-Text) AIST++|Bleu@4$\\uparrow$ | CIDEr$\\uparrow$|\n|----|----|----|\n|A2T (Single task)| 9.24 | 24.55 |\n|$M^3$GPT (Instruction-tuned)| **11.95**| **28.88**\n\n|Methods (Text-to-Dance) AIST++|R Top1$\\uparrow$| FID$\\downarrow$|\n|----|----|----|\n|T2D (Single task)| 0.541 | 0.095\n|$M^3$GPT (Instruction-tuned)| **0.588** | **0.077**\n\n**Q3: For zero-shot tasks, can they combined together for more application? For instance, generating a long dance while we specify text prompt for one segment in the middle of sequence. How about zero-shot text2music?**\n\n(1) Our model can perform additional zero-shot tasks. It can generate a long dance while specifying text prompt for one segment, as visualized in **Figure 3 of uploaded PDF**. This application integrates music-to-dance, zero-shot **music+dance-to-dance**,  and zero-shot **music+dance+text-to-dance** tasks. As shown in that Figure 3, the generated dance is coherent and consistently aligning the input text prompt.\n\n(2) Our model can perform zero-shot text-to-music. The table below shows a quantitative evaluation on text-to-music task. Even without training on text-to-music task, $M^3$GPT can obtain comparable even superior performance over existing methods.\n\n|Methods (Text-to-Music) AIST++|BCS$\\uparrow$|BHS$\\uparrow$|\n|----|----|----| \n|MusicLDM| **74.5** | 73.8\n|Mubert|73.3 |73.0\n|$M^3$GPT (Instruction-tuned)| **74.5** | **74.7**\n\n**Q4: For tab.4, what is the result of Instruction-tuned only on AIST++ or FineDance?**\n\n Please refer to our **reply to Q3 in Global Response**.\n\n\n**Q5:  How did you implement motion in-between in pretrained stage use the equation(4)?**\n\n Motion in-between can be implemented using Equation 4. Formally, given a motion sequence $\\boldsymbol{m} = \\left(m_1, \\dots, m_N\\right)$, motion in-between task uses the first $n_1$  and last $n_2$ frames, $\\boldsymbol{m_1}=\\left(m_1, \\dots, m_{n1}, m_{N-n2+1}, \\dots, m_{N}\\right)$, to predict intermediate frames $\\boldsymbol{m_2}=\\left(m_{n1+1}, \\dots, m_{N-n2}\\right)$. So the future frames in $\\boldsymbol{m_1}$ are only used as **input**, and the **target** frames are the middle frames in $\\boldsymbol{m_2}$.\n\nIn  implementation, $\\boldsymbol{m_1}$ is fed into the motion tokenizer to produce a token sequence, serving as **source input** of LLM ($\\boldsymbol{q_s}$). Also, $\\boldsymbol{m_2}$ is fed into the motion tokenizer to produce a token sequence, serving as **target output** of LLM ($\\boldsymbol{q_t}$). Thus motion in-between can be formulated as $p_{\\theta}\\left(\\boldsymbol{q}_t^i |\\boldsymbol{q}_t^{<i}, \\boldsymbol{q}_s \\right)$ (Equation 4).\n\n**Q6: Is there any special token for motion and music in the unified token vocabulary, like start/end of motion/music token? If not, how to ensure the pretrained model generate desired modality?**\n\n Yes. There are special tokens for motion and music, such as *<start_of_motion>* and *<start_of_music>*. These special start and end tokens control the beginning and end of the model's decoding process. \n\nAdditionally, we use **task-specific instruction** to control model to generate desired modality. For example, the instruction *Generate a motion for the caption* is used for text-to-motion task; Instruction *Generate a music based on the dance* is used for dance-to-music task.\n\n\n**Q7: The statistics of music2text (A2T) and text2dance (T2D) paired dataset are missing.**\n\nPlease refer to our **reply to Q4 in Global Response**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O73pU94D8a", "reviewer": "Reviewer_KMVC", "summary": "The paper presents M3GPT, which creates a unified representation space for different motion-relevant modalities, including text, music, and motion/dance. The framework employs discrete vector quantization for these modalities, enabling integration into a large language model (LLM) with a single vocabulary. The model operates directly in the raw motion space to avoid information loss associated with discrete tokenizers. Extensive experiments highlight M3GPT’s superior performance across various motion tasks and its zero-shot generalization capabilities.", "questions": "- Could you provide more details on the training process, including the computational resources required and the time taken to train M3GPT?\n\n- Have you conducted any ablation studies to understand the contribution of each component of M3GPT to the overall performance? If so, could you include those results?\n\n- Can you provide more examples or case studies where M3GPT has been applied to real-world scenarios? What challenges were encountered?", "limitations": "Limitations have been discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a novel approach to integrate multiple modalities (text, music, motion) within a single framework, addressing a significant gap in existing research. The unified representation space and the strategy of using text as a bridge to connect different modalities are theoretically sound and contribute to the field of multimodal learning.\n\n- M3GPT handles six core tasks related to motion comprehension and generation, demonstrating versatility and robustness. The experiments show that M3GPT achieves competitive performance across multiple tasks.\n\n- The model's ability to perform well in zero-shot scenarios, such as long-term dance generation and music-text conditioned dance generation, is particularly impressive and valuable for practical applications.", "weaknesses": "- The proposed framework is complex and may require significant computational resources to implement and extend.\n\n- While the experimental results are promising, the paper could benefit from a more detailed discussion of the evaluation metrics and benchmarks used to assess performance. A more extensive comparison with a broader range of baselines could strengthen the argument for its superiority. Also, the paper would benefit from a more in-depth discussion on the potential real-world applications and limitations of M3GPT.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: The proposed framework is complex and may require significant computational resources to implement.**\n\nTo ensure reproducibility, we will open-source the training and inference code of proposed $M^3$GPT framework. \n\nFor computational resources, please  refer to **reply to Q2 in Global Response**.\n\n\n**W2-1: The paper could benefit from a more detailed discussion of evaluation metrics and benchmarks.**\n\nThanks for your suggestions. We will add these discussion in final version.\n\n**(1) Evaluation Metrics.** The table below shows the evaluation metrics for each task. \n\n|Task| Text-to-Motion (T2M) |Motion-to-Text (M2T)| Motion Prediction/In-Between| Music-to-Dance (A2D) | Dance-to-Music (D2A)| \n| ----| ----- | ----- | ----- |----- |----|\n|Metrics| FID, R-Precision, Div| R-Precision, Bleu, CIDEr, Rouge| FID, Div, MPJPE| FID$_k$, Div$_k$, BAS|  BCS, BHS|\n\n- FID: Measures distance between the distributions of generated and real motion.\n- R-Precision: Measures retrieval accuracy of generated motion/text relative to input, reflecting semantic similarity.\n- Div: Measures diversity of generated motions. \n- Bleu, Rouge, CIDEr: Measures accuracy, matching degree and semantic consistency between generated and real text.\n- FID$_k$, Div$_k$: FID and Div calculated using kinetic features.\n- MPJPE: Measures the mean per joint position error between generated and real motion.\n- BAS: Computes beat alignment degree between generated dance and input music. \n- BCS/BHS: Measures convergence/alignment between generated and real music.\n\n**(2) More Benchmarks.** The tables below adds two benchmarks. As shown in below table, and Table3/4 of main paper, we observe that: \n- Text-to-Motion: $M^3$GPT achieves comparable FID and best R-Top1 and Div score, showing high-quality motion generation with strong semantic alignment and diversity.\n- Motion-to-Text: $M^3$GPT achieves  best R-precision and CIDEr score, indicating fluent and semantically consistent motion descriptions generation.\n- Motion Prediction and Inbetween: $M^3$GPT achieves  best FID, Div and MPJPE score, reflecting superior motion prediction quality, diversity and precision.\n- Music-to-Dance (Table 4):  $M^3$GPT achieves comparable Div/BAS and best FID$_k$ score, indicating high-quality dance generation with good beat.\n- Dance-to-Music (Table 4), $M^3$GPT achieves  comparable BCS and best BHS score, indicating improved semantic alignment of music with input dance.   \n\n\n|Methods on Motion-X| T2M (R TOP1$\\uparrow$   \\| FID$\\downarrow$ \\| Div$\\uparrow$) | M2T (R TOP3$\\uparrow$ \\| Bleu@4$\\uparrow$ \\|CIDEr$\\uparrow$) | Motion Prediction (FID$\\downarrow$ \\| Div$\\uparrow$\\| MPJPE$\\downarrow$)| Motion In-between (FID \\| Div \\| MPJPE)\n| ---- | ----- | ----- | ----- |----- |\n|MotionLCM| 0.658 \\| 0.078 \\| 2.206| -| -| -|\n|MotionGPT| 0.659 \\| 0.078 \\| 2.166 | 0.840 \\| 11.21 \\| 31.36 | 0.701 \\| 1.818 \\| 71.3 | 0.648 \\| 1.875 \\| 59.9\n|$M^3$GPT| 0.661 \\| 0.076 \\| 2.273 | 0.845 \\| 11.50 \\| 42.93 |0.682 \\| 1.838 \\| 54.2 | 0.612 \\| 1.900 \\| 51.0 \n\n\n\n**W2-2: The paper could benefit from a more in-depth discussion on potential real-world applications and limitations of $M^3$GPT.**\n\n**(1) Potential applications:** $M^3$GPT has may real-world applications, such as **animation creation, visual digital human control, music-based  dance choreography creation**. Taking animation creation as a example, traditional manual methods adjusts   character joints frame-by-frame, which is extremely time-consuming and labor-intensive. With $M^3$GPT, one can simply provide a textural description of desired motion, and $M^3$GPT will automatically generate the corresponding animation, largely reducing labor and time cost. \n\n**(2) Limitations:** $M^3$GPT focus on modeling human body movements, but currently lacks the capability to handle hands. As a result, $M^3$GPT cannot address speech-driven motion generation, which involves gestural rather than body movement. \n\n**Q1: Could you provide more details on the training process, including the computational resources required and the time taken to train $M^3$GPT?**\n\nYes, please refer to our **reply to Q2 in Global Response**.\n\n**Q2: Have you conducted any ablation studies to understand the contribution of each component of $M^3$GPT to the overall performance**\n\nYes, in Table 2 of main paper, we have conducted ablation studies to show the effectiveness of each component of $M^3$GPT. Specifically, $M^3$GPT includes 3 major components designed to improve performance: **(1)** Introduction of two auxiliary tasks, music-to-dance  (A2D) and text-to-music (T2D). **(2)** Joint optimization of LLM and motion de-tokenizer. **(3)** Instruction-tuning. As shown in the table below, each component contributes to the overall performance of $M^3$GPT, specifically for the complex music-to-dance task.\n\n\n|Methods|  T2M on Motion-X (R TOP1   \\| FID    \\| Div) | A2D on AIST++ (FID_k \\| Div_k   \\| BAS)\n| ---- | ----- | ----- |\n|$M^3$GPT (Pretrained without A2T and T2D)| 0.547 \\| 0.104 \\| 2.099 | 37.14 \\| 7.61 \\| 0.200\n|$M^3$GPT (Pretrained without joint optimization)| 0.598 \\| 0.089 \\| 2.218 | 32.71 \\| 7.43 \\| 0.209\n|$M^3$GPT (Pretrained)| 0.601 \\| 0.092 \\| 2.251 | 27.65 \\| 7.52 \\| 0.210 | \n|$M^3$GPT (Instruction-tuned)| 0.615 \\| 0.093 \\| 2.253 | 24.34 \\| 7.50 \\| 0.205\n|$M^3$GPT (Instruction-tuned only single task )| **0.661** \\| **0.076** \\| **2.273** | **23.01** \\| **7.85** \\| **0.226**\n\n**Q3: Can you provide more examples or case studies where $M^3$ has been applied to real-world scenarios? What challenges were encountered?**\n\nWe have applied $M^3$GPT to several real-world scenarios, including animation creation, dance choreography creation, music creation from dance. \n\nThe main challenge is generating long sequence (longer than 5 minutes), where the quality of  generated content degrades over time. We leave the generation of high-quality  long-sequence (longer than 5 minutes) as a future research  direction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LR5ePxKzJm", "reviewer": "Reviewer_2Gpw", "summary": "This paper introduces M3GPT, an advanced Multimodal, Multitask framework for motion comprehension and generation. M3GPT operates based on three fundamental principles. Firstly, it aims to create a unified representation space for various motion-relevant modalities. We employ discrete vector quantization for multimodal control and generation signals, such as text, music, and motion/dance, enabling seamless integration into a large language model (LLM) with a single vocabulary. Secondly, it involves modeling directly in the raw motion space. This strategy circumvents the information loss associated with discrete tokenizers, resulting in more detailed and comprehensive model generation. Thirdly, M3GPT learns to model the connections and synergies among various motion-relevant tasks. Text, being the most familiar and well-understood modality for LLMs, is utilized as a bridge to establish connections between different motion tasks, facilitating mutual reinforcement.", "questions": "1.\tIs it just a matter of migrating multimodal model to the motion domain? \n2.\tWhat new issues have been encountered in the process？\n3.\tWhat is your innovation in solving these problems?", "limitations": "The author has addressed the limitations. This paper focuses on modeling human body movements, excluding hands and faces details.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.     M3GPT is equipped with multimodal tokenizers capable of compressing raw multimodal data, including motion, music, and text, into a sequence of discrete semantic tokens.\n2.\tThe paper jointly trains LLM and motion de-tokenizer, optimizing LLM in both discrete semantic space and raw continuous motion space.\n3.\tThis Paper is well writing and the video results in Supp. are well presented.", "weaknesses": "1.\tIn Table 3, some comparation results are missed. For text to motion generation task, you could compare your method with 1. MoMask: Generative Masked Modeling of 3D Human Motions; 2. Plan, posture, and go: Towards Open-World Text-to-Motion Generation; and 3. Real-time Controllable Motion Generation via Latent Consistency Model. If it’s not suitable for the comparations, you could explain the reason.\n2.\tThe parameterizations (dimensions) of the variable in Section 3 are all missing.\n3.\tIn line 167, T5 is utilized as the language model backbone. While most of the previous model utilize CLIP or BERT, you should include ablation study about the text encoder.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: In Table3, some comparation results are missed for text-to-motion task.**\n\nThanks for your suggestions. We will add the related works mentioned, MoMask [1], PRO-Motion [2] and MotionLCM [3], in revised version. \n\n(1) To ensure fair comparison, we reproduce **MoMask**  and **MotionLCM** on Motion-X dataset  using their official code. As shown in the table below, our method achieves superior performance than MotionLCM, and comparable performance with MoMask.  \n\n(2) Since **PRO-Motion** is not open-sourced, we were unable to reproduce it on Motion-X dataset, so its result is not included in Table3.\n\n|Methods (Text-to-Motion) Motion-X |  R TOP1$\\uparrow$ | FID$\\downarrow$ |Div$\\uparrow$ |\n| ---- | ----- |----|----|\nMoMask | **0.668** | **0.074**| 2.241 | \nMotionLCM | 0.658 | 0.078 | 2.206 |\n$M^3$GPT (Instruction-tuned) | 0.615 | 0.093 | 2.253 |\n$M^3$GPT (Instruction-tuned only T2M) | 0.661 | 0.076 | **2.273**|\n\n**W2: The parameterizations (dimensions) of the variable in Section 3 are all missing.**\n\nThanks for your suggestions.  We will add the dimensions of variable in section 3 as follows  in revised version. \n\n- $N_m$ (line 128), the size of motion codebook $\\mathcal{B}_m$, is set to 512.\n- $d_m$ (line 131), the dimension of raw motion features, is 263.\n- $d$ (line 133), the dimension of embeddings in motion codebook, is set to 512.\n- $d_a$ (line 149), the dimension of raw music, is 128.\n- $T_m$ (line 131) and $T_a$ (line 149) denotes the sequence length of motion and music, respectively. So their values depend on the input sequence.\n- $L_m$ (line 133), the sequence length of diverse motion tokens, is $T_m/4$. Here, 4 is the downsampling rate of motion tokenizer. \n- $L_a$ (line 150), the sequence length of diverse music tokens, is $T_a/128$. Here, 128 is the downsampling rate of music tokenizer.\n\n**W3: T5 is utilized as the language model backbone. While most previous model utilize CLIP or BERT, you should include ablation study about the text encoder.**\n\nFollowing your suggestions, We add ablation study on CLIP and BERT as text encoder. Specifically, we use T5, CLIP and BERT as text encoders respectively, each combined with T5 decoder. We then conducted experiments on the text-to-motion task using these configurations. As shown in below table, T5 encoder can achieve the best performance, validating its superiority over CLIP and BERT in our framework.\n\nMethods (Text-to-Motion) Motion-X  |R TOP1$\\uparrow$   |FID$\\downarrow$ |Div$\\uparrow$|\n| ---- | ---- | ---- | ---- |\nCLIP | 0.641 | 0.088 | 2.110\nBERT | 0.652 | 0.083 | 2.124\nT5   | **0.656** | **0.078** | **2.133**\n\n**Q1: Is it just a matter of migrating multimodal model to the motion domain?**\n\n $M^3$GPT is not simply migrating multimodal model to motion domain. Unlike most multimodal models such as BLIP2, LLaVA, Ferret, which primarily focus on understanding tasks, $M^3$GPT takes a step further by incorporating multimodal tokenizers and de-tokenizers, achieving both motion **comprehension and generation** across **text, motion/dance and music** modalities. Additionally, $M^3$GPT **jointly optimizes LLM and motion de-tokenizer**, further enhancing its capabilities in motion generation.\n\n**Q2: What new issues have been encountered in the process?** \n\nWe have encountered two new issues:\n1. Within the multi-task framework, music-to-dance task poses greater challenges than text-to-motion. This  difficulty arises because both input and output modalities of music-to-dance are unfamiliar to LLMs, and there is limited training data available for this task. Therefore, **effectively leveraging text-to-motion task to enhance more complex music-to-dance task is a critical issue that needs to be addressed.**\n2. The multi-stage training leads to error accumulation in motion generation, due to the combined effects of LLM's prediction and de-tokenizer. Therefore, **mitigating  error accumulation from multi-stage training is another critical issue that needs to be addressed.**\n\n**Q3: What is your innovation in solving these problems?**\n \n1. To solving the first issue outlined in Q2, we design two strategies.  (1) a **shared tokenizer** for motion and dance data, and (2) two **auxiliary tasks: music-to text and text-to-dance**. Through these auxiliary tasks, $M^3$GPT implicitly learns to decompose complex music-to-dance into simpler music-to-text and text-to-dance tasks. Also, the shared tokenizer ensures that text-to-dance and text-to-motion tasks occupy the same matching space, enabling mutual reinforcement. In this way, $M^3$GPT builds the connection between text-to-motion and music-to-dance, leveraging text-to-motion to enhance  its music-to-dance capabilities. \n2. To solving the second issue outlined in Q2, we **jointly optimize LLM and motion de-tokenizer** during training. This strategy can help mitigate error accumulation caused by multi-stage training, thereby enhancing the model's motion generation capabilities.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## **Global Response**
We sincerely thank all reviewers and ACs for reviewing our work. Some common questions are answered.

### **Q1: The evaluations for different size of T5 (Reviewers #ZwnC, #Wkeh, #KMVC)**

We conduct experiments on different sizes of T5: T5-small (60 M), T5-base (220 M), T5-large (770 M). As shown in the tables below, as the model size increasing, the performance on each task improves, but the training time  also increases. Considering the trade-off between performance and computational cost, we use T5-Base model in $M^3$GPT.

|Methods  on Motion-X | LLM  | Training time |Text-to-Motion (R TOP1$\uparrow$ \| FID$\downarrow$ \| Div$\uparrow$)| Motion-to-Text ( R TOP3$\uparrow$ \| Bleu@4$\uparrow$ \| CIDEr$\uparrow$) 
| ---- | ---- | ----| ----| ----| 
|  $M^3$GPT | T5-small (60M) | 5 days |0.598 \| 0.096 \| 2.202 |0.822 \| 10.43 \| 38.22
|  $M^3$GPT | T5-base (220 M) | 7 days| 0.615 \| 0.093 \| 2.253 | 0.845 \| 11.50 \| 42.93
|  $M^3$GPT | T5-large (770 M)| 8 days| **0.619** \| **0.090** \| **2.256** | **0.848** \| **11.64** \| **43.05**


|Methods on AIST++| LLM | Training time| Music-to-Dance (FID$_k$$\downarrow$   \| Div$_k$$\uparrow$ \| BAS$\uparrow$) | Dance-to-Music (BCS$\uparrow$  \| BHS$\uparrow$) |
| ---- | ----- | ----- | ----- |----|
| $M^3$GPT | T5-small (60M) | 5 days |28.05 \| 5.96 \| 0.2091 |89.1 \| 91.2
|  $M^3$GPT  | T5-base (220M)| 7 days| 23.34 \| 7.50 \| 0.2056 | 93.6 \| 94.0
| $M^3$GPT  | T5-large (770M)| 8 days| **23.26** \| **7.54** \| **0.2061** | **93.8** \| **94.1**

### **Q2: More details for training process and computational resources. (Reviewer #KMVC)**

Our training process is divided into three stages: Multimodal Tokenizers Training (Stage1); Modality-Alignment Pre-training (Stage2); Instruction Fine-tuning (Stage3). All experiments are conducted on a machine with 8 A40 (48G) GPUs. Detailed implementation can be found in Lines 276-285 of main paper. Specifically:

- Stage1: We combine Motion-X (64867 motion samples), AIST++ (952 dance samples) and FineDance  (177 dance samples) to train a motion tokenizer. This stage involves 50K training steps, with a batch size of 1000 and a learning rate of 1e-4. **Stage1 takes ~2.5 days to complete.**
- Stage2: We train on 6 main tasks and 2 auxiliary tasks. The detailed statics of training data is shown in the table below.  This stage involves 1000K training steps, with a batch size of 56 and a learning rate of 2e-4. **Stage2 takes ~5.5 days to complete.**
- Stage3: We use 200 instruction templates to construct instruction samples for finetuning the model. This stage involves 200K training steps, with a batch size of 48 and a learning rate of 1e-4. **Stage3 takes ~1.5 days to complete.**

|Tasks| T2M | M2T | Motion Prediction/In-between| A2D | D2A | A2T | T2D|
| ---- | ---- | ----| ----| ----| ----|----|----|
|Training dataset| Motion-X | Motion-X| Motion-X/AIST++/FineDance|AIST++/FineDance | AIST++/FineDance|AIST++/FineDance | AIST++/FineDance|
|Training samples number | 64867 | 64867 | 64867/952/177 |952/177 | 952/177|952/177|952/177|


### **Q3: The results of instruction-tuned only on AIST++ or FineDance (Reviewer #Wkeh)**

 As shown in the tables below, we add the results of $M^3$GPT instruction-tuned only on AIST++ and FineDance. As shown, instruction-tuned on a single task further improves performance. We will add these results in final version.

|Methods on AIST++| Music-to-Dance (FID$_k$$\downarrow$   \|Div$_k$$\uparrow$ \| BAS$\uparrow$)| Dance-to-Music (BCS$\uparrow$ \| BHS$\uparrow$) |  
| ---- | ----- | ----- |
$M^3$GPT (Pre-trained) | 27.65 \| 7.52 \| 0.2105 | 93.4 \| 93.8 
$M^3$GPT (Instruction-tuned)| 24.34 \| 7.50 \| 0.2056 |  93.6 \| **94.0**|
$M^3$GPT (Instruction-tuned for single task) | **23.01** \| **7.85** \| **0.2261**| **94.3** \| **94.0**

|Methods on FineDance| Music-to-Dance (FID$_k$$\downarrow$   \| Div$_k$$\uparrow$ \| BAS$\uparrow$)| Dance-to-Music (BCS$\uparrow$ \| BHS$\uparrow$) |  
| ---- | ----- | ----- |
$M^3$GPT (Pre-trained) | 92.35 \| 7.67 \| 0.2134 | 83.16 \| 73.65| 
$M^3$GPT (Instruction-tuned)| 86.47 \| 7.75 \| 0.2158 | 84.10 \| 74.66|
$M^3$GPT (Instruction-tuned only A2D) | 65.27 \| 7.83 \| 0.2195 | 84.79 \| 75.20 |
$M^3$GPT (Instruction-tuned only A2D with 30-second clips) | **42.66** \| **8.24** \| **0.2231** | **86.72** \| **79.64**

### **Q4: The statistics of music2text (A2T) and text2dance (T2D) paired dataset (Reviewer #Wkeh)**

A2T and T2D datasets are constructed from existing dance dataset (AIST++ and FineDance). Given a music-dance pair, we construct paired textural descriptions for the music/dance data.  Specifically, we use the style annotations of music to create paired texts, such as *a person is dancing Jazz* for *Jazz* style. The table below shows the statistics of A2T and T2D datasets. We will added this statistics in final version.

| Task |  Training sample number (AIST++, FineDance) | Testing sample number | Average sequence duration 
|----|----|----|----|
| A2T| 1129 (952, 177) | 60 (40, 20) | (13.31s,  135.8s)  |
| T2D| 1129 (952, 177) | 60 (40, 20) | (13.31s,  135.8s) |

Meta-review: This paper introduces $M^3$GPT, a multimodal, multitask framework for motion comprehension and generation, integrating text, music, and motion into a unified representation. The model demonstrates strong performance across tasks and impressive zero-shot generalization capabilities.

Reviewers praised the model’s novelty and its ability to handle multiple input modalities seamlessly within a large language model. However, concerns were raised about the complexity and whether the model’s benefits would generalize to broader tasks. Some reviewers also noted a lack of comparisons with other state-of-the-art frameworks.
The rebuttal provided some clarifications and comparisons, though not all concerns were fully addressed. Despite this, the strong results and novel approach make it a valuable contribution.

Overall, the AC leans towards acceptance, given the paper’s solid contributions to multimodal motion comprehension and generation, while recommending that the authors address the concerns about generalization and further comparisons in future revisions.
Decision: Accept (poster)