MotionGPT3: Human Motion as a Second Modality
Authors: Bingfan Zhu
Biao Jiang
Sunyi Wang
SHIXIANG TANG
Tao Chen
Linjie Luo
Youyi Zheng
Xin Chen
Venue: ICLR 2026 Poster (2026)
Keywords: 3d motion
text-driven motion generation
text-to-motion
human motion synthesis
motion caption
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose MotioinGPT3, a bimodal motion-language framework designed to address the challenges of unified motion understanding and generation.

Abstract: With the rapid progress of large language models (LLMs), multimodal frameworks that unify understanding and generation have become promising, yet they face increasing complexity as the number of modalities and tasks grows. We observe that motion quantization introduces approximation errors that cap motion quality, and that unifying discrete text and continuous motion within a single-stream backbone amplifies cross-modal interference. Motivated by recent multi-branch Transformer designs that separate signals from different modalities, we propose MotionGPT3, a bimodal motion–language model for both understanding and generation. MotionGPT3 encodes raw motion into a continuous latent space using a variational autoencoder (VAE), thereby avoiding quantization-induced artifacts, while leveraging the semantic prior of pretrained language models. A dual-stream Transformer with shared attention preserves modality-specific routes while enabling controlled, bidirectional information flow, which reduces interference, stabilizing optimization, and empirically accelerates convergence without degrading fidelity. For multimodal joint training, a generate-then-align three-stage schedule further improves stability and limits cross-task interference. Experiments show that MotionGPT3  achieves 2× faster convergence in training loss and up to 4× faster convergence in validation, while maintaining state-of-the-art performance on standard motion understanding and motion generation benchmarks.

Published as a conference paper at ICLR 2026
MOTIONGPT3: HUMAN MOTION AS A SECOND MODALITY
Bingfan Zhu'
Biao Jiang"
Sunyi Wang'
Shixiang Tang'
Tao Chen?
Linjie Luo" Youyi Zheng" f
Xin Chen
'Zhejiang University Fudan University
ByttDancce
*The Chinese University of HongKong
https://gite www. cww.://paalMoiioottaaaooooooooot
ABSTRACT
With the rapid progress of large language models (LLLMs), multimodal frameworks that unify understanding and generation have become promising, yet they face increasing complexity as the number of modalities and tasks grows. We observe that motion quantization introduces approximation errors that cap motion quality, while unifying discrete text and continuous motion within single-stream backbone amplifies cross-man modal interference. Motivated by recent multi-branch designs that separate signals from different modalities, propose MotionGPT3, bimodal mo- tion-language model for both understanding and generation. MotionGPT3 encodes raw motion into a continuous latent space, thereby avoiding countization-induce artifacts, while leveraging the semantic prior of pretrained language models. A dual-stream Transformer with shared attention preserves modality-speciffe routes while enabling controlled, bidirectional information flow, which reduces interfer- ence, stabilizing optimization, and empirically accelerates convergence without degrading fidelity. For multimodal joint training, generate-then-aligh three-stage schedule further improves stability and limits cross-task interference. Experiments show that MotionGPT3 achieves 2x faster convergence in training loss and up 4x faster convergence in validation, while maintaining state-of-thee performance standard motion understanding and motion generation benchmarks.
INTRODUCTION
Multimodal large language models (MLLMs) have recently achieved rapid progress understand-
ing and generation across text, images TTam 2024 Wu et al. 2024a] Zhou et al.] (024), au-
dio (Agostinelli et al. 2023 Copet et al. 2023 Liu et al. 2024). and video Kondratyuk et al. 2023 Zhang et al.] 2023a 2024d). Built on the strong semantic priors and in-context learning capabilities of pretrained LLMs, these models capture long-range dependencies and compositional structure, enabling few-shot transfer and controllable across modalities (Alayrac et al. 2022) Chowdhery et al.)
2023; Dong et al.] 20233 Li et al., 2023 Touvron et al.) 2023). Toward Unified Motion-Language Modeling. While most prior work has focused mainly text-driven motion synthesis (Shafir et al. 2023 Tevet et al. 2022a b Xin et al. 2023 Zhang et al., 2024a), unified motion-language models for both understanding and generation remain comparatively underexplored. Pursuing both tasks in a single model demands representations and training strategies that respect the distinctive statistics of human motion without sacrificing the reasoning benefits of language models.
Tokenizing motion into fixed codebook, typically via VQ-based models, facilitates integration with Transformer- based LLMs (Guo al., [202cc] Zhang et al.) 2023b 2024c), however inevitably intro- duces quantization error, attenuating high-frequency components and degrading semantic-physiin
consistency. More importantly, treating motion as "language" (Jiang et al., 2023; Wang et al.) 2023
Siyao al. 2022) overlooks the gap between symbolic sequences and continuous trajectories (Wang ct al. 2025b). Consequently, cross-modal alignment often remains at a symbolic level and struggles to capture the ane-grained kinematics demanded by nuanced linguistic semantics. In practice, limited codebook capacity and training coverage further constrain realism and controllability.
Recent MLLMs tend to process multiple modalities within single backbone and attach modality- specific heads and supervision (Park et al. [2025] Team 2024 Zhang et al.) 2024b Zhou et al.) 2024). However, jointly optimizing multimodal objectives induces gradient interference and loss-scale

---

Published as a conference paper at ICLR 2026
mismatch, which increases hyperparameter sensitivity, destabilizes training, and can erode language competence (Driess et al.] 2023; Kendall et al. 2018 Tsimpoukelli et al. (021). Moreover, forcing distinct modalities into shared space crodes modality-specife information and inductive biases, causing negative transfer. For robust, controllable motion-language modeling, a method is needed that (i) adopts representations respect the continuous nature of human movement and (ii) explicitly balances multimodal, multi-objective training. Addressing these representational and optimization bottlenecks is, therefore, key advancing unified motion understanding and generation.
Continuous Motion Latent Space Our approach first replace the motion tokens with continuous, low-dimensional latent representation learned by pretrained motion VAE (Xin et al.) 2023). By 'continuous' we mean: (i) the latents are real-valued vectors rather than discrete code indices, and (ii) the VAE induces a smooth latent manifold which nearby points correspond to gradually varying motions. Compared with VQ-based tokenization, this latent space is perceptually aligned with the original trajectories yet computationally compact, avoiding quantization artifacts and preserves high-frequency micro-dynamics for efficient and stable motion synthesis.
Diffusion Bridge within the LLM Framework Autoregressive generation and cross-entropy- supervision in LLMs presumes díscrete token targets and is therefore ill-suited to continuous motion latents. Conditioned on the LLM's hidden states, we further attach a lightweight diffusion head that perform denoising directly in the motion latent space to predict motion VAE latents, which the motion decoder then converts into motion sequences. Operating in a low-dimensional latent domain with a relatively small expert, this diffusion scheme bridges the gap between LLM hidden states and motion latents while only brings little overhead in both training and inference.
Bimodal Architecture Following Mixture-of-Transformee Transformers (MoT) (Liang al. 2024). we treat human motion as second modality and introduce a motion branch symmetric to the language backbone. The two independent branches interact via shared attention layers, yet retain modality- specific embeddings and allow each module to be guided by its own objective. This bimodal design mitigates interference between modalities and preserves each modality's structure, thereby enabling high-quality motion understanding and generation within unified framework.
Three-Stage Training To effectively model motion branch under guidance of a pre-trained language model, we design a three-stage training scheme. First, we perform Uni-task Pretraining, with the text branch frozen, the motion branch is pre-trained on text-to-motion generation. Next, Cross-Modal Alignment, motion-to-text and motion prediction objectives are introduced to align two branches. Finally, all parameters are optimized in Joint Tune-Tunin.
We summarize contributions follows:
Latent diffusion for motion. Unlike quantization- based pipelines (Zhang et al., 2023b) 2024c), we integrate latent diffusion (Rombach et al.) [022] Xin et al.) (223) into autore- gressive backbone via diffusion head, bridging the continuous motion with the next-token prediction framework for higher-fidelity and diverse synthesis.
Architecture and training. We propose a bimodal motion- lotion-languuae framework with per- modal branches communicating through shared attention, reducing interference while pre- serving modality-specine -specifcc intelligence. A three-stage generate-then-align scheme further stabilizes joint training and curbs negative transfer.
Results and efficiency. Under comparable settings, MotionGPT3s achieves state-of-the-art performance on both text-to-motion and motion-to-text tasks, within unified framework. while reducing training time by approximately 2-3x.
2 RELATED WORK
Human Motion Modeling Early approaches leverage strong text encoders (Li et al. 2022 Radford
et al., |021 Raffel et al. 2020 Sanh et al.
2019) to develop motion- understanding/retries
via shared embeddings Guo et al.
2022c
Tevet et al. 2022a; Yin et al. 2024) or contrastive learning
(Chen et al. 2024 Petrovich et al.
2023
Recent methods (Athanasiou et al. 2024| Cohan et al.)
2024| Shafir et al.] 2023) Tevet et al. 2022b] et al. 2023 Zhang et al. [222c]] 2024b) advance text-
to-motion generation with diffusion backbones (Ho et al. 2020 Song al. [2020), either operating
directly on raw motion sequence or reconstructing a VAE latent. Working in a compressed latent
space, LDMs (Rombach al. 2022) keeps training computationally cheaper and inference faster
2

---

Published as a conference paper at ICLR 2026
lipe onbvoling
Mexion
Motion Decoder
M
Atiention
cendiiss quarz
E
Diffusion head 3f'
Motion Branch
D
E
ResBlock
FinalLayer
Matios
Tokenizer
T
LM Elead
Text Branch
Livrar
Latest
Output Tokesa
AdaLN
Motion Representation
Bimodal Motion-language Model
Motion Diffusion Autoregressive Backbone
Figure 1: MotionGPT3 introduces hybrid motion-language model that takes motion as second modality and processes the data through a new branch, with cross-modal attention mechanism to communicate with text branch (Sec. 3.2). We leverage a VAE network for continuous motion representation (Sec. 3.Ip, and design separate training objective for each modality (Sec.
while maintaining synthesis quality. In parallel, to fit next-token prediction recipes in large language model (LLM), several works discretize motion with VQ- (Esser et al. 2021 Van Den Oord et al. 2017) into token indices, enabling transformer-bass generation (Zhang et al. 2023b 2024c mismatch"
2025 However, quantization induces approximation error and
that attenuates fine-graing grained kinematics and limits controllability, while refined tokenization such as residual VQ (RVQ) (Guo et al.) (024) and post-training schemes (Wang et al., (2025b) only partially allieviate these issues and cannot fundamentally avoid the numerical and semantic discontinuities induced by tokenization. Accordingly, we adopt an approach that interface language models directly with unquantized VAB latent representations.
Human motion modeling has evolved from task-specific designs to unified frameworks for multi- modal understanding and generation. Recent motion models (Jiang et al. 2023 Park al. 2025
Wang et al. 2024 Wu et al., 2023) adopt single-stream LM backbones with discretized motion tokens to support bidirectional mext-motion mapping, and have been attended other modalities such as music (Luo et al.) 2024: You et al., 2024). In parallel, language-centric multimodal frameworks extend LLMs beyond text via lightweight adapters and cross-attention conditioning, offering generic recipe for vision- and audio-gre -edoundd reasoning (Alayrac et al.) 20022 Li et al. 2023 Copet et al.)
2023 Liu et 2023a). Works such as NEXT-GPT (Wu al. (202bb) and Janus (Wu et al.) 2024a
Ma et al. 2024 employ pretrained encoders/decoders to map inputs into modality-specific latent spaces and attach adapters for flexible multimodal generation. In vision-language, Chameleon (Team)
2024) and Transfusion (Zhou et al. 2024) discretize encode images into token sequences to support interleaved text-image training, while Show-o (Xie et al. 2024) and Fuyu (Bavishi et al. 2023) employ masked or causal attention for joint reasoning. Despite their versatility, single-stream architectures often suffer from cross-modal interference, limiting scalability and robustness. Even with carefully tuned objectives, newly introduced modalities can disrupt existing representations, underscoring the challenge of preserving modaliti-specifit capability while scaling to new domains.
Mixture-of-Expere and Multi-Stream Architecture address these limitations by routing inputs modality-specific experts while maintaining a shared fusion interface (Alayrac et al. 2022 Li et al. 2023 Tsimpoukelli et al. [021). This separation reduces gradient interference between modalities and enables branch to be guided by its own objective (Liu et al., 2021 Sener & Koltun 2018). and
simplifies the introduction of new modalities. These insights motivate hybrid strategies (Cho et al.
2024, (Shi et al. 2025 Wang et al.) 2025a) that combine discrete and continuous representations and decouple modality-speciife encoders with minimal modification on the LLM backbone, thereby enhancing alignment and expressiveness. Mixture-of-TTrnsfor instantiates this idea with coddality-speciff Transformer experts coupled through shared attention. facilitating modular training and reducing interference when incorporating new modalities. Guided by these observations, we adopt MoT-style architecture that isolates motion representation learning while leveraging the language competence of pretrained LMs (Bai et al. 2023 Radford al.
(MoT) (Liang et al. 2024)
2019
3

---

Published as a conference paper at ICLR 2026
3 METHOD
To couple motion understanding and generation into language-centrie LLMs, we observe that although
discretization prior unified systems (Jiang et al.] 2023 Wang et al., 2024 Wu al. 2025) facilitates reuse of text-style training and inference pipelines, it inevitably removes fine-grained
details and complicates optimization. Moreover, single-stream backbones exacerbate cross-modal interference and yield imbalanced training. We circumvent these limitations by representing motion in continuous, perceptually faithful VAE latent space (Sec.3..1 and adopting hybrid motion-ter text backbone that processes the two streams separately while permitting controlled interaction via shared self-attention (Sec. [32]]. On top of this backbone, we attach a diffusion head conditioned on LLM hidden states to bridge language and motion latents, enabling bidirectional understanding and generation (Sec. 3.3). Finally, together with a shree-stage training schedule (Sec. (.4)). our MotionGPP33 avoids quantization bottlenecks and improves training and inference efficiency while maintaining generation quality.
3.1
MOTION REPRESENTATION CONTINUOUS TOKENS
To align motion with the autoregressive generation paradigm of large language models (LLMs)
(Bai
al. 2023; Radford et al. 2019 Raffel et al. 2020 Touvron et al., 2023). previous approaches
typically descretizes motion with vector qustor--uanizeed autoencoders (Esser et al. 2021 Van Den Oord
al. 2017), converting a N-length sequences into latents y E प and replace each vector E Rd by its nearest code eg. from a learned K-entry codebook. The corresponding indexes gi.... then serve as tokens for LLM training (Zhang et al., 2023b; Jiang et al., 2023 Wang et al. they integrate cleanly with standard next-token prediction objectives in LLMs such as cross-entropy, quantization process inevitably introduces approximation error and disrupts motion continuity, weakening fine-grained dynamics and constraining controllability. [uuo et al. (2024), equiped with residual vector quantization (RVQ), leverages multiple codebooks whose decoded latents are summed to reduce information loss. In parallel, [Wag al.) (2025b) explores refined post-training tokenization. However, neither strategy fundamentally resolves the numerical and semantic discontinuities inherent to discretization.
In contrast, we adopt a continuous latent space learned by a motion VAE. Given N frame motion sequence = the encoder & map m into compact continuous latent vector Rd, and the decoder ID reconstructs m I: = (D2) = ((E(m I:M) ). The VAE is trained once with a reconstruction term (optionally including kinematic losses on pose and velocity) and KL regularizer (Kullback & Leibler 1951) to prevent high-variance latents and promote a smooth manifold. This compressed, continuous representation learns the inherent structure of =, and preserves subtle variations and maintains numerical and semantic continuity, while providing a compact domain in which our downstream generator operates. Further details can be found in the supplement.
3.2 BIMODAL NOTION-LANGUNA FRAMEWORK
To accommodate the distinct characteristics language and motion while enabling efficient cross- modal interaction, we augment a decoder-only transformer backbone Radford al. (2019) with parallel motion branch. Unlike single-stream designs that merge all modalities into one pathway Jiang al. (023); y; Wu et al. (2025) our architecture preserves -odality-speeific routes: text branch 7 and motion branch M. Each branch maintains its own embeddings, feed-forward blocks, and normalization, and information exchange occurs only shared self-attention layers (Alayrac) et al. 2022 Shi et al. 2025). The motion branch is initialized from scratch and trained primarily under its own objective, thereby capturing motion-specift inductive biases and reducing cross-modal interference during multimodal training (Yu et al. 2020; Zhou al. 2023).
Hybrid Sequence Route As illustrated in Fig. 1 given an input sequence S = _.*k, each element is embedded either as text embedding T; or motion latent 2,, with a routing indicator ú; {0, 1} dispatches them to T or M. The branches compute hidden states hy and hm seperately, which then reassembled in input order for shared self-attention layers. This hybrid routing supports interleaved text motion processing without collapsing modalities into a single embedding space, laying the foundation for high-quality, condition-aware generation.
4

---

Published as a conference paper at ICLR 2026
Sage
Motion Pre-training
Cross-modal Alignment
Joint Optimization
Tokenizer
Tokentzer
Text Branch
*
Motion Branch
Motion Branch
Branch
Motion Branch
Oifluiion Head
M
Craso
+
Head
WM
Crows Eintropy
Head
M
(a) Model learn to generate motion properly.
(b) Bidirectional tasks eahances alignment.
(c) Fine-tunc all parameterr with instruction.
Motion Generation
Motion Understanding
D
(d) We -omotoo out> takens to query motion hidden states, then guide 2 to generate a single motion latent.
(e) Text tokens generated auto-regressive
manner until the <cos? is predicted.
Figure 2: We propose a three-stage alignment strategy for our hybrid motion-language model: (a) The text branch is frozen, and only motion output is supervised. (b) Motion reasoning is introduced further align the motion branch with language, with supervision on both modalities. (c) All modules are jointly fine-tuned with text branch unfrozen. (d)(e) shows inference time behavior of two branches which process data only with the same modality tags (differtiated by colors). Each rectangle block represents for whole text/motion branch, while shadowed ones denote inactive modules. Modalities are color-coded: blue for text and orange for motion. Shadowed orange squares out>, represent <motion _utioon and orange- outlind squares indicate boundary tokens <som> <eom>.
Interfaces for Continuous Motion Latents Because that the continuous motion representation (Xin et al. 2023) does not rely on tokenized vocabulary or codebook, the index-to-eminn -o-eeeddde lookup and softmax decoding employed for text cannot be reused for motion. We therefore introduce motion-specific interfaces that bridge continuous latents and transformer hidden states. First, we augment the text vocabulary with a small set of motion- boundary/holder tokens (i.c. <som> <com>, <motion jn>, <motion_ouu out>) to mark motion spans and I/O positions in interleaved sequences in Zhou et al. (004)). Second, a Motion Understanding Head (MUH) linearly maps motion latents into the Transformer's s input embedding space for captioning and prediction. Finally, lightweight Motion Generation Head (MGH) projects hidden states back to the VAE latent space via diffusion (Ho et al.
2020
Rombach et al. 2022) Xin al. 2023)
3.3
MOTION DIFFUSION IN AUTOREGRESSIVE BACKBONE
Continuous representations are inherently misaligned with the discrete nature of token-based genera- tion in LLMs, and requires more sophisticated modeling to support generation. Inspired by recent advances in diffusion- based generative modeling (Tevet et al. 2022b Song et al. 2020: Rombach jt al. 2022: Li et al.) 2024a), we attach a lightweight diffusion module 7 to bridge this gap. H. predicts motion latents directly from the backbone's 's hidden states, enabling integration of continuous motion representation within an autoregressive framework.
Diffusion Process Given ground- truth motion sequence x, obtain the target latent 2g E(x) Rd via the motion encoder E. We adopt a fixed forward noising process over / E (1....,T) with Gaussian perturbations: 2 v Zg + /1 1 ãg e, where _N 1) and ā; T' s= is the cumulative product of noise scheduling coefficients. A time-aware denoiser Transformer hidden states from the motion stream (denoted hm)) and learns to reverse the diffusion process. Conditioning is implemented via lightweight linear projections into the denoiser inputs, following recent practice (Ho & Salimans, 2022 Li et al.) 2024a). We train H with the standard
conditioned on the
DDPM objective
(Ho et al. 2020: Song et al. 2020 Laitt Ezg. t, [|e H ((2 L hmm)11
5

---

Published as a conference paper at ICLR 2026
Inference The text branch autoregressively generates tokens until motion start marker <som> is produced. We then insert K placeholder tokens (i.e., <motion out>) to elicit the span-all span-aligned hidden states h in a single forward pass. The diffusion head runs the reverse process conditioned on / to sample the noise-free motion latent žp, which the VAE decoder 'D finally decoded to the raw motion sequence. As in [Team] (2024), generation then resumes in the text stream with concatenated <com> until end-of-sequence. Operating on compact motion latents, this diffusion head adds only minimal overhead during training and inference.
3.4 TRAINING PROCEDURE
We first train a motion VAE to obtain compact, continuous latent space, following prior works (Rom- bach et al. 2022 Xin et al. 2023). The bimodal backbone then adopts pretrained decoder-only LLM (Radford et al. 2019) as the text branch T. As described in Fig. 2 the motion branch M is initialized from scratch and brought into alignment with TT through a three-stage schedule.
Stage I: fext-toomooionn pretraining We begin by pretraining M text-to toottonn while freezingT. This provides stable linguistic conditioning and biases the model toward motion-specific semantics. In this stage, M conditions on the frozen language representations and is trained via diffusion, to synthesize VAE motion latentss, where diverse text-motion pairs encourages a rich and flexible mapping from language the latent space (Rombach al.. 2022 Xin et al., 2023).
Stage II: Cross-Modal Alignment Keeping T frozen, we introduce additional objectives to couple understanding and generation. Concretely, training includes multiple tasks of text-to-motion (T2M), motion-to-text (M2T), and motion prediction. Following instruction-style formulations in Jiang et al. (2023), these tasks are further presented as prompts covering generation. captioning, prediction, and inbetweening. Multi-task optimization fosters bidirectional alignment without forcing single shared representation and encourages motion representations that are semantically coherent with language features (Alayrac et al. 2022 Li et al. 2023).
Stage III: Joint Fine-Tuning Finally, we unfreeze T and fine-tune all parameters via instruction tuning on mixture of paired text-motion data and, optionally, text-only prompts (Dai et al. 2023 Liu et al.) 2023b WWii et al., [02T]. Including text-only prompts can further improve language competence for downstream applications.
4 EXPERIMENTS
We empirically validate MotionGPT3, a dual-stream architecture for efficient, language grounddd multimodal motion understanding and generation, across motion-centric tasks. Dataset configurations, evaluation metrics, and implementation details are summarized in Sec. 4.1] Begin with analyzing optimization dynamics and inference efficiency via training loss and validation curves (Sec. (42)). we then present controlled ablations that isolate the contributions of the continuous VAE motion representation and the bimodal design (Sec. 4.4 Next, We benchmark MotionGPT3 on text-to- motion generation and motion-to-text understanding, comparing against both specialized single-task methods and unified state-of-thed systems (Sec. 4.3). Finally, we ablate the proposed three-stage training scheme (Sec. 4.4). Additional qualitative results are provided in the supplement.
4.1 EXPERIMENTAL SETUP
Datasets We train and evaluate our model mainly on HumanML3D Guo et al. 2022b). alarge-scale benchmarks for text-motion generation and understanding, and KIT-ML (Plappert et al.
2016) with
3,911 motionsequences. For comparison with prior works (Xin et al. 2023 Jiang et al. 2023). we adopt the 263-dim pose proposed in Guo et al. (20226), which combines joint velocities/ positions/ rotations, and foot-contact signals, following the standard data split.
Evaluation Metries We evaluate two tasks. For the text-to-motion, we follow the previous works (Guo et al. 2 Jiang et al.] 2023; Xin et al. 2023 Zhang et al. (023b) to report motion quality (FID), diversity (DIV and MM), and text-motion alignment RR-Precision and MMDist). For motion-to-text, use both alignment metrics (R-Precision and MMDist) and linguistic metrics from NLP (Bleu (Papineni et al. 2002). Rouge-L (Lin) 2004), CIDEr (Vedantam et al., 2013), and BertScore (Zhang al. 2019)). See Sec. D.3 for metric definitions and computation details.

---

Published as a conference paper at ICLR 2026
Diffusion Loss
Precissoo TOP
MultiModal Distance
0.3
0.23
VAE--Bimoda (Ours)
<4 speed
VAE+Uaifel
0.26
VQ+Binucal
~2x speed
0.24
0.22
2X speed
0.2
0.38
20
40
60 100
120
20 80 100
120
100
120
Figure 3: Training loss and validation curves on motion generation on HumanML3D for architecture variants of dual-stream and single-stream and representation variants of VAE and VQ latents. The right figures illustrate validation metrics of R- R-Precision TOP 3 (R @3)3f) and Multimodal Distance (MMDist)). Triangle markers indicate matched- checkpoints (~00222). Our hybrid architecture with continuous motion representation helps accelerating convergence for about 2x, well as achieves better quality especially in early training stage.
Implementation Details Our framework comprises three main components: a motion VAE, lightweight diffusion head, and a dual-stream backbone. We adopt the Transformer-basee motion VAE of al. (2023), where both encoder and decoder consists of 9 layers and 4 heads with skip connections, producing a 1 x 256 latent per motion sequence. Our Diffusion Head H is implemented as a 3-layer MLP with ResBlock-style layers and hidden dimension 1024, following Li al. (0024a). We train diffusion with scaled linear noise schedule for 1000 denoising steps, while inference uses 100 steps by default. The text and motion branches share the GPT-2 base configuration but use disjoint parameters, both are decoder-only with 12 Transformer layers, model dimension 768, and MLP dimension 3072, unless stated otherwise. The text branch is initialized from a pretrained 124M GPT-2 checkpoint, while the motion branch from scratch, yielding total 238M parameters.
Training Protocol We use A for all components, with a learning rate of 1 x 10-4 for the motion backbone and x 10-4 for the diffusion head. Training uses mini-batch size of 32 on NVIDIA RTX 3090 GPUs, with identical training/inference settings on HumanMM.33 (Guo et al. 2022b). The motion VAE is trained with learning rate of x 10- batch size 256, over 150K iterations. The motion-language backbone is trained for 100k iterations in text-to-motion pretraining, followed by 300k iterations for cross-modal modal alignment.
4.2 TRAINING EFFICIENCY WITH BIMODAL ARCHITECTURE AND CONTINUOUS LATENTS
To assess two main design choices: (i) the dual-stream (bimodal) backbone and (ii) continuous representation from motion VAE, we analyze training dynamics on the text-to-motion task under identical settings HumanML3D. Fig. plots training loss and validation metrics over time for three variants of single- stallessteam++ VAE, dual-stream- VAE, and V...-stream+VV.
We observe that Discrete VQ latents plateau at a lower quality ceiling. The VQ baseline (VQ + bimodal; green curve) reaches 20.5 R-Precision Top 3 (R @3) early in training and then saturates, yielding substantially lower @@3 and higher MultiModal Distance (MMDist) than the VAE- based counterparts. This likely stems from quantization-induuee
information loss (Guo et al.) 2024 Wang al. 2025a) and tokenization that disrupts the semantic continuity of motion. ii) Bimodal design accelerates optimization. Compared with single-stream backbone, the dual-stream variant reduces diffusion loss roughly 2x faster and sustains superior validation performance over the entire training trajectory, in terms of R @3 and MMDist. iii) Superior quality at matched optimization states. At comparable diffusion loss, the dual-stream model still leads. For instance, we mark by triangles a loss of 00.22, reached at 20 epochs for dual-stream model and 400 epochs for the single-stream model, the former achieves higher R@3 and lower MMDist.
We attribute these effects to modality-aware optimization: the motion branch learns motion-specific semantics while the language branch focuses on textual cues. Decoupling the streams and supervising them separately mitigates gradient interference and avoids the representational compromises common in single-stream models (Liu et al. 2021 Senerr & Koltun| 2018 Yu et al.) 2020). Although a shared space might be expected bring modalities "closer". in practice single-stream coupling can entangle modality structure and yield counterintuitive outcomes. In summary, continuous VAE latents within a dual-stream backbone strike favorable efficiency-fidelity trade-off, enabling high-quality motion synthesis with reduced training time.
7

---

Published as a conference paper at ICLR 2026
4.3 COMPARISONS
By modeling human motion as a second modality alongside language, our bimodal motion-language model supports both text-to-motion (T2M) and motion-to-text (M2T). We report results for two settings: (i) single-task models trained specifically for target task (MotionGPT3)), and (ii) a unified model (MotionGPT3) trained on both tasks with the shree-stage scheme
Mext-to-Motion Generation The text-to-motion task involves generating realistic and diverse motion sequences conditioned natural language descriptions. We evaluate single-task generator MotionGPT3t as well as a multi-task generator MotionGPT3. We compare MotionGPT33 against
recent methods (Guo al. 2202bb c; Lou al. 2023 Jiang et al. 2023; Xin al. 2023 Zhang
et al. 2023b c
Guo et al. 2024, Wu et al. 2024c Li et al. 2024b) Zhang et al., 2025 and evaluate
the performance on both HumanML3D and KTT-ML datasets. Following Guo
al. (2022b) each
evaluation is repeated 20 times and reported with 95% confidence intervals. As shown in Tab, and
Tab. 2 MotionGPT3i matches or exceeds generation-onj, baselines (Zhang al., 2023b Lou et al.
2023 Guo et al. 2024) on alignment metrics (R @@, k, MMDist), with competitive FID and diversity.
Further, the unified MotionGPP13peffomss better than recent unified systems (Guo et al. 2022c Jiang
al.
2023
Wu et al. 2024c 2025). See comaprison with more approaches in Sec. C.1
Motion-to- Text Understanding The motion-to-text task involves understanding motion sequences and generating semantically appropriate textual descriptions. We train single-task captioner MotionGPT3 for 100 epochs and compare with recent SOTA (Guo et al. 2022c Jiang et al. 2023 Li 2024b Wu et al., (2024c). Following Jiang et al. (2023). evaluate on the ground truth texts using the TM2T protocol (Guo et al. c022c). Results in Tab, 3 show that both MotionGPT31 and MotionGPT3 achieve strong retrieval performance and language metrics. Notably, observe marked reduction Multimodal Distance (MMDist), indicating effective motion-language alignment under the dual-stream design.
Table 1: Evaluation of text-guided motion generation on HumanMI.3D (Guo et al. 2022ap. Rows are grouped by training tasks: Gen. only for generation-only and Gen. & Und. for both. Real obtained by ground-truch motions, and + indicate values closer to Real are desirable. marks our single-task model trained for 200 epochs, and MotionGPT3 is a three-stage model trained with unified tasks. Best and second- best results are highlighted in bold and underline.
Types
Methods
R@1
R@2
R@B
FID,
MMDist
Divensity
MModality
Real
0.511
0.703
0.797
0.002
2.974
9.503
T2M-GPT Zhang all
2023b
0.491
0.680
0.775
0.116
3.118
9.761
1.856
DiverseMolica Lou al
2023
0.515
0.706
0.802
0.072
2.941
9.683
1.869
Gen, only
DitionDiffuse Zhing
20
0.491
0.681
0.782
0.630
3.113
9.410
1.553
MoMask 2t 2024
0.521
0.713
0.807
0.045
2.938
9.620
1,241
MotionGP13]
0.535
0.731
0.826
0.239
2.797
9.688
1.560
TM21 Guo
2022c
0.424
0.729
8,589
2.424
MotionGPT Jiank et ml
20233
0.492
0.681
0.733
0.232
3.096
9.528
2,00
Und.
Wu
2.399
MG MotionLEM. Wu
2025
0.516
0.706
0.802
0.303
2.952
9.960
2.125
MotionGPT3
0.553
Table 2: Evaluation of text-guided motion generation on KIT-ML
Plapperrt et al. 2016).
Methods
R@
R@2
R@3
FID4
MMDist
Diversity
MModality
Real
0.424
0.649
0.779
0.031
2.788
11.08
2023
0.734
3.204
2.192
MationGPT Jiang
2023
0.366
0.558
0.680
0.510
3.527
10.350
2.328
MationDiffung: Zhang cl 4024a
0.417
0.621
0.739
1.954
2.958
11.100
0.730
ReMoDiffuse Zhane el
20250
0.427
0.641
0.765
0.155
2.814
10.800
1,239
MoMask Guo
0.433
0.656
0.781
0.204
2.779
1.131
MoTe Wu
0.419
0.627
0.741
0.256
3.216
2.615
MoGen Yuan al.
2024
0.445
0.671
0.797
0.143
2.711
10.918
MotionGF133
0.803
2.704
0.9036
8

---

Published as a conference paper at ICLR 2026
Table Comparison motion captioning on H33 (Guo et al. 2022a), evaluation follows (Guo et al. 2022c). MotionGPT33 f denotes our single-task captioning model trained for 100 epochs. and MotionGPT3 an unified model trained on both tasks with the three-stage scheme (Sec. 3.4). Both variants achieve R@k on par with recent state of the art, and surpass the GT metrics.
Methods
R 601
R@2
R@3
MMDist |
Bleu@
Bleu@
Rouge 1
Cider 1 BertScore
Real
0.523
0,725
0.828
2,901
TM2T Guo et al. 2022c
0.516
0.823
2,935
48.9
7,00
38.1
16.8
32.2
MotionGiP Daang al 2023
0.543
0.827
2,821
48.2
12.5
37.4
29.2
32.4
LaMPM2T ( 2024b
0,547
0.831
2,808
47.8
13.04
37.1
28.9
32.7
MoTo (Wa èt 20240
0.577
0.871
2.649
46.7
11.15
37.4
31,5
30.3
MotionGP13
0.553
0.756
0.853
2,524
56,363
17.661
44.997
30.980
35,850
Mot
0.573
0.773
0.864
2,426
19.412
46.173
28.721
35.231
4.4 ABLATION STUDIES
This section reports quantitative ablations. In contrast to training-curve analysis Sec. 4.2 we evaluate final test-set performance on both text-to-motion (T2M) and motion-to-ter (M2T). First, we assess the contributions of durl-stream backbone and continuous VAE motion latents by varying one factor at time. Then, we analyze the proposed three-stage training schedule and quantify its effects We also examine the impact of hidden-state processing in the Diffusion Head H and the use of classifier free guidance (CFG). See Sec. C for more detailed experiments.
Model Design Tab. summarized test-set results on HumanML33D (Guo et al. 2022a). We evaluate T2M and M2T separately and compare four variants obtained by crossing architecture (single- vs. dual-stream) with representation (discrete VQ vs. continuous VAE). Under the same evaluation protocol, replacing VAE with VQ replacing dual-stream backbone (Bimodal) with single-stream one (Unified) consistently degrades performance on both tasks. Notably, changing the architecture change Bimodal yields larger gains on M2T, whereas changing the representation VAE yields larger gains on T2M. This deseddepndd sensitivity is consistent with Sec. [.22 decoupling streams mitigates cross- modal interference and benefits semantic-level alignment, while continuous latents reduce quantization loss and improve synthesis fidelity for motion generation.
Table 4: Component ablations on HumanML3D for representation choice and architecture design. Unified denotes single-stream backbone, where one branch shared by text and motion, as employed in Diang et al. 2023). and Bimodal denotes dual-stream backbone described in Sec. 3.2 VQ and VAE indicate discrete and continuous motion latents, respectively. For each configuration we train separate models for motion generation (T2M) and motion captioning (M2T) under the same protocol and report test-set metrics. All variants share the same GPT-2-style branch and hyperparameters. and training is run for 100 epochs on M2T and 200 epochs on T2M. Best and second-bes best results highlighted in bold and underline.
Text-lo-Motion
Motion-to Text
Settings
R@
R@31
MMDist
R@11
R@31
MMDist
BertScore
Real
0.511 200000
20.002
2.974 177429.0000 008
0.002
0.523
0.828
2.901
Unified+VO
0.237
0.435 0.445
5.684 5.684+0.015 +0.018
0.403 0.403-0.014
VAI.
0.501 =0 co0
0.792
20.002
2.841#0.001
0 0.489=0.or 0.17
0.234
0.426
5.976
16.197
Vi
0.300 00000=0.00 005
0.002
4.937+00007 077
0.454=0.075
0.379
0.702
3.545
VAn
0.53340.00
0 00a
2.797
0.239-0.000
0,553
0,853
2,524
35.850
T2M with Bimodale and Unifee extended 400 epochs appreach convergence.
M2T resalts for Unified+VV VQ is not reported are omnited because performance remained unevaluable after more than 400 training epochs.
Cross-Modal Attention We further enable CMA in the last T. layers and sweep I. (Fig Increasing I, from to generally improves quality, reflected by lower FIID scores and MMDist and higher R-Precision. However, the trend is non-monotonic:
observe slight drop at L 6 in R-Precision, relative to L = 5, may suggesting that late but not ubiquitous CMA is preferable. More results on CMA schedules can be found in Sec.
Training Stage We ablate the three-stage schedule Sec. 3.4 including 100k iters on text-to- motion pretraining (SI), 300k iters cross-modal alignment (SII), and 50k iters joint fine-tuning (SIII), and evaluate on both T2M and M2T. Results are summarized in Tab. S SI already yields strong generation and provides motion-specialized initialization. Optimization in SII confers

---

Published as a conference paper at ICLR 2026
Recall (%)
MMDist
3.1
3.0
2.9
Rg01
2.8
Number CMA layers (L)
Number CMA layers (L)
Ablation on the number of cross-modal attention (CMA) layers for T2M on HumanMI.3D. CMA is enabled in the last L layers (L E I,...,6), 6). Performance improves as increases up to layers, then shows slight degradation at 6, indicating non-monotonically pattern.
Figure 4:
Table 5:
Ablation on training-scheme Enabled stages are marked with V. and colors encode the text branch updated or frozen. Best results are bold and second best are underlined.
Text- to Motion
Motion-to- Text
Stage I
Stage II
Stage III
R TOP3 f
FID 4
MMDist
R TOP1
Bleu@44
Bert
x
0,826
0.239
2.797
0.831
0.215
2.755
0.571
18.328
33,993
0.837
0.208
2.725
0.573
19.412
35,231
X
0.772
0.325
3.108
0,573
18.277
35.546
M2T capability and, importantly, further improves T2M, by -0.10 on FID and -0.2 on MMD. indicating that explicit alignment benefits both directions. Without extra text-only supervision, SIII adds small additional gains on M2T while preserving T2M, serving as light joint refinement rather than substitute for $2. shown the last row of Tab. two-stage model that omits keeps M2T largely intact but markedly degrades T2M, underscoring the role of S1 in learning motion- specific features. Overall, the full three-stage schedule provides the best trade-off, delivering reliable generation and captioning with well-aligned motion motion-language representations. Additional variants are reported in Sec. C.6. including experiments with an unfrozen text branch.
DISCUSSION
To address quantization-induued degradation and cross-modal interference in multi-objective training, we present MotionGPT3, a dual-stream motion-language framework that unifies motion under- standing and generation while preserving modality-specffic inductive biases. By encoding motion as continuous VAE latents and generating in latent space with lightweight diffusion head, the model avoids quantization artifacts and improves synthesis fidelity. The dual-strean Transformer with shared attention enables controlled bidirectional exchange, which strengthens text-motion alignment, reduces cross-modal interference, and empirically accelerates single-task convergence without degrading quality. For joint training of understanding and generation, generate-then-alinn three-stage training schedule further stabilizes optimization and mitigates cross-tast -ask interference.
Fine- aine-grained control cam fail directional cues (e.g., left/right). Because the current VAE yields a single latent per sequence, segment-level composition and local semantic alignment for long motions descriptions is constrained by data coverage. Potential remedies include incorporating diverse text- only corpora in the final alignment stage, adopting stronger language backbones, and exploring hierarchical or segment-wise latent representations to enable compositional control. Future Work We will scale training to (i) larger, more diverse datasets, (ii) develop controllable motion with local semantic alignment and segment-level for long morizon generation, and (iii) evaluate the framework with stronger language models and larger-scale training regimes to assess efficiency and robustness.
Limitations and Failure Cases
not explicitly supported. Generalization out-of-domain
Acknowledgment This work was supported in part by NSF China (No. 62172363). This work was supported by the JC STEM Lab of AI for Science and Engineering, funded by The Hong Kong Jockey Club Charities Trust, the MTR Research Funding (MRF) Scheme (CHU-24003), the Research Grants Council of Hong Kong (Project No. CUHK142132244
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Andrea Agostinelli, Timo Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, ct al. Musiclm: Generating music from text. arXiv preprint arXiv:2301. 113.5, 2023.
Jean-Baptiste Alayrae, Jeff Donahue, Pauline Luc, Antoine Miech, Tain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-2377 237336 2022.
Nikos Athanasiou, Alpár Cseke, Markos Diomataris, Michael J Black, and Giil Varol. Motionfix: Text-driven 3d human motion editing. In SIGGRAPH Asia 2024 Conference Papers, pp. 1-11, 2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenhin Ge, Yu Han, Fei Huang, et al. Qwen technical report, arXiv preprint (tt:::3309.660009 2023.
Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye. Augustus Odena, Arushi Somani, and Sagnak Taşrrlr. Introducing our multimodal models, 2023. URL
Fuyu- 8b
Ling-Hao -aa Chen, Shunlin Lu, Ailing Zeng, Hao Zhang, Benyou Wang, Ruimao Zhang. and Lei Zhang. Motionllm Understanding human behaviors from human motions and videos. arXiv preprint arXiv:2405.203:: 2024.
Jungbin Cho, Junwan Kim, Jisoo Kim, Minsco Kim, Mingu Kang, Sungeun Hong, Tac-Hyun Oh, and Youngjae Yu. Discord: Discrete tokens to continuous motion via rectified flow decoding. arXiv preprint 2rX1:2211.19527, 2024.
Jungbin Cho, Junwan Kim, Jisoo Kim, Minseo Kim, Mingu Kang, Sungeun Hong, Tae-Hyun Oh, and Youngjae Yu. Discord: Discrete tokens to continuous motion via rectified flow decoding, 2025. URL ptt
/ //arxiv.orr wwtp:oorggabbb/aaaee/e 19527
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay. William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, al. Scaling cnatuuction-inetunee language models. arXiv preprint arXiv:2210.1 1416, 2022.
Setareh Cohan, Guy Tevet, Daniele Reda, Xue Bin Peng, and Michiel van de Panne. Flexible motion in- betweening with diffusion models. In ACM SIGGRAPH 2024 Conference Popers, pp. 1-9, 2024.
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnacve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation. Advances in Neural Information Processing Systems, 36:47704 2023.
Wenliang Dai, Junnan Li. Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung. and Steven Hoi. Instructblip: Towards ceneral-purpose vision-language models with instruction tuning. Advances neural information processing systems, 36:49250-4999 492677 2023.
Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal comprehension and creation. arXiv preprint 1rXiv:2309.11499 2023.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, et al. Palm-e: An embodied multimodal language model. 2023.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 12873-12883, 2021.
Chuan Guo. Shihao Zou, Xinxin Zuo, Sen Wang. Wei Ji, Xingyu Li, and Li Cheng, Generating diverse and natural 3d human motions from text. In Proceedings of the ILEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3152-5161, June 2022a.
Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang. Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 5152-5161, 2022h.
11

---

Published as a conference paper at ICLR 2026
Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng. Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts. In ECCV, 2022c.
Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. Momask: Generative masked modeling of 3d human motions. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 1900-1910, 2024.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:220.11259. 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic modcls. Advances in Neural Information Processing Systems, 33:6840 6851, 2020.
Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu. Gang Yu, and Tao Chen. Motiongpt: Human motion as foreign language. Advances in Neural Information Processing Systems, 36:20067-2000 20079, 2023.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 7482-7491, 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 1312.61 2013.
Dan Kondratyuk, Lijun Yu, Xiuye Gu. José Lezama, Jonathan Huang. Grant Schindler, Rachel Hornung. Vighnesh Birodkar, Jimmy Yan, Ming-Chang Chiu, et al. Videopoct: A large language model for zero-shot video generation. arXiv preprint arXiv:2312.11125, 2023.
Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics, 22(1):79-88 86, 1951.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pp. 12888-12900. PMLR, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre- pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2.30. 1.0.. 2023.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. In A. Globerson, L. Mackey, Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (cds.), Advances in Neural Information Processing Systems, volume 37, pp. 264244-6645. Curran Associates, URL Inc., 2024a.
neur
Zhe Li. Weihao Yuan, Yisheng He, Lingteng Qiu, Shenhao Zhu, Xiaodong Gu, Weichao Shen, Yuan Dong, Zilong Dong, and Laurence T Yang. Lamp: Language-motin pretraining for motion generation, retrieval, and captioning. arXiv preprint arXiv:2410.07003, 2024h.
Weixin Liang, Lili Yu, Liang Luo, Srinivasan Iyer, Ning Dong, Chunting Zhou, Gargi Ghosh, Mike Lewis, A sparse and scalable architecture for multi-modal foundation models. arXiv preprint arXiv:241 2,0449996, 2024.
Wen-tau Yih. Luke Zettlemoyer, et al.
Chin- Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 81, 2004.
Bo Liu, Xingchao Liu, Xiaojic Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 34:1887: 11888-18899) 2021.
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mci, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audhe audio generation with latent diffusion models. arXiv preprint arXiv:2301.125000 2023a.
Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiugiang Kong. Qiao Tian, Yuping Wang. Wenwu Wang. Yuxuan Wang. and Mark D Plumbley, Audioldm 2: Learning holistic audio generation with self-supervised pretraining. JEEE/ACM Transactions on Audio, Speech, and Language Processing, $2:2871-2883 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neurul information processing systems, 36:34892-34916, 2023b.
Yunhong Lou, Linchao 7hu, Yaxiong Wang. Xiaohan Wang, and Yi Yang. Diversemotion: Towards diverse human motion generation via discrete diffusion. arXiv preprint arX0v:2309.013772, 2023.
12

---

Published as a conference paper at ICLR 2026
Mingshuang Luo. Ruibing Hou, Zhuo Li, Hong Chang, Zimo Liu, Yaowei Wang. and Shiguang Shan. M gpt: An advanced multimodal, multitask framework for motion comprehension and generation. arXiv preprint arXiv:2405. 16273, 2024.
Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu. Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie. Haowei Zhang. Liang Zhao, et al. Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation. arXiv preprint urXiv: 1..1111079777 2024.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 31 1-318, 2002.
Jeongeun Park, Sungjoon Choi, and Sangdoo Yun. A unified framework for motion reasoning and generation in human interaction, 2025. /tlphtts:///xxiv.ooro
Mathis Petrovich, Michael J. Black, and Güil Varol. Action-conditioned 3D human motion synthesis with transformer VAE. International Conference on Computer Vision (ICCV), 2021.
Mathis Petrovich, Michael J. Black, and Gül Varol. TEMOS: Generating diverse human motions from textual descriptions. European Conference on Computer Vision (ECCV), 2022.
Mathis Petrovich, Michael Black, Gül Varol. Tmr: Text-to-motion retrieval using contrastive 3d human motion synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9488-9497, 2023.
Matthias Plappert, Christian Mandery, and Tamim Asfour. The kit motion-language dataset. Big Data, 4(4): 236 252, dec 2016. doi: 10. 20109//bi.2016..00 0028. URL http:////........rg///000.00 wtt.:////..001//....22.2.
dol.
0028
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang. Jianxin Yang. Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Kegin Bao, Kexin Yang. Yu, Mei Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu.
Qwen2.5 technical report, 2025, URLhttps: aariiv..vv...a//aaa/.aaaaaaaaa.a org/ abs 2412.1511.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models unsupervised multitask learners. OpenAl blog, 188):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yangi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer lcarning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and BjAmm Ommer. High-resolutinn image synthesis with latent diffiusion models. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2022. URL http:: //tp::/ggtttttb www.iitt....oooooooeeeeeeeeeeeee
7/arxiv. wttp:/rg/aaba///////e// 10752
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pp. 234-244 244.241.56 Springer, 2015.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 108, 2019.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. Advances in neural information processing systems. 31, 2018.
Yonatan Shafir, Guy Tevet, Roy Kapon, and Amit Bermano. Human motion diffusson as generative prior. arXiv preprint arXiv:2303.01410 2023.
Weijia Shi, Xiaochuang Han. Chunting Zhou, Weixin Liang, Xi Victoria Lin. Luke Zettlemoyer and Lili
Yu. Lmfusion: Adapting pretrained language models for multimodal generation, 2025. URL httpa
//arsiv..org/abs//112..55
13

---

Published as a conference paper at ICLR 2026
Li Siyao, Weijiang Yu. Tianpei Gu, Chunze Lin, Quan Wang. Chen Qian, Chen Change Loy, and Ziwei Liu. Bailando: 3d dance generation by actor-critic gpt with choreographic memory. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1050-11059, 2022.
Jiaming Song, Chenlin Meng. and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint 295.882405.08818 2024.
Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or. Motionclip: Exposing human motion generation to clip space. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXII, pp. 358-374. Springer, 2022a.
Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Amit H Bermano, and Daniel Cohen-Or. Human motion diffusion model. arXiv preprint arXiv:2209. 14916, 2022b.
Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H. Bermano. Human motion diffusion model. arXiv preprint arXiv:2209. 14916, 2022c.
Hugo Touvron. Thibaut Lavril. Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint a3Xiv:2302. 13971, 2023.
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34: 200 212,222222222 2021.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Ramakrishna Vedantam, C I awrence Zitnick, and Devi Parikh. Cider: Consensus-basce image description evaluation. In Proceedings of the IEF. conference computer vision and pattern recognition, pp. 4566-4575, 2015.
Wenhui Wang, Hangbo Bao, Li Dong. Johan Bjorck, Zhiliang Peng. Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a forcign language: Beit pretraining for vision and vision-language tasks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 19175-19186, 2023.
Yuan Wang, Di Huang, Yaqi Zhang, Wanli Ouyang, Jile Jiao, Xuetao Feng, Yan Zhou, Pengfei Wan, Shixiang Tang, and Dan Xu. Motiongpt-2: A general-purgy purpose motion-language model for motion generation and understanding. arXiv preprint 100247241, 2024.
Yuqing Wang, Zhijie Lin, Yao Teng, Yuanzhi Zhu, Shuhuai Ren, Jiashi Feng, and Xihui Liu. Bridging continuous and discrete tokens for autoregressive visual generation. arXiv preprint arXiv:2503. 16430, 2025a.
Yuqing Wang, Zhijie Lin. Yao Teng, Yuanzhi Zhu, Shuhuai Ren, Jiashi Feng. and Xihui Liu. Bridging continuous and discrete tokens for autoregressive visual generation, 2025b. Ittt::: httpss///aavvv.. 6430
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.0163: 2021.
Bizhu Wu, Jinheng Xie, Keming Shen, 7he Kong, Jianfeng Ren, Ruibin Bai, Rong Qu, and Linlin Shen. Mg-motionlim: unified framework for motion comprehension and generation across multiple granularities. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 27849-27858 2025.
Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xic, Xingkai Yu, Chong Ruan, el al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint 2410: 13848, 2024a.
Shengqiong Wu, Hao Fei, I Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal lmm In Forty-first International Conference on Machine Learning, 2024b.
14

---

Published as a conference paper at ICLR 2026
Yiming Wu, Wei Ji, Kecheng Zheng, Zicheng Wang. and Dong Xu. Mote: Learning motion-text diffusson model for multiple generation tasks, arXiv preprint arXiv:2411 2024c.
Jinheng Xie. Weijia Mao, Zechen Bai, David Junhao Zhang. Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint or 240: 12528, 2024.
Chen Xin, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, Jingyi Yu, and Gang Yu. Executing your commands via motion diffusion in latent space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 2023.
Kangning Yin, Shihao Zou, Yuxuan Ge, and Zheng Tian. Tri-modal motion retrieval by learning a joint embedding space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1596-1605, June 2024.
Fuming You. Minghui Fang. Li Tang, Rongjie Huang. Yongqi Wang, and Zhou Zhao. Momu-diffusion diffusion: On learning long-term motion-music synchronization and correspondence. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in neural information processing systems, 33:5824-5836, 2020.
Weihao Yuan, Yisheng He, Weichao Shen, Yuan Dong, Xiaodong Gu, Zilong Dong, Liefeng Bo, and Qixing Huang. Mogents: Motion generation based on spatial-temporal joint modeling. Advances in Neural 2024.
Information Processing Systems,
Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.0288 2023a.
Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, and Xi Shen. T2m-gpt gpt: Generating human motion from textual descriptions with discrete representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2023b.
Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, and Ziwei Liu. Remodiffuse: Retrieval augmented motion diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 364-373, 2023c.
Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motion- diffuse: Text-driven human motion generation with diffusion model. IEFE transactions pattern analysis and machine intelligence, 46(6):4115 4666):4115-4128, 2024a.
Mingyuan Zhang, Daisheng Jin, Chenyang Gu, Fangzhou Hong, Zhongang Cai, Jingfang Huang, Chongzhi Zhang, Xinying Guo, Lei Yang, Ying He, et al. Large motion model for unified multi-modal motion generation. In European Conference on Computer Vision, pp. 397-421. Springer, 2024b.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 2019.
Yaqi Zhang, Di Huang, Bin Liu, Shixiang Tang, Yan Lu, Lu Chen, Lei Bai, Qi Chu, Nenghai Yu, and Wanli Ouyang. Mationgpt: Finetuned llms are Conference on Artificial Intellience, 2024c.
motion generators. In Proceedings of the AAAI
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.0271 2024d.
Zeyu Zhang, Yiran Wang, Wei Mao, Danning Li, Rui Zhao, Biao Wu, Zirui Song, Bohan Zhuang, Ian Reid, and Richard Hartley. Motion anything: Any to motion generation. arXiv preprint arXiv:2503.06955 2025.
Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint 2rXiv:2408. 1109, 2024.
Yuwei Zhou, Xin Wang, Hong Chen, Xuguang Duan, and Wenwu Zhu. Intra-and inter-modal curriculum for multimodal learning. In Proceedings of the 31st ACM International Conference on Multimedia, pp. 3724-3735, 2023.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
This appendix provides qualitative comparison results (Sec. B). disccusion of continuous/ discrete motion representation (Sec. A). additional quantitative results and ablation (Sec. [ motion branch size and connection type, motion supervision scheme, training stages. We also provide more implementation details in Sec. Please note examination of metrics report TMR evaluator (Sec. CC2). analysis on bimodal architecture (Sec. C.3 ablation on our training scheme (Sec. C.6). and more language backbones (Sec. C.7).
Website & Video A supplementary website provides visualizations of quantitative results, mo- tion data, and demonstration videos. standalone video is also available on the website and at supp/webst seeevvvviii text-to-motion comparisons, (ii) motion- coptioncaptiionng comparisons, and (iii) additional results on motion generation and captioning.
cettti/vvieoo /Mot i OnGPT3/ //t//Voeeo. .mp, showcasing (i)
Code Example code files are available in the supplementary materials, which cover the training and evaluation processes of our MotionGPT33 models, along with several example results.
A DISCRETE TOKEN VS CONTINUOUS TOKEN
Reconstruction Task We compare continuous latents from MLD VAE (Xin et al. 2023) with discrete latents from VQ- IQ-VAE (Jiang et al. 2023) for motion reconstruction under identical settings. As shown in Sec. A VQ-VÄE yields higher errors on MPJPE, PAMPJPE, ACCL, and APE/AVE for root, trajectory, pose, joints, indicating reduced fidelity and temporal smoothness relative to the continuous VAE. This gap is expected: quantization maps a continuous motion manifold to a finite codebook and turns motion modeling into token classification, introducing unavoidable approximation error and information loss. In practice, many distinct frames collapse to the same code (one-to-many mapping), yielding ambiguous reconstructions and frame-wise noise that harms smoothness.
Table Reconstruction performance of a continuous VAE (Xin al. 2023) versus discrete VQ- VAE (Jiang et al. (003)) The VQ-VAE shows consistently higher errors, consistent with information loss introduced by quantized encoding and decoding. Sec.
D.3 presents the metric definitions.
APE
AVE
Method
MPJPE
PAMPIPE
ACCL
root
posc
joints
root
truj
pose
joints
VAE
43.906
31.356
5,93
0.0581
0.0504
0.0277
0.0619
0.0179
0.0177
0.0012
0.0185
VQ
46.828
33.668
7.629
0.0829
0,0804
0.0316
0.0930
0.0240
0.0239
0.0015
0.0253
On Discrete VQ Latents Recent work Guo al. (024) addresses the limitations codebook capacity using hierarchical Residual Vector Quantization (RVQ) with separate predictors for base and residual tokens. Wang al. 20256) combines quantized and continuous latents via post-training quantization. [ho et al. (2025) introduces diffusion- -asee decoder that progressively maps discrete tokens back to continuous raw motions, improving fidelity and smoothness, and uses the symmetric Jerk Percentage Error (sJPE) to detect under-reconstruccion and frame noise. Despite these advances, discrete pipelines remain prone to expressiveness bottlenecks and token-induced jitter. We further evaluate VQ and VAE latents within dual-stream architecture under single-task training. The results (Tab. 5) consistently favor continuous representations for both generation and understanding. While discrete codes facilitate token- based modeling, continuous representations better capture fine-grained dynamics and motion continuity, achieving higher alignment and synthesis quality with fewer epochs.
Table Discrete (VQ) vs. continuous (VAE) motion representations under single-task training. We report both T2M on R@ 1, FID, MMDist, DIV and M2T on R @@3, 3, BLEU@ 1/4, ROUGE. With fewer training epochs, VAE-variants achieve stronger alignment and better quality. VQ requires extended training of 399 epochs while still remains behind on most alignment and language scores.
Text-to- Text-to-Mation
Mation to: Text
Representation
epoch
R@ 11
FID
MM Dist
DIV
epoch
R@3*
Bleu@ 11
Bleu@4f
Rouget
VQ
199
0.258
0.542
5.364
9.274
99
0.765
47.043
7.234
39.244
399
0.300
0.454
4.937
9.626
199
0.752
41.579
6.304
35.746
VAE
199
0.525
0.191
2.667
10.095
99
0.859
50,707
8,383
38.225
16

---

Published as a conference paper at ICLR 2026
Generation Task (T2M) With the same 200 training epochs, the VAE representation delivers substantially better quality than VQ. with 0.525 vs. 0.258 on R@ and 2.667 vs. 5,364 on MMDist, while maintaining competitive diversity(DIV. Extending VQ to 399 epochs reduces FID to 0.454, however, it still lags in alignment, with R @ 0.300 and MMDist 4.937, indicating a lower quality ceiling for discrete codes.
Understanding Task (M2T) At matched training over 99 epochs, VAE attains stronger retrieval and language scores than VQ, with 0.859 vs. 0.765 on R@3 and 50.707 vs. 47.043 on BLEU@) @ 1, while ROUGE is comparable. Prolonged VQ training to 199 epochs unexceptedly reduces perfor- mance across all language scores with @@3 dropping by 0.13 and BLEU@ by 5.464, suggesting optimization instability and poorer generalization under the discrete setting.
B QUALITATIVE RESULTS
We provide qualitative comparisons for text-to-motion (T2M) Fig. 5 motion-to-text (M2T) Fig. 6 and multi-task gallery Fig. 7 To ensure fair visualization, we use fixed list of prompts from the HumanML3D test split; all clips are rendered at 20 FPS with identical camera and skeleton settings. Baselines are run with the their official checkpoints. Overall, MotionGPT3 exhibits stronger smoother transitions and text-motion alignment. Discrete-code variants tend to show token-induced frame noise and temporal drift, whereas single-stream models can produce semantically inconsistent motions on complex prompts. Additional examples and animations are provided in the supplementary video.
C ADDITIONAL EXPERIMENTS
This section provides supplementary experiments for further evaluation of MotionGPT3, First, we report comprehensive comparison with more baselines on texl-to-motion generation (Sec. C.T and further assess rext-to-motion with the TMR retrieval evaluator (Sec.
We then analyze design choices of the Cross-Modal Attention (CMA) (Sec. C.3). examine scaling effects of both language and motion branches (Sec. C.4. analyze the diffusion-based supervision used for motion generation (Sec. C.5), and evaluate the effectiveness of the three-stage training strategy (Sec. (.6). Fianlly, we further provide additional experiments with more language backbones (Sec.
C.I QUANTITATIVE RESULTS
Tab. reeoorss the full text-to-motion results on HumanMI.3D, grouped by training regime (generation- only vs. unified dual-task). Notably, evaluated on the HumanMI.31 dataset by T2M evaluator (Guo al. 2022b), recent models consistently achieve very high scores, and several recent approaches [Guo et al.) 2024/Zhaagg et al.) 2025) Li et al. [2024b] Wu et al. (004cc 2025), including MotionGPT3, achieve scores even above those of the ground- -trutt data (Real).
Table 6: Comprehensive comparison of text-to-motion generation on HumanML3D Guo al. 2022ah. We report generation-only models (Gen. only) here, and visualize unified dual-task models (Gen, & Und.) in Fig. g Real denotes ground -truth statistics; arrows ( (-+ indicate that values closer to Real are desirable. marks our single-task model trained for 200 epochs, and MotionGPT3 is the unified staee-stage model. Best and second-best results are bold and underlined.
Tyys
Malledls
R
MMDial
MModaliyy
Real
0
0.703
0,797
0.00229
2,074
0.30g
T2M Guo
0.467 1.007
0.639 -000
0.74 203
1.367 .022
3.34 1-000
202
2.09 f.045
MLD Kin &t ml A344
0.481
0.073
0.772 0012
0.47g
3.16g 01
B.724 262
2 2.414
Mitienhilinzz Ziure al 20p4
0.491 h1001
0.681 0.681
0. 0.782"
0.630
3.115
9.410
1 1.56
T2M-GPT
0.491 t.003
0.68 h.00
0.775 .002
0.116 .004
5.118 f.
9.761 201
1 1.01
Gen. unly
ReMoDilluuz
0.510
0.008 ond
0.795 2014
0.103 m04
2.974#-010 ou
9.018 306
1.795 179.t.00
DivascMarisa
Lune 3005
0.516 1.00a
0.706 -002
0.802
.002
0.072 1.04
2.941
9.685
202
1 1880 1.00
McMask
Guo A Ai14
0.521 f.002
0.713
0.807
0.045
2,958
9.620
1.241
201
0.719==000 002
0.812 .002
0.033 001
2,867 on0
9. 07
Modionknyliing
Ziaang c.
0.546 F.0ca
0.735 .002
.002 0.529
0.028 .005
2.8 F.01
9,521 1-osa
2.705 1.00
MotnuGPP33
0.60g
0.731
0.526
2, 2,707 002
8.688 107
1 1.06p
TM2T
0.424$.003
0.618 000
0.72g .012
1.501 046
3.407 000
8.58p 205
2.424 003
MoccnGPTimaa Jank ata
20:3
0.4024.00
0.081
0.73g cos
0.202 on8
3.006 one
9. 9.52g 091
2 2,008 084 2.137
Monep(iP1 1 Wine 41)
0.002
0.001
0.782
0. 0.181
3.08
1
Gen, & Und.
LiMP 2025h
0.557
0.51
0.34 001
0.002 002
2.739 007
9.371 069
Mole
0.545
0,737 000
0.825 oo3
0.075 os4
2,867 012
2.399 1775
MG Ma Wa
2025
0.516 F.022
0.706
0.802 0003
0.303
2.052 000
9.960 073
2. 2.125 135
MothnGPP33
0.00g
0.747
0.537
0.208
2.725
9.700
1.018
17

---

Published as a conference paper at ICLR 2026
"A standing throws something forward, which makes step forword fout."
"Person walks the right and makes u-turn clockwise and returns to the left initial position facing away.
"A person squats down, her left leg crosses her right leg, and then she stands back up.
MotionGPT
MoMask
mid
Ours
person takes slighly hurried
"A person walks forward until he loses his balance and rilts for to the left, then stumbles to
steps without raising his orms..
MP
MoMask
mld
Ours
Figure 5: Qualitative comparison of text-driven motion generation HumanML3D Guo al.
2022ah. Baselines (Jiang et al.) 2023 Guo al.) 2024 Xin et al. 2023) are run with their official released checkpoints. Red annotations (text, boxes, curves) highlight prompt-motion mismatches. Our bimodal motiion-laguuree framework yields motions that with closer correspondence to the textual prompt and smoother temporal coherence.
18

---

Published as a conference paper at ICLR 2026
Input Motions
person with his hands his
"figure walks forward 3 steps with arms straight in fronz of him."
"a person who sccms to evade something from left side and run
Real
hips does two brief squats."
at insane pace.
"a person sits down in chalr and stands up.
"a person runs forward, then turns and the right.
MotionGPT
"person walking down slowly."
"a person stand with their leg aport and arm bent the elbow.
"a person valk: forward with both raise above their head."
"a person run forward and jump over something.
TM2T
"a standing step slightly blegged
"a man walks slowly in a zombie
"running, jast against small obstacle:
Ours
with right leg arched bend."
walk with his stretched in front of him.
Figure 6: Example results on motion caption. The misalignment is hilighted with red.
Text-to-Motion Task
"someone execures roundlhouse kick with his left foot."
to figure steps backward cookily, swinging their
"a person walks curved line."
person runs right and then the left and continues to run then stops.
"Person ruus forward, panseex, then continues running
Motion-to-Tue Text Task
"A person swings their left arm back to side then throws something with their right hand."
"Person right and then right, while rasing their right hand."
person raises their right and puts it back down.
Textual Question-to-Answer Task
Can you give me a random motion description?
"A person hops forward then turns comnterclockwwse and starts walking back." "A person hops four times, walking from right to left." " "A person lifts both his hands and rubs them.
Figure 7: Gallery for the results of MotionGPT3.. Top: ext-to-motion generation. Middle: motion-to text captioning. Bottom: textual question answering about motion. Examples produced by our unified model trained with instruction based objectives (three-stage scheme). Animated visualizations provided in the supplementary video.
19

---

Published as a conference paper at ICLR 2026
MotionGPT3
MationsPP
LaMP
MG-MationLLM
TM2T
MationGPT-2
MoTO
80
3.6
Recal (%)
60
2.7
MMDist
40
1.8
20
0.9
R@1
R@3
MMDist
Figure 8: Comparison on mext-to-motion, with recent approaches trained with unified tasks(Gen. & Und.). Our model performs betther than recent unified models: MotionGPT (Jiang et al. (023).
TM2T (Guo et al. 2022c), MotionGPT-2 (Wang et al. 2024), MG-MotionLLMM (Wu et al. 2025).
MoTe (Wu et al. 2024c), and comparable with LaMP (Li et al. 2024b).
Generally speaking, high evaluation scores indicates that the generated motions can correspond well with the text, and approximate high-fidelity motions. However, considering that the GT embeddings effectively represent an upper bound for matching scores, the practical significance of differences among methods achieving near or above GT performance might be limited. This reflects limitation of the T2M evaluator, where the metrics are computed in a learned embedding space which relies on contrastive learning on HumanML3D, Methods that overfit to that space can saturate the proxy and even surpass the ground-trun reference, without a proportionate improvement motion fidelity. Hence, differences among methods near or above the Real line are hard to interpret.
C.2 EVALUATION WITH TMR
For more nuanced assessment, we further evaluate with the TMR retrieval framework (Petrovich) et al. 2023), under four gallery protocols: All, entire test set, All with threshold, gallery items whose textual similarity to the query exceeds fixed threshold, Dissimilar, 100-pair subset with mutually distant texts, and Small batches, mini-batch of size 32 to mimic/Guu et al. (2022b) setting. We report
text-motion retrieval with RØ 1/2/3/5/10 and MedR, results are summarized in Tab. 7
Not all methods in Tab. 6 release TMR results or checkpoints, so the comparison is limited to publicly available models. Within this scope, although inferior to Li al. (2024b) on T2M metrics (Fig. (8. our method achieves significantly stronger retrieval under the TMR evaluator and improves substantially over the T2M baseline Guo al. (2022b) across protocols. These findings suggest that our model achieves more robust cross-modal alignment rather than overfitting to specific evaluator.
Table 7: Retrieval on HumanML3D with the TMR evaluator (Petrovich et al., 2023). We report @ 1/2/3/5/10 and MedR for text-motion retrieval under the four official protocols: (a) All, (b) All with threshold, (c) Dissimilar subset, and (d) Small batches (see Sec. C.2 for definitions). Results for TEMOS (Petrovich et al. (022). T2M (Guo et al. (202b)), and TMR (Petrovich et al. 2023) are taken from the TMR paper. LaMP (Li et al., (024b) is reported only for (d). MotionGPT and MoT are evaluated with the released checkpoints using the official TMR code. Mo- tionGPT3 attains strong performance across protocols.
Protocel
Methods
R@i1
R@21
Text-motion R@31
R@51
R@:00
McdR_
R 1-
21
Motion-ter retrieval
R@3
R@:11
R@ 101
McdR.
All
TEMOS
2.12
4,09
5.87
8.26
13.52
173.00
3.86
4.54
6.94
9.38
14,00
183.25
T2M
TMR
5.68
10.59
14.04
20.34
30.94
28.00
9.95
12.44
17.95
23.56
32.69
28.50
MotionGPT
7.16
12.50
15.85
21.53
30.20
38.00
11.31
13.9]
19.39
24.13
31,80
36.25
MotionGPT3
9.60
17.36
22.45
30.43
41,06
17.00
14.90
18.20
24.43
31.32
40.72
17.50
All with lhreshold
TEMOS
5.21
8.23
11.14
15.09
22.12
79.00
5,48
6.19
9.00
12.01
17.10
129.00
12M
5.30
7.83
10:75
14.39
22.51
$4.00
4.95
5.68
5.93
11.64
16.94
69.50
TMR
11.60
15.39
20.50
27.72
38.52
19.00
13.20
15.73
22.03
27.65
37.63
21.50
MotionGPT
21.01
33.39
27.69
30.50
MrXionGP13
20.73
27.03
34.03
42.66
52.97
9.00
19.34
22.40
29.40
36.91
46.30
13.00
Dissimilar subset
TEMOS
33.00
42.00
49.00
57.00
66.00
4,00
35.00
44.00
50.00
56.00
70.00
3.50
T2M
TMR
47.00
61.00
71.00
80:00
86.00
2.00
45.00
63.00
69.00
80.00
$4.00
2.00
MreionGiP
51.00
64.00
71. 01.00)
74.00
X0.00
1.00
53.00
62.00)
68.00
76.00)
$1.00
1.001
Mot
68.00
77.00
85.00
92.00
95.00
63.00
73.00
89.00
93.00
1.00
Small butches
TEMOS
10.49
53.52
61.14
70.96
84.15
2.33
39.96
53.49
61.79
72.40
85.89
2.35
T2M
52.48
71.05
80.65
89.66
96.58
1.39
52.00
71.21
81.11
89.87
96.78
1.38
TMR
81.32
86.81
91.43
81.20
1.03
LaMP
67.18
81,90
87.04
92.00
95.73
68.02
82.10
87.50
92.20
96,90
Mo
58.07
69.91
74.34
79.17
86.96
1.18
58.71
69 64
74.36
79:45
86.02
1.16
74.25
86,70
91.29
94.82
97.35
1.00
74,00
86.86
91.04
94.62
97.35
1.00
20

---

We explore several CMA schedules that differ only in the layers where -ross- connections enabled(Tab. DT Across paired settings with the same spacing pattern, shifting the CMA blocks to later layers typically improves generation quality and distribution similarity to the ground-trute (i.e.. lower FID and MMDist, higher R- A-Precision). This is most evident in B1 v.s. B2: both use uniformly spaced CMA with identical count, B1 enable CMA from the first to the second-last layer, while B2 shifts them by one layer to span the second through the last layer. Despite the minor offset, Published as a conference paper at ICLR 2026
C.3 MOTION BRANCH WITH CROSS- MODAL CONNECTION
Our hybrid model allows asymmetric capacities for the text and motion branches and supports different patterns of cross-modal information exchange. In this section we focus on where place cross-modal attention (CMA) in the backbone for the text-to-motion task, keeping all other factors fixed (Tab. 9 Ablation on branch capacity deferred Sec. C.4
B2 achieves noticeably better scores. The same tendency appears in B3- C1-C3, and D1-D2. where the distribution pattern is matched but the CMA positions differ.
Table 8: Cross-modal attention configurations used in ablation. (a) Layer-wise CMA schedules for configurations A-D across the 12-layer backbones. Within each branch, the symbol = marks cross modal attention (CMA) operation at that layer, blanks empty indicate intra-modal attention only, (b) Schematics diagrams for configurations A, Bi. Ci.
B1
Model
2
3
7
10
II
CMA
A
B
Bs
Ci
C2
Di
D2
output hiddeess
(a) Layer-wise CMA settings of cach experiment in Tab.
(b) Connection visualization of A, B1 and Ci.
Table 9: Quantitative results for several CMA settings on T2M with 200K training iterations, settings visualized in Tab. The text branch is pretrained GPT-2 (1 (124M) and the motion branch has 114M parameters. Increasing the number of CMA layers and placing them later in the network generally improves performance, and A is the best among tested settings. See Sec. C.3 for further analysis.
R@1 1 f
R@21
R@31
FID,
MMDist
DIV
MModality
Real
0.518
0.713
0.813
2.811
9.976
A
0.536
0.728
0.819
0.241
2.767
10.379
2.454
B
0.502
0.707
0.807
0.311
2.895
10.318
2.489
B2
0.508
0.714
0.812
0.288
2.8637
10.261
2.315
Ba 0.508
0.712
0.811
0.22
2.879
10.405
2.664
BA 0.514
0.716
0.814
0.243
2.839
10.347
2.534
C1 0.506
0.701
0.801
0.236
2.894
10.386
2,684
C2
0.502
0.702
0.795
0.285
2.948
10.333
2.631
C3 0.503
0.705
0.803
0.171
2.886
10.221
2.819
DI 0.473
0.663
0.767
0.283
3.105
10.176
3.770
D2 0.477
0.672
0.777
0.164
3.092
10.189
3.197
C.4 ABLATION ON MODEL. SIZF
We examine model size along three axes while keeping all other settings fixed: (i) the overall capacity, achieved by scaling the text and motion branches proportionally, (ii) motion-branch capacity with fixed text branch, and (iii) hanguage-backbon size with comparable motion branch.
Tab. 10 compares overall backbone sizes. With roughly 3x parameters, the medium model yields modest gains on @@k and MMDist despite slightly higher FID. This suggests greater capacity helps capture high-level semantics, though realizing its full benefit may require careful optimization.
21

---

Published as a conference paper at ICLR 2026
R@1
R@3
MMDist
85
84
54
2.8
Text 774M
2.7
2.6
20
40
60
80
100
120
20
40
60 Motion params
100
120
20
40
60
90
100
120
Motion params (M)
Motion params (M)
so
120 params (M)
Figure 9: Ablation on branch capacity on motion generation. All models are trained for 200K itera- tions. A 124M text branch already performs competitively with much larger backbones (355M/774M),
Our model can achieve competitive performance with only halfed motion parameters (~51MM..
With the text branch fixed GPT-2 small (124M), we scale the motion branch from 23M 14M parameters by setting hidden size to 76, 192, 384, 576. 768 (Fig. 9). Increasing motion capacity generally improves motion alignment (higher @@k) and reduces MMDist, while diversity remaining stable. in a high level A half-sized motion branch (~51M) already offers a strong trade-off, delivering competitive overall performance, and, the best FID among the 124M-text configurations.
Text-branch Size. To isolate the effect of the language backbone, we replace GPT-2 small (124M) with GPT-2 medium (355M) and GPT-2 large (774M), keeping the motion branch of comparable size (pairs: 355M/59M vs. 124M/51M, and 774M/1 16M vs. 124M/1 14M). Larger text branches further improve alignment and lower MMDist, and tend to increase diversity/MModalityy
Table 10: Effect of GPT-2 backbone size and CFG guidance scale w on text-to-motion task. All models are trained for 200K iterations. The medium backbone, with more parameters (692M v.s. 238M) consistently outperforms the small model.
Size
Text Params
Total Params
3
R@1 f
R@21 1
R@3
FID 1
MMDist
DIV
small
124M
238M
1.0
0.534
0.739
0.842
0.222
2.61
10.256
medium
355M
692M
1.0
0.558
0.756
0.852
0.235
2.553
10.238
small
124M
238M
3.0
0.552
0.759
0.852
0.173
2.554
10.239
medium
355M
692M
3.0
0.568
0.766
0.860
0.192
2.489
10.084
C.5 VAE AND DIFFUSION HEAD
Ablation on Diffusion Head, We ablate the diffusion head H in our motion branch to study how conditioning design affects generation. We vary (i) supervision with a diffusion head H v.s. direct MSE regression, (ii) the mapping from backbone hidden states to the diffusion condition (multi- attention, MHA vs. linear layer), (iii) the number of motion holders h num E (1, 4, 8} used to query hidden states from the autoregressive backhone, and (iv) classifier-free guidance (CFG) at sampling. All variants are trained for 200K iterations with results in Tab. 11
Table 11: Ablation of motion generation head and loss on HumanMI.3D. All variants are trained on T2M task for 200k iterations, with same backbone and data. We vary: supervision with diffusion head H (Diff.) or direct MSE regression (MSE), (ii) mapping of multi-head attention (MHA) or Linear layer in H. (iii) number of motion holder Emotion_oute (h (k_nm)) (i.e., the count of hidden states passed from the backbone to Ht), and (iv) classifier-free free guidance (CFG) at sampling.
k_mu
Head
CPG
RØ1
R@2
RØ3 1
FID 1
MMDis
DIV
MMedality
(4)
Diff.
MHA
4
0.547
0.751
0.850
0.149
2,578
10.041
2.265
(b)
Dilf.
MHA
8
X
0,531
0,735
0.836
0.185
2.655
10.154
2.198
(c)
MHA
4
X
0.529
0.730
0.166
2.350
Diff.
X
0.525
0.164
2.514
Diff.
Lineur
4
X
0.521
0.731
0.829
0.178
2.713
9.985
2.603
Diff.
Linear
1
X
0.525
0.729
0.829
0.283
2,689
10.069
2,719
(2)
MSE
Lincar
4
0.518
0.725
0.823
0.276
2.705
9.758
2.175
Following MotionGPT's evaluation protocol (Jiang et al. 2023),
results are averaged over two runs. Best results are bold, second- are underlined. The default configuration is ghay-shaded.
22

---

Published as a conference paper at ICLR 2026
We observe that (i) Direct MSE supervision (the last row) yields the weakest performance, confirming the benefit of diffusion- based training. (ii) Increasing num (b-d) enriches the conditioning signal and improves retrieval accuracy R@ k. but also raises FID, suggesting a harder denoising problem. moderate setting of /_nwmm = 4 offers the best trade-off. (iii) With h_num = and no CFG (c)(e) MHA head outperforms linear mapper, achieving higher @@3 (0.529 v.s. 0.521), lower MMDist 2.6455 v.s.2.713), and lower FID (0.166 v.s. 0.178) The advantage is even larger under sparse conditioning (h_rwm
in (d)c), FID: 0.164 vs. 0.283); (iv) Enabling CFG on the same configuration further improves both alignment and fidelity, Accordingly, we adopt diffusion supervision with an MHA head and h h_numm as the default.
Guidance scale. We sweep the CFG guidance scale w on the unified model (results in Tab. 7 Note that guidance applied only to motion generation. Moderate guidance performs best: w 5.0 minimizes FID on the T2M, while = 3.0 yeilds the best R-Precision and MultiModal Distance. Very weak (w 1) or overly strong (w 2 10) guidanee degrades alignment and diversity. We thus use 4 in all main results.
Table 12: Ablation guidance scale in CFG, for motion latent diffusion HumanML3D, with model trained on unified tasks. Best and second-best results are bold and underlined.
R@1
R@2
R@3
FIDJ
MMDist
DIV
MModality
Real
0.519
0.724
0.820
0.002
2.753
9.941
0.534
0.727
0,828
0.143
2.714
10.086
1.717
0.555
0.754
0.843
0.123
2.601
10.006
1.321
0.554
0.756
0.850
0.103
2.385
9,926
1.272
4
0,552
0.753
0.848
0.098
2.589
9.911
1.258
5
0.552
0.752
0.848
0.094
2.593
9.906
1.248
6 0.546
0.751
0.849
0.094
2.398
9.900
1.243
0.546
0.748
0.844
0.109
2.620
9.870
1.281
0.541
0.739
0.839
2.653
9.873
0.533
0.728
0.826
0.134
2.739
9.827
1.385
Following MotionGPT's protocol, results are averaged over two runs. The default configuration is gray-shaded.
C.6 EFFECTIVENESS OF TRAINING SCHEME
We adopt three-stage schedule (see Sec. 3.4 Fig. 2): SI, text-to-motion (T2M) pretraining; SII. cross-modal alignment with joint optimization on T2M and motion-to-text (M2T) (SII); and SIII. joint fine-tuning. We evaluate three settings: (i) Three Stages, the full three-stage schedule, (ii) Two Stage, a two-stage schedule without SI, and (iii) Trained Text Branch, a two-stage variant which the text branch is unfrozen during SI-SII, rendering SIII unnecessary. We report results on both generation (T2M) and understanding (M2T) in Tab.
Table 13: Training-scheer scheme evaluation on HumanML3D (Guo et 2022a), with protocol in Jiang et al. (2023). St T2M Pre-training, Stage2: Cross-Modal Alignment, Stage3: Joint Fintuning, / marks enabled stages, while colors encode the state of text branch, updated frozen. Jointly updating the text branch from the start improves early T2M in SI but degrades final T2M after SII and markedly lowers M2T scores (rows "Trained Text Branch").
Text-to-Motion
Motion-to- Text
Type
Stagel
Stage2
Stage3
R@3
FID 4
MMDist 4
R@11
Bleu@:4f
BertScore
X
X
0.826
0.239
2.797
Three Stages
0.831
0.215
2.755
0.571
18.328
33,993
0.837
0.208
2.725
0.573
19.412
35,231
x
X
0.298
0.561
18.295
34.676
Two Stages
x
0.772
0.325
3.108
0.573
18.277
35.546
X
0.822
0.239
2.832
Trained Text Branch
0. 0.801
0.243
0.505
14.119
33.385
Following MotionGPI's protocel, metrics are averaged over two runs. Best and second best results are hold and undertined.
23

---

Published as a conference paper at ICLR 2026
Text-to-motion Pre-training and Cross-Modal Alignment. Pretraining T2M (SI) yields strong motion generation and provides a motion cootion-specialized initialization. Entering SII confers M2T capability and further improves T2M (alignment improves and MMDist/FID drop), indicating that explicit cross-modal alignment benefits both directions. Training directly with multi-task objectives from scratch (i.e., without SI) markedly degrades T2M quality, even after subsequent joint optimization, underscoring the importance of motion-specific warm start, i.c., initializing the motion branch by T2M pretraining.
Joint Fine-tuning. Once SII has established cross-modal alignment, SIII yields modest gains, primarily stabilizing M2T while preserving T2M, and thus serves as light refinement. The "Two Stages" variants (SII+SIII without SI) show that joint optimization can boost both tasks when the model under-initialized. However, also degrades the language branch's s competence, leading to worse T2M and M2T than the full three-stage schedule. Although incremental gains beyond SI+SII are modest, SIII can mitigate residual negative transfer and calibrate cross-modal alignment under noisy shifted training conditions, yielding more stable results.
Freezing v.s. training the text branch. To promote modality representations and reduce negative transfer onto well-trained language branch, we propose to freeze the text branch in SI-SII. Freezing preserves linguistic competence while the motion branch specializes. By contrast, updating all parameters from the start ("Trained Text Branch") can give slightly higher T2M scores in SI (e.g., @@3 3 0.834 vs. 0.820: MMDist 2.698 vs. 2.787). but after SII these models exhibit degraded T2M and notably weaker M2T (e.g., BertScore 0.713; Bleu 4 3.577), consistent with 'catastrophic forgetting'. We attribute this decline to negative transfer from the new motion branch onto the text branch during early training. In practice, keeping the LM frozen lets the motion branch learn more stable, modality-specific space and achieve more reliable alignment under /imited paired data.
In summary, SI provides essential motion specific initialization; SII delivers the bulk of cross- modal gains; SIlI offers small, stabilizing improvements. Freezing the text branch through SI-SII prevents loss of linguistic ability and yields the best overall balance between understanding and generation.
C.7 CHOICE OF BACKBONE
We instantiate our framework with three language backbones of GPT-2 Radford et al. (2019). Flan- T5-base Chung et al. (2022), and 1anm2.5-0.5--Intrrree Qwen et al. 2025), ) under both single- and dual-stream architectures. In the dual-stream setting, we set the hidden dimension of the motion branch to 374 for T5 and 448 for Qwen2.5, as roughly balance the number of newly introduced parameters. preliminary experiments, using a motion branch with the same size as the text branch in Qwen2.5 led to unstable training and degraded performance, presumably because a large number of parameters trained from scratch would require a much larger benchmark. Exploring larger-capacin motion branches with stronger pretraining or larger-scale datasets is left for future work.
As shown in Tab. 14 across all three language backbones, introducing our dual-stream architecture consistently improves retrieval metrics and MMDist the single-stream variants, indicating that our framework is not tied to a particular language model and scales well to more recent LLMs.
Table 14: Text-to-motion generation results with GPT-2, Flan-T5, and Qwen2.5 backbones under single- and dual-stream architectures. All models are trained on NVIDIA RTX 3090 GPUs, for 400 epochs in single-stream setting and 200 epochs in dual-streamm setting. For each configuration, report metrics from the checkpoint with the lowest validation FID.
Architecture
Backbone
Parameters
R@I
R@2
R@3
FID|
MMDist
Diversity
Real
0.511
0.703
0.797
0.002
2.974
9.503
GPT2
152M
0.513
0.708
0.808
0.485
2.962
9,633
single
T5
275M
0.491
0.681
0.777
0.243
3.102
9.365
Qwen
524M
0.483
0.669
0.768
0.491
3.143
9.653
GPT2
238M
0.533
0.731
0.826
0.239
2.797
9,688
dual
T5
369M
0.509
0.702
0.799
0.285
2.978
9.772
Qwen
617M
0.542
0.739
0.832
0.309
2.782
9.878
24

---

Published as a conference paper at ICLR 2026
Motion Latent
Global query
Projector
holeer
Linear
Diffusion bead 20
holder
AdALN
boloer
Probability distribttion
Embed
QKV Projector
FFN
LM Head
Softman
Nest Yoken Prediction
666
Text Tokens
reamanat
Motion Representation (SEC. 3.1)
Dual-Straan Motion-lagguaee Model (SEC. 3.2)
Dilfission for Aulonggraasvv Model (SEC. 3.3)
Figure 10: Details our bimodal motion-language model. Motion and text inputs are encoded by separate branches and then reordered to their original sequence order before cross-modal self- attention. After N hybrid layers, text is generated autoregressively by next-token prediction, while motion produced via a diffusion head H. Panels correspond motion representation (Sec. 3.I..
the dual- stre-stream motion-languue language backbone (Sec. 3.2p, and the diffusion module (Sec. 3.3).
D DETAILS ON MOTIONGPT3
D.I MOTION GENERATION IN UNIFIED MODEL
Diffusion Loss We use diffusion head H. to map backbone hidden states into a denoised motion latent. As illustrated in Figs. 2 [nd we insert motion holder tokens <motion out> as queries to extract the corresponding output states from the motion branch. Let the backbone hidden size be de and the motion latent size dm (we set 768, 256 in our default model). After one forward An MHA pooling module aggregates h and produces pass, the queried states form h E IR 1xdm via an internal mapping to match the motion-latent dimensionality, global condition vector e IR
which is then fused with the timestep embedding 7t)) to condition With the cumulative product of ..,,.., T) and corrupt the ground-truth motion
the noise schedule ã₂ = 0is. sample latent 2o dR
4= va + v1 ā,e, E e~ 10,,
(2)
Given a noisy latent 2 IR 1xdm at timestep t, the head predicts the noise &g and is trained with the standard c-prediction objective
Laifs Ezg.r,lle- éa(ar= + 4, é H(z,,
where ae follow a linear schedule equivalent to the forward noising process. At inference, we start from 27~~ N0(, I) iteratively denoise with the sampler of H down t-0 obtain 2o, which is then decoded by the motion decoder D into a raw motion sequence.
Architecture of H. The diffusion head first processes the K queried hidden states with Trans- formerEncoderrLayer and aggregates them via multi-head attention pooling into a single condition vector c. We fuse e with the timestep embedding and modulate each block AdaLN. HH consists of stack of 1024-wide residual blocks. Each block applies AdaLN followed by two-layer MLP with SiLU nonlinearity, and the final block projects to the noise prediction é
Cross-Entropy Loss for Boundary Tokens To delimit motion from text decoding, we introduce two boundary tokens<som>> (start motion) and <eom> (end of motion). At inference, once the LM predicts <som> via next-token prediction, generate the motion latent in single forward pass, with K <motion out> holders concatenated to the sequence, and then append <eom> deterministically. During training, apply cross-entropy only to the <som> prediction, and <com> is not supervised.
25

---

Published as a conference paper at ICLR 2026
D.2 MOTION VAE
AutoEncoder We adopt Transformer-based motion VAE (Xin et al. 2023) with encoder & and decoder D that maps an M-frrmee motion sequence m 1:M to compact continuous latent & E Rnxd
(n = L, 256) and reconstructs the motion via m 1:M D(z)
1:M) Both & and D transformers (Vaswani et al. (017) with long skip connection [Ronneberger et al. 2013), and without the action biases used in Petrovich al. (2021). This design yields expressive latent space that supports accurate semantic understanding and high-fidelity, diverse motion synthesis.
Architecture Given input motion sequence mi:id of length M. the encoder & processes the sequence together with a small set of learnable distribution tokens and outputs the Gaussian parameters (Pmm 7m). A latent z is sampled by reparameterization
1'mm + mm einN(O, /). The decoder Dmddd performs cross-attention over the latent vector to query L motion tokens, which are then projected back into 1: in the raw motion space.
m I:L (m
Loss We train the motion VAE with reconstruction term over framewise poses and KL regularizer on the latent, following standard practice in VAEs (Kullback & Leibler 1951 Kingma & Welling 2013). Let 1:Md denote sequence of M ground-truti truth poses m, E R226 and m 1aal the decoder outputs. The encoder produces Gaussian posterior (0 | = NN,, diag(o")). The objective is
C = Lree + AKL. LKI.,
(5)
with
I Σ
Lre
my
me 1
(6)
T
LKL DKL (N(, (o))) |N 1))
For completeness, the KI. term admits the closed form LKL. = E, (n3 03 -og o3 1).
Raw Motion Representation Following Guo et al. (2022a). each frame m R concate- nates a tuple of root angular velocity j" E R along Y-axis, root linear velocities (²) E R on XZ-plane, root height rd I, local joints positions j" R&N, velocities j" E R3N, and rotations j" E R6N, in root space, with N, denotes the joint number, and binary foot ground contact features of E R by thresholding the heel and toe joint velocities. This finally results in m² {+a 1², 82 rd j",""""""" c"
D.3 METRICS DEFINITIONS
We adopt standard metrics for text-motion alignment, motion quality and diversity, caption quality, and (for VAE analysis) motion reconstruction. Unless noted, features are computed with the official HumanML3D/T2M evaluator (Guo et al. 2022b), with motion encoder ((m) and text encoder (t).
Text-motion alignment To evaluate semantic consistency between generated motions and input texts, we adopt motion-text retrieval precision (R-Precision) at Top-k(R @ k), and the Multimodal Distance (MMDist), which measures the embedding measures retrieval accuracy within candidate set: for each query (text or motion), of the other modality by cosine similarity and report the fraction of cases where the paired item appears in the top-k. MMDDis is the average embedding distance between paired items:
distance between paired modalities. R@k
rank candidates
1 N
MMDist
S S (l((m)) - (1.1))l2
Motion quality FID assess how closely generated motions match ground truth ones in feature space, indicating overall quality, and is computed between the Gaussian fits of {Φ(m) for generated and ground-truth motions in the evaluator feature space.
26

---

Published as a conference paper at ICLR 2026
Diversity Diversity (DIV) measures feature variation across samples, and MultiModality (MM), which quantifies variation among motion generations from the same textual description. Following Guo et al. (2022b Xin et al. (203)) randomly sample all generated motions into two subsets, {2% 1==
and (x1) (=))
of the same size Xa- Then DIV is formalized as:
DIV =
Xx -
(9)
Xd
Randomly sample a of text descriptions with size Jm and sample two subsets of X,, motions generated by j-th for each text description, denote as {2g,) Xm and fay,sk=00 MM calculated as:
MM
E/ X |و
2',.1l
(10)
Jm x Xm
Motion captioning We follow prior work chuan2022tm2r and adopt standard NLP metries in- cluding BLEU (Papineni et al. 2002), ROUGE-L (Li 2004), CIDEr (Vedantam et al. 2015),, and BERTScore (Zhang et al. 2019) to evaluate the fluency, relevance, and diversity of generated captions.
Reconstruction We evaluate reconstruction fidelity of motion autoencoders with: MPJPE and PAMPJPE for global and local errors in millimeter, ACCL (Acceleration Error) computed from second-order finite differences, and APE/AVE (Absolute Position/Velocity Error) reported over root, trajectory, pose, and joints components.
THE USE OF LARGE LANGUAGE MODELS (LLMS)
We used ChatGPT only for grammar/typo checks; all technical content, experiments, and analyses were authored and verified by the authors without substantive LLM contribution.
27


Reviews:
[{"review_id": "WUXcBHHgoc", "reviewer": "Reviewer_Hcm9", "summary": "This paper propose a bimodal motion–language model for text-to-motion and motion-to-text generation. The key designs include: a continuos latent modtion space with a latent diffusion header to bridge its gap between the next-token predition framework, a dual branch framework with shared attention to bridge the gap of the two modalities, and a three stage training strategy for more stable optimization of the proposed framework. Experimental results show the effectiveness of the proposed framework on both the text-to-motion and motion-to-text tasks.", "questions": "None.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The motivation of utilizing the continuous motion latent space for lossless motion encoding and the diffusion header to bridge the gap between the next-token generation framework is reasonable. \n- The dual-branch framework to preserve modality-specific information and the shared attention for cross-modal communication is well motivated, and the three-stage training schemes stabilize the optimization of the proposed framework. \n- Experimental results on benchmarks of the two tasks are strong, and the effect of different design choices is validated with ablation studies. \n- The writing is good and the paper is easy to understand.", "weaknesses": "- The paper claims continuous VAE for motion encoding is better, but lacks an experimental comparison on motion encoding and decoding quality with previous schemes. Specifically, how is the improvement of the continuous VAE compared to the recently stronger motion quantization methods, e.g., the residual VQ proposed by MoMask (CVPR 2024) and the 2D motion quantization in MoGenTS (NeurIPS 2024)?\n- Experiments are only conducted on the HumanML3D datasets. Adding more diverse datasets, e.g., Motion-X and  KIT-ML, will better illustrate the generalizability of the proposed framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bpY8yljEke", "reviewer": "Reviewer_VYUy", "summary": "This paper propose MotionGPT3, a bimodal motion-language framework to unify the motion understanding and generation. It introduces two different modality branches to endow the model with motion-to-text and text-to-motion abilities.", "questions": "1. It is hoped that the authors can elaborate on the training details of the three stages, including the inputs and outputs. In particular, they should clarify why such inputs are used in the second stage and what advantages they bring. Additionally, what is the main purpose of keeping the text branch frozen in the second stage? Could the authors provide an input example for Stage 2?\n\n2. What exactly are the advantages of MotionGPT3 compared to MotionGPT and MotionGPT2, and which problems that the latter two failed to solve have been addressed? Relevant experiments would be highly appreciated.\n\n3. The authors are expected to clarify the points mentioned in the \"Weakness\" section.\n\nIf the authors can address the questions I raised, I may consider increasing the score.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper contains numerous figures and tables, as well as abundant visualization results, with a relatively clear overall structure.\n\n2. Video demos are provided, demonstrating excellent performance.\n\n3. Motion generation and motion understanding tasks are realized through two different branches and fine-tuning of the LLM.\n\n4. The experimental results have achieved significant improvements.", "weaknesses": "1. The description of Figure 2 and the method section is not clear enough, making it difficult to intuitively grasp the authors' entire training process design and the detailed reasoning procedure.\n\n2. Although the paper achieves good results, the method feels relatively incremental and highly hierarchical, lacking overall simplicity. Compared with MotionGPT and MotionGPT2, it does not bring a strong sense of novelty.\n\n3. The autoregressive continuous token proposed by the authors has been used in many motion generation papers, which makes the contribution seem insufficient.\n\n4. There are some typos. For example, is \"L40\" supposed to be \"LLM\"? The content from Line 82 to Line 89 is better presented in bullet points. Rarely seen in introductions is the writing style from Line 57 to Line 80, which fails to convey the necessity and practicality of the method proposed by the authors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jBvx57BGLQ", "reviewer": "Reviewer_cYVn", "summary": "This paper presents MotionGPT3, a unified model that addresses motion quantization errors and cross-modal interference in motion-language tasks. Its key innovations include: 1) a VAE-based continuous motion latent space, 2) a dual-stream Transformer for controlled cross-modal interaction, and 3) a diffusion head to connect discrete text and continuous motion. Trained in three stages, MotionGPT3 sets new state-of-the-art performance on HumanML3D for both text-to-motion and motion-to-text generation, while converging 2-4 times faster than baselines.", "questions": "Please see the weakness part.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The technical design in this paper is highly targeted, with each core component—a continuous VAE, a dual-stream architecture, and three-stage training—precisely solving a specific problem, and its necessity is rigorously validated through ablation studies, resulting in a lean and non-redundant overall architecture.\n\n2. The experimental validation is comprehensive, encompassing quantitative comparisons, qualitative examples, and thorough ablation studies.\n\n3. The study ensures high reproducibility by detailing implementation specifics—including data preprocessing, training protocols, and evaluation tools—and adheres to open-source standards with released code and materials.", "weaknesses": "1. The bimodal branch architecture proposed in this paper to address cross-modal interference in motion-language modeling is not a particularly novel approach, as similar frameworks have been proposed in existing unified text-image understanding and generation work, such as BAGEL [1]. However, the paper lacks discussion on how the proposed method differs from these existing approaches.\n\n2. Baseline comparisons are outdated, lacking recent models (e.g., MotionGPT-2 2024 [2]，MG-MotionLLM 2025 [3]), which may conceal performance gaps in key metrics like M2T BERTScore.\n\n3. The paper provides analysis on training convergence speed but lacks evaluation of inference latency, which remains a critical metric for motion generation applications.\n\n[1] Emerging Properties in Unified Multimodal Pretraining, 2025, arxiv, 27 July\n[2] MotionGPT-2: A General-Purpose Motion-Language Model for Motion Generation and Understanding, 2024 arxiv, 29 Oct\n[3] MG-MotionLLM: A Unified Framework for Motion Comprehension and Generation across Multiple Granularities, 2025 arxiv, 3 April", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JjMgDGE3gu", "reviewer": "Reviewer_9qBu", "summary": "This paper presents MotionGPT3, a motion–language model designed to jointly handle motion understanding and generation while addressing key limitations of existing multimodal frameworks. To avoid the motion quantization issue, it encodes raw motion into a continuous latent space with a lightweight diffusion head, eliminating quantization-induced artifacts and enabling higher-fidelity synthesis. The architecture employs a dual-stream Transformer with shared attention, which maintains modality-specific information while allowing cross-modal exchange. Besides, a three-stage generate-then-align training schedule is proposed to further enhance the  convergence  efficiency.", "questions": "* typos: line 32, citepacross", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well written, easy to follow, and clearly organized. The figures are self-explanatory.\n* The motivation is concrete and reasonable, and the method design is aligned well with the corresponding motivations, presenting sound performance improvements.\n* This paper offers a well-reasoned perspective on the gap between discrete language token sequences and continuous motion latent representations, which can provide valuable insights to the community.", "weaknesses": "* Lack of detailed comparisons of inference latency or FLOPs against discrete-token baselines. Authors should provide the computational overheads induced by each proposed component to demonstrate the method's efficiency.\n* The evaluation of the method is constrained to a single dataset (HumanML3D), which can not sufficiently demonstrate the generalization of the proposed framework to other motion domains. Providing more results on other benchmarks with diverse features would better illustrate the method's versatility. Besides, it is expected to offer more results with more recent language models other than GPT-2.\n* As shown in Table 1, why does the proposed method significantly lower on the MModality metric compared to other models? Does this indicate that the model can only fits a specific distribution, thereby sacrificing the diversity of generated outputs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)