Motion-Aligned Word Embeddings for Text-to-Motion Generation
Authors: Ke Han
Yueming Lyu
Nicu Sebe
Venue: ICLR 2026 Poster (2026)
Keywords: text-to-motion generation
large language model fine-tuning
word embeddings
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Existing text-to-motion (T2M) generation models typically rely on pretrained large language models to encode textual inputs. However, these models, trained on generic text corpora, lack explicit alignment between motion-related words (e.g., "clockwise'', "quickly'') and human skeletal movements. This misalignment, fundamentally rooted in the word embedding layers, severely limits the ability of T2M models to understand and generalize fine-grained motion semantics. To tackle this issue, we propose Motion-Aligned Text Encoding (MATE), a novel framework that explicitly incorporates motion semantics into the word embedding layers of large language models to enhance text-motion alignment for motion generation. To address the challenge of inherent semantic entanglement in motion sequences, MATE introduces two key components: 1) a motion localization strategy that establishes localized correspondences between sub-texts and motion segments, enabling soft attention guidance for semantic localization; and 2) a motion disentanglement module that isolates word-specific motion semantics via contrastive kinematic prototypes, ensuring word-level alignment between linguistic and kinematic representations. Remarkably, language models enhanced with MATE can be seamlessly integrated into existing T2M methods, significantly surpassing state-of-the-art performance on two standard benchmarks with minimal modifications.

Published as a conference paper at ICLR 2026
MOTION-ALIGNEI WORD EMBEDDINGS FOR GENERATION
Ke Han
Yueming Lyu Nanjing University vmnv edu. cn
Nicu Sebe University of Trento n niculae. sebe@unitn, it
University of Trento
ke. hanhaan... aca ama@gma
i 1. com
ABSTRACT
Existing text-to-motion (T2M) generation models typically rely pretrained large language models to encode textual inputs. However, these models, trained on generic text corpora, lack explicit alignment between motion-related words (e.g., "clockwise", "quickly") and human skeletal movements. This misalignment, funda- mentally rooted the word embedding layers, severely limits the ability of T2M models to understand and generalize fine-grained motion semantics. To tackle this issue, we propose Molion-Aligned Text Encoding (MATE), a novel framework that explicitly incorporates motion semantics into the word embedding layers of large language models to enhance text-motion alignment for motion generation. To address the challenge of inherent semantic entanglement in motion sequences, MATE introduces two key components: 1) a motion localization strategy that estab- lishes localized correspondences between sub-texts and motion segments, enabling soft attention guidance for semantic localization; and 2) motion disentanglem module that isolates word-specific motion semantics via contrastive kinematic prototypes, ensuring word-level alignment between linguistic and kinematic repre- sentations. Remarkably, language models enhanced with MATE can be seamlessly integrated into existing T2M methods, significantly surpassing state-of-the performance on two standard benchmarks with minimal modifications. Code is available at https://iihub..co//ee wtwp//e-aaaacc//MMTEg...... acT//MTT.gii...
INTRODUCTION
Text-to-motion (T2M) generation aims to synthesize sequences of human skeletal movements con- ditioned on textual descriptions Zhou al. (024a): Chi et (204); Liu et al. (2024a): Fan et al.) (2024); Wang et al. (2024b). As cross-modal generation task, T2M requires models to accurately translate textual descriptions into motion semantics and decode them into realistic human motions. However, cxisting T2M approaches still often exhibit limited cross-modal understanding. As shown in Fig. M while current models achieve the sentence-level alignment in example (I), they often struggle to robustly understand motion-related words such "clockwise", leading poor generalization and failure to generate plausible results for descriptions like "jogs in clockwise motion" in example (2).
This limited word-level understanding largely stems from the limitations of text encoders, which process the textual inputs and directly determine the semantic information conveyed to the mo- tion generator. While most T2M methods adopt pretrained large language models (LLMs) such as CLIP Radford et al. (202T) or DistiBBERT' [Sanh et al. (2019) to leverage their strong textual understanding capabilities, these models are trained on general text corpora (or text-image pairs in the case of CLIP), lacking fine-grained alignment between motion-related words and human skeletal movements. In particular, the word embedding layers LLMs fundamentally define the word semantics, which can differ substantially between linguistic and kinematic contexts. For instance, while "clockwise" functions linguistically as an adjective or adverb, in the kinematic domain denotes concrete rotational motion with a specific directional orientation. Without addressing such cross-modal word-level misalignment, L.MMs struggle to encode motion-aware information effectively, inherently limiting the generation quality and generalization ability of T2M approaches.
"Corresponding author

---

Published as a conference paper at ICLR 2026
(1) "a person walks in clockwise circle"
(2) person jogs in clockwise motion and falls their knees, he then gets back up onto his feet"
MoMask
MoMask
MoMask + MATE
Figure I: Examples generated by the state-of-ther MoMask model Guo al. 2024 with darker colors indicating motion progression. MoMask correctly produces "clockwise" motion in (1), but fails in (2), revealing limited robustness and generalization in capturing motion-related word-level semantics. Incorporating our Motion-Aligned Text Encoding (MATE) enables MoMask to produce the correct motion.
To address this limitation, we propose Motion-Alije Aligned Text Encoding (MATE), a novel framework that incorporates motion semanties into the word embedding layers of L.MMs to enhance text-motion alignment for motion generation. MATE optimizes only the word embedding layers while freezing the subsequent layers, which retain strong contextual modeling abilities acquired during large-scale language model pretraining. We hypothesize that these higher layers generalize effectively to motion semantics, as language and motion share structural properties, i.e., both consisting of compositional elements (e.g., words and actions) organized in temporal sequences.
However, incorporating word-specific motions semantics into word embeddings remains highly challenging and largely underexplored, primarily due the intrinsic entanglement of motion seman- tics. Existing datasets typically provide only sentence-level annotations for entire motion sequences Guo al. 2022a); Plappert al. (206)). lacking explicit alignment between specific words and corresponding motion segments. This limitation restricts the model's ability to temporally ground word-level semantics, particularly in sequences involving multiple compositional actions. More importantly, the semantics associated with related words are inherently intertwined, making it difficult to attribute distinct motion patterns to individual words, thereby limiting the model's capacity for fine-grained semantic understanding.
To address these challenges, MATE introduces two key components: 1) motion localization strategy that jointly decomposes paired textual descriptions and motion sequences into semantically aligned sub-units. This enables the construction of a soft attention prior that guides the temporal localization of word semantics; and 2) A motion disentanglement module that isolates word-specific motion semantics through two complementary mechanisms: self-disentanglement, which extracts shared semantics across related motions via contrastive kinematic prototypes; and pooss-dxentanglemennt which enforces the exclusion unrelated semantics, jointly ensuring semantic purity and inter-word discriminability. The disentangled motion semantics then aligned with their corresponding word embeddings, effectively addressing the word-level misalignment inherent in L.Mss
MATE offers resource-efficient solution for LLM fine-tuning by optimizing only word embedding layers, while maintaining broad compatibility with various LLMs. The MATE-enhanced LLMs can be seamlessly integrated into existing T2M methods with minimal architectural modifications. Extensive experiments demonstrate that MATE consistently improves text- text-motion alignment and generalization capability, significantly advancing the state of the art on standard benchmarks including Human Mahmood et al. (2019) and KIT Plappert et al. (2016). The main contributions of this work are summarized as follows:
1) To the best of our knowledge, MATE is the first framework to explicitly address the text-motion misalignment fundamentally rooted in the word embeddings of LLMSs for motion generation. 2) We introduce text-motion joint segmentation strategy that automatically establishes correspon- dences between sub-texts and motion segments, enabling action-level semantic localization for paired text-motion data. 3) We propose motion disentanglement module that achieves word-level semantic disentanglement, mitigating the challenge of semantic entanglement in motion sequences. 4) Extensive experiments demonstrate that MATE-enhanced language models can be seamlessly inte- grated into existing T2M pipelines, yielding substantial performance improvements and significantly surpassing state-of-the-ar results across two standard benchmarks.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORKS
Text-to-Motion Generation typically consists of two stages: text encoding and motion synthesis Shafir et al. (002)); Xie et al. 2024); : Liu et al. (2023): Liang et al. (202)). In this pipeline, textual descriptions are first transformed into text embeddings, which are then mapped to motion sequences. Most existing works keep the text encoder frozen and instead focus on improving the motion generation module, often through advanced architectures such as diffusion models [eevet al. (2023)
Zhang et al. (2023b); p in et al. (2024ap Ren et al. (2023); Wang et al. (2023) and quantized variational
autoencoders Chen et al. (2023 Zhang et al. (2023a); Van Den Oord et 2017); Dai al. (2024).
However, these methods typically rely on shelth embeddings from pretrained large language
models (LLMs), leaving unaddressed the intrinsic semantic gap between linguistic representations
and kinematic motion features that arises from the LLMs themselves.
To address text-motion misalignment, some works pretrain new language models using text-motion pairs to obtain motion- -ware text embeddings, such as LaMP Li et al. (2025), MotionGPPT Diang et al. (2023) and MotionGPT-2 Wang et al. (2024a). Our method differs from them in two key aspects. 1) These methods primarily operate at the sentence level, whereas our approach focuses on word-level alignment without requiring any word-level annotations, enabling modeling of finer-grained motion semantics. 2) They train new language encoders largely from scratch on relatively small text-motion datasets, which may limit their language modeling capacity. In contrast, our method fine-tunes only the lightweight word embedding layer of pretrained LLM while keeping all higher layers frozen, thereby preserving the rich linguistic knowledge learned from large-scale corpora.
Contrastive Learning has been widely applied for learning text-motion aligned representations in both retrieval and generation tasks. Methods such as MotionCLIIP Tevet et al. (2022), TMR Petrovich (023). TM2T Guo et al. (2022b), TEMOS Petrovich et al. (2022), and HumanTOMATON Lu al.
et al. or to regularize motion representations. LAVIMO Yin al. (2024) further incorporates videos as an additional modality and learns trimodal alignment through contrastive learning across text, motion, and video. More recently, LL---bsed models like M*GPT et al. (2024) also adopt contrastive objectives during multi-task pretraining to enhance motion-language understanding.
2024 construct sentence-level contrastive objectives to obtain motion- text embeddings
However, these methods primarily focus on sentence-level alignment. In contrast, our method introduces a prototype- based contrastive mechanism that explicitly operates at the word level, enabling finer-grained semantic disentanglement and alignment without requiring word-level annotations.
Large Language Model Fine- Tein--Tunigg has emerged as a powerful paradigm for adapting general- purpose textual representations to downstream tasks such as domain adaptation Ding et al. (2023);
Susnjak et al. (2025), image generation Li ct al. 2024);; Liu et al. (202b)). and video genera- tion Rasheed et al. (2023); Wu et al. (202)). In the context of T2M, recent works LMM Zhang et al. (024a)), AvatarGPT Zhou et al. (2024b), Motion-Agent Wu et al. 2025) unify multiple text-motion
tasks within a single framework, often incorporating expanded token vocabularies and instruction
tuning techniques Ouyang al. (022). Our method differs in two key ways: Instead of fine-tuning full LLMs, which is computationally expensive, we adapt only the lightweight word embedding layers, providing more essource-ffficeen approach. 2) Rather than introducing new standalone framework, we produce soug-and--lay notion-aligned text encoders that can be directly integrated into existing T2M models, improving performance without modifying their architectures.
3 METHOD
3.1 OVERVIEW
Word embedding layers, typically placed at the input of LLMs, function as a lexical lookup table that maps discrete word tokens to continuous vectors for contextual modeling. They play a crucial role in encoding word meanings and inter-word relationships. However, word-level semantics in linguistic domain often differ fundamentally from those in the kinematic domain. For example, "clockwise" and "anti- clnticclockwiee" are linguistically similar, due to sharing morphological structure and grammatical function. In contrast, they kinematically denote opposite directions of rotation and incompatible, thus leading to significant modslity discrepancies.
3

---

Published as a conference paper at ICLR 2026
solid black
trainable module
dashed block
frozen module
turns forward. back to storthng point.
<h forward clockwise
Text Encodor
jogs
jogo back storting point
man staggers forward,
Text-Mation Joint Segmentation
turns clockwiis, then jogs back storting point
W:: clockwwie
ym
Motion Decoder
(2) Motion Localization
word embeddiigs
motion features
prototypes
frieetuter
Ealign
fetschwise
(m
foloewise
Eatti
Salf-
TI
Disentanglemmet
K.V
Mation Encoder
MuB Hoad Adaiknn
Ferroves
p
fiz
fate
Cross-
Motion Localization
Altention Prior
Disantanglement
Motion Disentanglement
attract
rapel
(1) Motion Wligned Text Encoding (MATE) framework
(3) Motion Disentanglement
Figure (I) Overview of the MATE framework, which comprises text encoder (with trainable word embedding layers), motion encoder, and a motion decoder, together with: (2) Motion localization, which establishes temporal text-motion correspondences prior for guiding word-level semantic localization; and (3) Motion disentanglement, which employs multi-head attention disentangle motion semantics of specific words for semantic alignment.
construct ceuusian-shaper attention
To bridge this gap, unlike existing T2M methods that use pretrained LLMs without adaptation, this work introduces Nooiinn-Alignee Text Encoding (MATE), a novel approach for explicitly aligning the word embedding layers in LLMs with word-specific motion semantics.
given triplet (t, m, w) sampled from the training set, where denotes textual description, is the corresponding motion sequence, and w is a word token sampled from t. the goal is to align the textual semantics of w with its associated motion semantics expressed in by optimizing its word embeddings in language models.
Specifically, we formulate the learning objective as follows:
To achieve this alignment, as shown in Fig. 2 MATE comprises two key components: 1) Motion localization, which establishes correspondences between temporally aligned subtexts and motion seg- ments for word semantic localization; 2) Motion disentanglement, which disentangles motion features that are semantically attributable individual words for alignment with their word embeddings.
3.2 TEXT-GUIDED MOTION LOCALIZATION
motion sequence is complex integration of multiple word-level semantics, making fine-grained semantic aligmment particularly challenging, especially in long sentences involving multiple words and actions. To tackle this issue, we adopt coarse-to-fing semantic extraction strategy, with the first sub-goal of localizing the motion semantics corresponding to an individual word to within the motion sequence m. However, most existing motion datasets Guo et al. (2022a): Plappert al. (2016) provide only sentence-level annotations for entire sequences, lacking explicit supervision for word-level localization. To overcome this limitation, we propose text-motion joint segmentation pipeline that automatically establishes correspondences between each sub-action described in the text and its counterpart segment in the motion sequence, as illustrated Fig. (2).
Specifically, employ ChatGPT Roumeliotis & Tselikas 2023) to decompose each textual de- scription into a set of sub-texts t'.... ,t. (N 2 1). where each sub-text describes one or more temporally coherent actions. We then seek segment the motion sequence into N non-overlapping clips, each aligned with its corresponding sub-text. To this end, we formulate an optimal partitioning problem, where segment boundaries adjusted minimize the matching loss between each sub-text and its corresponding motion segment. Given a sentence decomposed into N sub-texts, the objective defined as:
N
min (n
Σ
1 cos (E,(t"), Em(mm e.,))),
(I)
n=1
4

---

Published as a conference paper at ICLR 2026
where Sn and en denote the start and end frames of the nith segment, constrained by 8x+1 en- Here, &g and Em are frozen text and motion encoders from a pretrained eext-to-motion retrieval model Lu et al. (2024), and cos(:,)) -) denotes the cosine similarity between encoded text and motion features. An exhaustive search over all valid partitions is performed to identify the boundaries that best align motion segments with their respective sub-texts.
The obtained segmentation is not directly used as ground-truth localization, but instead serves as soft prior to guide the discovery of the semantics of the word within the motion sequence m. Specifically, a motion encoder simultaneously extracts a sequence-level representation f E R and frame-level features F E R 1 from the motion sequence m, where T denotes the number of frames and D is the feature dimension. We introduce a multi-head attention mechanism, in which the word embeddings serve as a query to explicitly attend to relevant motion features, formulated as:
word MultiHHadd V),
Q = Proj(WE(w)), K = (1 + X (atentionPrior(t)) @ F",
Fm
Here, WE(w)) is the trainable word embeddings of w from the text encoder, Proj is linear projection layer, and AttentionPrior(tt) is temporal attention prior derived from segmentation information:
AttentionPrior(() = exp( (t- (t-c,)² with Gn 20
Sn Cn 2
En
Syz
On
2
where is the frame index, and 8,, cn are the start and end frames in the localized segment of w, respectively. The Gaussian-shaped AttentionPrior(() softly highlights frames near the center of the localized segment while smoothly attenuating distant frames, thereby improving robustness against localization errors introduced by the segmentation process.
3.3 WORD-GUIDED MOTION DISENTANGLEMENT
While the above approach enables localization of word-level semantics, the semantics of related words remain highly intertwined. For instance, as shown in Fig. (1) and (2), although the segment corresponding "turn clockwise" can be identified, it remains challenging for the model to accu- rately distinguish between the semantics of "turn" and "clockwise",, thereby hindering the precise understanding of individual words. To address this limitation, we propose word-guides motion disentanglement approach that explicitly isolates motion semantics attributable to individual word units. Toward this goal, we introduce the following three criteria for effective motion disentanglement.
1) Stability: A given word query should consistently attend to shared motion features across different motions that exhibit the corresponding semantics (e.g.. the features disentangled by the word "clockwise" from motions of "turn clockwise" and "jog in a clockwise circle" should remain similar.
2) Discriminability: Different word queries should result in semantically distinct motion features (e.g., the features disentangled by "turn" should be distinguishable from those by "clockwise").
Disentanglement should yield meaningful features only when the motion sequence contains semantics associated with the queried word (e.g.. querying "clockwise" from the motion of "walk forward" should not produce a meaningful feature representation).
3) Rationality:
To satisfy Criteria and 2, we introduce self-disentanglemmet mechanism based on prototype representations. Specifically, we predefine set of motion-word prototypes consisting of K learnable vectors (ftr jk= each representing the motion semantics associated with a specific word urk- Suppose {t,, m,, w; } the i-th triplet sample in mini-batch, the disentangled motion features fm satisfy the self-disentanglement loss Leeur
exp(cos( ft! f2,)/T) Ci exp(cos ( (/T)
Liell =
Σ KEV
log
(5)
IV
where is the set of samples in a mini-batch. The loss Leef encourages f to be pulled closer to its corresponding prototype f2, while being pushed away from all other prototypes. Simultaneously. each prototype is optimized towards stable and shared semantic representation across motions that express the semantics of w, In contrast to conventional contrastive losses Radford et al. (202T); Oordd et al. (2018) that operate at the batch level, the introduction of prototypes enables contrastive
5

---

Published as a conference paper at ICLR 2026
learning over dataset-wing wide word semantics, thereby enhancing the stability and discriminability of motion disentanglement.
To satisfy Criterion 3, further formulate cross-disentaagllemen mechanism: the motion m. is also queried by the word token w from the j-th sample, and m, does not contain the semantics of u. the model is encouraged to produce motion features that are orthogonal to the reasonably disentangled features f The rros-diientangleemnt loss Lemes is defined as
Leroas
1
Σ (i.j)EN
[fs fW) (f: fwy ){s
where A is the set of negative pairs, and fwy denotes the disentangled motion features from the i-th sequence m, with the word query wy from the j-th sample. Leuss drives the model to discriminate whether the queried word semantics is expressed in the motion sequence to ensure the accuracy of motion disentanglement.
3.4 MOTION-ALIGNED WORD EMBEDDING
To align the disentangled motion features with corresponding word embeddings, we formulate an alignment loss Calign as
exp(cos( fm, fm:://T) exp(cos(ff, fw; )/T)
cxp(cos( f em; fR.)/T exp(cos( f.. ( )/T)
Calign
log
log
(6)
12V|
Σ jeV
EVV
JEV
where fú, = Prooorrreere Proj(WE(w:)) is the projected word embeddings, consistent with the query in Eq
This loss adopts a symmetric InfoNCE formulation Dord (2018) to encourage alignment between paired word embeddings and their motion semantics while simultancously promoting separation between mismatched pairs. The motion-aligned word embedding loss is summarized as
w = Leett + Leross + Latign-
(7)
However, the above approach primarily focuses on individual word-level alignment and overlooks the contextual dependencies among words. To tackle this issue, further introduce sentence-level alignment objective, which aligns the text feature vector f extracted from with the corresponding motion feature vector fm from m using InfoNCE loss Lsent- Additionally, fr is passed through a motion decoder to reconstruct the original motion sequence m. guided by reconstruction loss free preserve detailed motion information. The overall training objective is:
Cal Lree + d Lword A2 Leents
(8)
where A and λ weighting factors.
EXPERIMENTS
4.1 EXPERIMENT SETTINGS
Dataset. We conduct experiments on two standard motion datasets: HumanMI.3D Mahmood et al. (201)) and KIT Plappert et al. (2016). HumanML33 contains 14,616 motion sequences, annotated with 44,970 textual descriptions, while KIT includes 3,911 sequences paired with 6,278 descriptions.
Evaluation Protocols. Following standard protocols Guo et al. 2022a). we use 5 evaluation metrics: R-Precision and Multimodal Distance (MMDist) measure how accurately generated motions match the texts. Frechet Inception Distance (FID) evaluates the distributional similarity between generated and real motion features. Diversity computes the average Euclidean distance across 300 randomly sampled pairs of generated motions. MultiModality (MModality) reflects the variation of generated motions, calculated as the average distance among 10 motions generated from the same text.
Implementation Details. The text encoder in the MATE framework could be various large language models, such as CLIP Radford et al.) (202T), DistilBERT [Sanh et al.] (2019), etc. The number of word prototypes (K) is automatically decided by the number of words included in the training set.

---

Published as a conference paper at ICLR 2026
Table I: Results on HumanML3D. "1", "I" and "-" indicate that higher values, lower values, or
values closer to real motion are better, respectively. Red and blue highlight the top two results.
Methods
Top-1
R- 3-Precision
Top-2
Top-3
FID 1
MM-Dist 4
Diversity
MModality
Real metions
4.511
0.703
0.797400005
0.002
2.974440006
9.503
GraphMotion lin 2024g
0.504
0 0.609-00.02
0.785 10.00
0.316 201
3.170 10.00s
9.89246.07
2.700 .o.0
Motion Mamhil Zhang al.
2224c
0.50240.005
0.693+0.02
0.792+0.002
0.281
3.000
9.871
2.204
ParCo
0.51540.ms
0.706.40.03
0.801 100.003
0.109 200
2.927#0.000
1.382 A0.000
CoMo Huang ct A024
0.50240.002
0.692400.07
0. 0.790+0.002
0.262
3.002-M0215
1013 0.906+
BAMM /nnyoaunssspnngg
2024a
0.52240.0m
0.715#00.00
0.808
0.055 2002
2.938 LIATTT
2.63666.00
1.732 10.005
MDM
Tevet et al. 2023
0.32)
0.498 0.498
0.611 1000007
0.544
5.508
9. 9.559 18.0
2. 2.7.02
+MATE
0.50gte..zz
0.608+0.002
0.797700.033
0.332
3.057
9. 16g
2.77320.002
Motion DorrienDifuue///anggo /hang et al.
2022
0.491
0.691
0.78240.001
0.630
3.1 3.11350.col
9 9.410 040
1.553
+MATE. (ours)
0.336 on
0.721 001
0.821 22:00
1,23350 awe
2.907=00.00
9. 9.666
1.703-0 0G5
MMM Plnyoomuntapong of ml.
2124b
0.515+60002
0. 0.70849.002 002
0.80440.002
0.00g
2.5226 10.000
9. 9.57756.00 0ri
1.226
MATE
0.511 te 001
0. 0.72g oos
0.820:40:00
0.06g30 aoa
2.88750012
9 9.5624
1.469 007
MoMask Giao ct 2024
0.521 16.002
0. 0.713
0.807 0.807-40.002 10.002
0.045 10
2,508 10.c0s
9 9.532 Lo.02
1:241 1n 10.000
MATE (ours)
0.550
0.802
0.040
2 290000
9.516
Table 2: Results on KIT, using the same notations as in Table
!
R-Percision Top-2
Methods
Top-1
Top-3
FID 1
MM-Dist 1
Diversity +
MModality 1
Real motions
0.6g9 40420
0.779400005
0.081#90004 204
2.78850812
11.08
GraphMMtion Jin et al.
2024b
80.2940.007
0.618a0.0s
1.789-15.00
0.51339.013
3.07630.c22
11.1246.15
3.627/0.313
Motion Marnil Alanng
m0
7024c
D.11g 00s
0. 0.6gg.20.00 o0s
0.765 10.com
0.307220.0 041
3.021 2.
11.0240.0s oss
1.67g oot
ParCo Zou al. 2024
0.619-Ao.w7 030
0.772
0.453
2.820
10.95*6.004
1.215#0.002
CoMe Hunng el al 2024
0.12240.009
0 0.638440.00 007
0.765 40.001
0.33220
nos
2.87350.cm
10.9540 196
1.2:9-20.00 oos
BAMM Prayvanuss.punng al. 2024a
0. 0.40610.007
0.6804...0s
0,791 LA0,005
0.200
2.71g4m006
10.9148.07
1.517
MDM Tevet al 2023
0.164
0.291 10.001
10.000
0.49720.0om
9.191 20,222
10.854..00 tow
20.314
MATE (ours]
0.407750.000
0.608
0.729+0.007
0.297
2.07g90024
10.93
1.985t0.204
DeticeDiffuse Zhang et
2022
0.621#0.000
0.739
1.95 202
2.958=0.00
11.10
0.730
+ MATE cours)
D. 0:132440000
0.61440000
D. 1.76400:000
0.965
2.85240000
11.1246:00
1,20140000
MMM Tnyyanuntaponng..m
2024h
0.404 Le.uss
0.621 10 tws
D. 10.000
0.316 10
2,777 10.11
10.91 18.10000
1.232 10.20..cov
MATE:(our)
0.42240000
0.64230
0.770
0.253
2.81530
10.38
1.39
MoMask Giao 2024
0.4gg 16.00r
0.636 10.00.0.000
0.781 10.00s
0.204 10.300000
2.77y 10.22
10.88 L0.0g
1.131 10.003
MATE: Cours)
0.44g3200006 oos
0. 6agganooo
0.79g40.007
0.19739.015
2.73230011
1.68g40.00 10.96
Specifically, we perform lemmatization on textual descriptions and summarize the vocabulary, estab- lishing K=5,16 and 1,191 prototypes for HumanML3D and KIT, respectively. These prototypes are initialized with their corresponding word embeddings in the pretrained language models.
After training MATE, we integrate the MATF-enhanced text encoder into existing T2M models in place of their original encoders, and retrain the models from scratch. The training and inference procedures strictly follow the official implementations of the T2M models, without any modifications. To ensure statistical reliability, we perform 20 rounds of inference and report the averaged results. Unless otherwise stated, we use MoMask Guo et al. (024) with the MATE-enhanced CLIP as the default T2M model for evaluation.
4.2 COMPARISON WITH STATE-OF METHODS
Quantitative Comparison. Table 0 and Table present the performance of state-of-the-art (SOTA) T2M models. The "+MATE" variants of MDM Tevet et al. (023)) MotionDiffuse Zhang et al. (2022). MMM Pinyoanuntapong et al. (2024b) and MoMask Guo et al. 2024h are obtained by retraining the original models with the pretrained CLIP text encoders replaced by MATE- enhanced CLIP, where only the word embedding layers are updated. Despite this minimal modification, our approach consistently yields substantial improvements across all evaluation metrics on both benchmarks. Notably, while the performance gains on HumanMI.3D are significant, the improvements on KIT relatively modest due to the smaller dataset size, which constrains the optimization of word embeddings in large language models.
Visualization Comparison. Fig. 3 compares motion sequences generated by different SOTA methods. MATE accurately distinguishes fine-graing -reigrddded semantics, such as "kick one time with the right leg" and "three times with the left leg" in the upper example, and faithfully captures key textual descriptions like "counterclockwis circle" and "yawn" in the lower example, demonstrating the superiority of our approach in fine-grained motion semantic understanding and text-motion alignment.
7

---

Published as a conference paper at ICLR 2026
(1) "a person kicks time with right and three times with left leg"
kick with right then left then right leg kick four times with left lag
(2) "a person walks a counterclockwise circle and raises their hand to their face to yawn."
circle
yawn
circle
both hands
Momask
BAMM
Figure 3: Comparison of motions produced by SOTA methods, with semantic misalignment in red.
Table Ablation study of loss functions on Hu-
Table 4: Evaluations optimizing different layers
ManML3D by removing losses in Eq. 7 (8)
of CLIP text encoder on HumanML3D.
Loss Removed
Top-1
FID
MM-Dist
Trainable Layers
Parameters
Top-1
FID. J
Dvar
0.498
0.339
2,982
trainable layers
0.045
feress
0.533
0.044
2.934
Word embedking layers
3.2M
0,550
0.040
Lalign
0.519
0.049
2.954
Subsequent layers
37M
0.022
7,611
layers.
Cum
0.324
0.524
2.983
Adapicr (LoRA) Zhang ctal.
2024b
0.4M
0.525
0.051
Lise
0.547
0.042
2.819
Full Model
0.550
0,040
2.811
4.3
ABLATION STUDY
Loss Functions. Table presents the ablation study of the loss functions. Removing Lyetts Leross or Lalign leads to varying degrees of performance degradation, highlighting their complementary and essential contributions. Specifically, Lier and Lares promote the discriminability and effectiveness of word-level disentangled motion semantics, while Lalign is critical for integrating these semantics into the word embeddings. Additionally, Lecut plays important role by enforcing alignment between entire sentences and motion sequences, thereby modeling contextual dependencies across words. Removing free also results in slight performance degradation, motion reconstruction can enhance the informativeness of motion features.
Optimization of Different Layers. Table compares the effects of optimizing different layers of CLIP within the MATE framework. Fine-tuning subsequent layers or the entire model markedly increases the number of trainable parameters, leading severe overfitting and degraded generation performance due to the limited size of motion datasets. Instead, MATE restricts optimization to the word embedding layers, effectively aligning word-level semantics while preserving the strong contextual representations captured by the subsequent frozen layers. We also evaluate the LoRA, which is a commonly used LLM muninu strategy Zhang et al. (2024b) by introducing additional lightweight layers while keeping the pretrained model frozen. However, it does not lead to notable performance improvements, suggesting limited alignment ability in our setting.
Table 5: Evaluation integrating with different Table 6: Ablation of the attention prior in Eq. 4 large language models HumanML3D.
on KIT.
Text Encoder
Top-1
Momask
FID 1
Top-17
MDM
Attention Prior
Top-11
FID.
MM-Dist_
No prior
0.253
2.794
CLIF RRdifrrd et al. 2021
0.521
0.015
0.191
0.630
+MATE
0.550
0.040
0.536
0.234
Binary
c2)
0.431
0.217
2,746
DistiREKT Sanh 2019
0.513
0053
0.493
Gaussian (8., < e..)
0.439
0.198
2,766
0045
Gaussian (0 5
0.443
0.197
2.732
Integration with Different Language Models. We tried constructing MATE with CLIP Radford
et al. (202T) and DistilBERT [aanh et al. (2019), two of the most commonly adopted language models in T2M methods, as the text encoder. As shown in Table incorporating MATE with either
8

---

Published as a conference paper at ICLR 2026
model consistently leads to remarkable performance gains, showing the strong compatibility and generalization capability of MATE across different language models.
Attention Prior. The attention prior, based motion localization, is evaluated in Table 6 "No prior" denotes using the raw motion features as keys in Eq. ) without any attention prior, making word-level semantic extraction from full sequences challenging. "Binary (s,, S S cn)" applies hard binary mask, assigning to frames within the target segment and elsewhere, which is highly sensitive segmentation errors. "Gaussian" the soft attention prior defined in Eq. H "Gaussian
(&n S 1 S e,,"" restricts the prior within the segment, with zero attention outside. "Gaussian (0 sts T)" extends the prior across the entire sequence, softly emphasizing the target region while gradually attenuating attention to neighboring frames, thus improving robustness to localization noise and achieving the best generation results.
4.4 VISUALIZATION RESULTS
"turn right and sit down"
"turn down"
"walk downstairs slowly"
"walk downstairs quickly"
(1)
(2)
Figure 4: Motions generated by MATE when individual words in the text prompts are replaced with their antonyms. In subfigure (2), both the left and right motions span the same time period, while the right motion descends more stairs and exhibits faster pace.
Motion Consistency with Word Change. To demonstrate that MATE effectively learns word-level semantic understanding, we present examples in Fig. where individual words in the text prompts replaced with their antonyms. MATE accurately captures the semantic differences between "right" and "left" as well as "slowly" and "quickly," and generates motions that are semantically aligned with the corresponding words, highlighting its robust fine-grained word-level understanding.
circle
sit
clockwise
jog
squat
slowly
upstairs
anti-clockwise
quickly
downstairs
word embeddings w/o MATE
word embeddings with MATE
attracted
repelled
Figure 5: Word embedding distributions on the unit sphere, visualized by DOSNES Lu et al. 2019 MATE brings kinematically related words closer together while separating antonyms.
Distributions of Word Embeddings. To better understand the mechanism underlying the perfor- mance gains of our approach, we visualize the effect of MATE on word embeddings Fig. 5 MATE draws together the embeddings of kinematically related motion words (e.g., "clockwise" and "circle". "run" and "jog") while pushing apart those with contrasting semantics (e.g., "quickly" and "slowly", "upstairs" and "downstairs"). This suggests that MATE structurally regularizes the word embedding space, promoting a closer alignment with motion semantics.
4.5 ANALYSIS OF TEXT-MOTION JOINT SEGMENTATION
Table 7: Quantitative evaluation of text-motion joint segmentation results.
Metries
HumanML3D
KIT
Text decomposition accuracy (%) Motion segmentation error (sec)
97.0
98.5
0.73
0.42
Quantitative Evaluation. To quantitatively evaluate text nottmmotion joint segmentation, we manually assess the text decomposition results and annotate ground-truth segmentation boundaries for 200

---

Published as a conference paper at ICLR 2026
motion sequences randomly sampled from the HumanML3D and KIT datasets, respectively. We then compute the average accuracy text decomposition and the boundary errors between our predictions and the ground truth, with results reported in Table 7 The decomposition accuracy reaches 97. .00% and 98.5% on the two datasets, demonstrating the strong text-processing capability of large language models within our approach. The small boundary errors (0.73s and 0.42s) further indicate that method achieves precise alignment, with deviations well within second, between decomposed texts and segmented motion clips. Notably, while some noise may be introduced around the segmentation boundaries, our Gaussian-bed based soft attention mechanism leverages soft localization signals rather than rigid hard segmentation, thereby improving robustness to such noise.
inaccurate boundary
"a person walks up stairs, turns around at the top, and walks back down the stairs'
"the figure spreads its arms, then claps, walks three steps forward: and finally halts and sits down"
(1)
Figure 6: Examples of text-motion joint segmentation. Different colors represent the decomposed sub-texts and their corresponding motion segments. The segmentation (1) is accurate, while in (2), the third motion segment corresponding to "halts and sits down" includes the partial semantics of the second sub-text "walks three steps", which reflects a minor boundary imprecision.
Qualitative Analysis. We present examples of text-motion joint segmentation Fig. The textual decomposition generated by ChatGPT is generally accurate. The primary source of error arises from imprecise boundaries predicted by the segmentation module. For instance, the boundaries in (1) are correctly identified, whereas in (2), the transition between "walks three steps forward" and "halts and sits down" is slightly misaligned. Such boundary inaccuracies are anticipated and mitigated by the Gaussian-shapee shaped attenuation mechanism in our framework.
LIMITATIONS
1) Although our method focuses on word-level semantic alignment, certain words (e.g.. "position". "starting", "area") inherently lack clear kinematic semantics or rely on contextual information. Future work will explore selective word-level semantic modeling strategies and the incorporation of contextualized queries to better handle such cases. 2) Word frequencies are imbalanced in the motion descriptions, which is not explicitly considered this initial exploration of word embedding fine-tuning, An important direction for future work is to explore re-weighting adaptive updating strategies to mitigate this imbalance.
6 CONCLUSION
In this work, we have proposed a systematic framework that integrates word-level motion localization, semantic disentanglement and alignment, addressing the text-motion misalignment fundamentally rooted in the word embeddings of LLMs for motion generation. Our approach not only demonstrates substantial improvements over state-of-the-art performance two benchmarks, but also highlights the strong potential of word embedding fine-tuning for enabling motion-aware language modeling.
ACKNOWLEDGMENTS
This work was supported by the EU Horizon project ELIAS (No. 101 101230,,7, the FIS project GUIDANCE (No. FIS2023-03251,, the National Natural Science Foundation of China (Grant No. 62502200), the Jiangsu Provincial Science and Technology Major Project (Grant No. BG2024042), and the Natural Science Foundation of Jiangsu Province (Grant No. BK20251203).
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu. Executing your commands via motion diffusion in latent space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18000-18010, 2023.
Seunggeun Chi, Hyung-gun Chi, Hengbo Ma, Nakul Agarwal, Faizan Siddiqui, Karthik Ramani, and Kwonjoon Lee. M2d2m: Multi-motion generation from with discrete diffusion models. European Conference on Computer Vision, 2024.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song. and Serge Belongie. Class- balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9268-9277, 2019.
Wenxun Dai, Ling- Hao Chen, Jingbo Wang, Jinpeng Liu, Bo Dai, and Yansong Tang. Motionlem: Real-time controllable motion generation via latent consistency model. European Conference on Computer Vision, 2024.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-effiiient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220-235, 2023.
Ke Fan, Junshu Tang, Weijian Cao, Ran Yi, Moran Li, Jingyu Gong, Jiangning Zhang, Yabiao Wang, Chengjie Wang, and Lizhuang Ma. Freemotion: A unified framework for number-free text-to-motion synthesis. European Conference Computer Vision, 2024.
Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5152-5161, 2022a.
Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng. Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts. In European Conference on Computer Vision, pp. 580-597, 2022b.
Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. Momask: Generative masked modeling of 3d human motions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1900-1910, 2024.
Yiming Huang, Weilin Wan, Yue Yang. Chris Callison-Burel Mark Yatskar, and Lingjie Liu. Como: Controllable motion generation through language guided pose code editing. European Conference on Computer Vision, pp. 180-196, 2024.
Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as foreign language. Advances Neural Information Processing Systems, 36, 2023.
Peng Jin, Hao Li, Zesen Cheng, Kehan Li, Runyi Yu, Chang Liu, Xiangyang Ji, Li Yuan, and Jie Chen. Local action-guided motion diffusion model for text-to-motion generation. European Conference on Computer Vision, 2024a.
Peng Jin, Yang Wu, Yanbo Fan, Zhongqian Sun, Wei Yang, and Li Yuan. Act as you wish: Fine- grained control of motion diffusion model with hierarchical semantic graphs. Advances in Neural Information Processing Systems, 36, 2024b.
Yanyu Li, Xian Liu, Anil Kag, Ju Hu, Yerlan Idelbayev, Dhritiman Sagar, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Textcraftor: Your text encoder can be image quality controller. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7985-7995, 2024.
Zhe Li. Weihao Yuan, Lingteng Qiu, Shenhao Zhu, Xiaodong Gu, Weichao Shen, Yuan Dong, Zilong Dong, Laurence Tianruo Yang, et al. Lamp: maaguagge-ootinn pretraining for motion generation, retrieval, and captioning. In International Conference Learning Representations, 2025.
11

---

Published as a conference paper at ICLR 2026
Han Liang, Jiacheng Bao, Ruichi Zhang, Sihan Ren, Yuecheng Xu, Sibei Yang, Xin Chen, Jingyi Yu, and Lan Xu. Omg: Towards open vecabuucyy motion generation via mixture of controllers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
482-493, 2024.
Hanchao Liu, Xiaohang Zhan, Shaoli Huang, Tai-Jiang Mu, and Ying Shan. Programmable motion generation for open-set motion control tasks. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 1399-1408, 2024a.
Jinpeng Liu, Wenxun Dai, Chunyu Wang, Yiji Cheng, Yansong Tang, and Xin Tong. Plan, posture and go: Towards open-world text-to-motion generation. arXiv preprint arXiv:23 14828, 2023.
Zhixuan Liu, Peter Schaldenbrand, Beverley-Claine Okogwu, Wenxuan Peng, Youngsik Yun, Andrew Hundt, Jihie Kim, and Jean Oh. Scoft: Self-contrastive fine-tuning for equitable image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10822-10833 2024b.
Shunlin Lu, Ling-Hao Chen, Ailing Zeng, Jing Lin, Ruimao Zhang, Lei Zhang, and Heung- Yeung Shum. Humantomato: Text-aligned whole-body motion generation. In International Conference on Machine Learning, 2024.
Yao Lu, Jukka Corander, and Zhirong Yang. Doubly stochastic neighhor embedding spheres. Pattern Recognition Letters, 128: 100-106, 2019.
Mingshuang Luo, Ruibing Hou, Zhuo Li, Hong Chang, Zimo Liu, Yaowei Wang, and Shiguang Shan. M gpt: An advanced multimodal, multitask framework for motion comprehension and generation. Advances in Neural Information Processing Systems, 37, 2024.
Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Mon Moll, and Michael Black. Amass: Archive of motion capture surface shapes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5442-545555 -5451, 2019.
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint
2018.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Mathis Petrovich, Michael J Black, and Gül Varol. Temos: Generating diverse human motions from textual descriptions. In European Conference on Computer Vision, pp. 480-497, 2022.
Mathis Petrovich, Michael J Black, and Güil Varol. Tmr: Text-to-motion retrieval using contrastive 3d human motion synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9488 0118-9997,700 2023.
Ekkasit Pinyoamuntapong, Muhammad Usama Saleem, Pu Wang. Minwoo Lee, Srijan Das, and Chen Chen. Bamm: bidirectional autoregressive motion model. In European Conference on Computer Vision, pp. 172-190, 2024a.
Ekkasit Pinyoanuntapong, Pu Wang. Minwoo Lee, and Chen Chen. Mmm Generative masked motion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1546-1555, 2024b.
Matthias Plappert, Christian Mandery, and Tamim Asfour. The kit motion-language dataset. Big data, 4(4):236-25) 252, 2016.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 9748-8763, 2021.
12

---

Published as a conference paper at ICLR 2026
Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Fine-tumed clip models are efficient video learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6545-6554 2023.
Zeping Ren, Shaoli Huang, and Xiu Li. Realistic human motion generation with cross-diffusion models. European Conference on Computer Vision, 2023.
Konstantinos I Roumeliotis and Nikolaos D Tselikas. Chatgpt and open-ai models: A preliminary review. Future Internet, 15(6):192, 2023.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.
Yoni Shafir, Guy Tevet, Roy Kapon, and Amit Haim Bermano. Human motion diffusion as generative prior. In International Conference on Learning Representations, 2024.
Teo Susnjak, Peter Hwang, Napoleon Reyes, Andre LC Barczak, Timothy McIntosh, and Surangika Ranathunga. Automating research synthesis with domain-specific large language model fine-tuning. ACM Transactions on Knowledge Discovery from Data, 19(3): 2025.
Guy Tevet, Brian Gordon, Amir Hertz, Amit Bermano, and Daniel Cohen-Or. Motionclip: Exposing human motion generation to clip space. In European Conference Computer Vision, pp. 358-374, 2022.
Guy Tevet, Sigal Raah, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffusion model. In International Conference Learning Representations, 2023.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in Neural Information Processing Systems, 30, 2017.
Yin Wang, Zhiying Leng, Frederick WB Li, Shun-Cheng Wu, and Xiaohui Liang. Fg-12m: Fine- grained text-driven human motion generation via diffusion model. In Proceedings of the IEEE/CVF 2023.
International Conference on Computer Vision, pp.
Yuan Wang, Di Huang, Yaqi Zhang, Wanli Ouyang, Jile Jiao, Xuetao Feng, Yan Zhou, Pengfei Wan, Shixiang Tang, and Dan Xu. Motiongpt-2: general-purpose purpose motion-language model for motion generation and understanding. arXiv preprint arXiv:2410.21744 2024a.
Zan Wang, Yixin Chen, Baoxiong Jia, Puhao Li, Jinlu Zhang, Jingze Zhang, Tengyu Liu, Yixin Zhu, Wei Liang, and Siyuan Huang. Move as you say, interact as you can: Language-guided human motion generation with scene affordance. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024b.
Jay Zhangjic Wu, Yixiao Ge, Xintao Wang. Stan Weixian Lei, Yuchao Gu, Yufci Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7623-7633, 2023.
Qi Wu, Yubo Zhao, Yifan Wang, Xinhang Liu, Yu- Yu-Wing Tai, and Chi-Keung Tang. Motion-agent: A conversational framework for human motion generation with 1lms. International Conference on Learning Representations, 2025.
Yiming Xie, Varun Jampani, Lei Zhong, Deqing Sun, and Huaizu Jiang. Omnicontrol: Control any joint at any time for human motion generation. In International Conference on Learning Representations, 2024.
Kangning Yin, Shihao Zou, Yuxuan Ge, and Zheng Tian. Tri-modal motion retrieval by learning a joint embedding space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1605,1605, 2024.
Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, and Xi Shen. T2m-gpt: Generating human motion from textual descriptions with discrete representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023a.
13

---

Published as a conference paper at ICLR 2026
Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208. 15001, 2022.
Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, and Ziwei Liu. Remodiffuse: patrieval-augmentee motion diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 364-373, 2023b.
Mingyuan Zhang. Daisheng Jin, Chenyang Gu, Fangzhou Hong, Zhongang Cai, Jingfang Huang. Chongzhi Zhang. Xinying Guo, Lei Yang, Ying He, et al. Large motion model for unified multi-modal motion generation. European Conference on Computer Vision, 2024a.
Yaqi Zhang, Di Huang, Bin Liu, Shixiang Tang, Yan Lu, Lu Chen, Lei Bai, Qi Chu, Nenghai Yu, and Wanli Ouyang. Motiongpt: Finetuned ilms general-purpont motion generators. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 7368-7376, 2024b.
Zeyu Zhang, Akide Liu, Ian Reid, Richard Hartley, Bohan Zhuang, and Hao Tang. Motion mamba: Efficient and long sequence motion generation. In European Conference on Computer Vision, pp. 265-282, 2024c.
Wenyang Zhou, Zhiyang Dou, Zeyu Cao. Zhouyingcheng Liao, Jingbo Wang, Wenjia Wang, Yuan Liu, Taku Komura, Wenping Wang, and Lingjie Liu. Emdm: Efficient motion difffusion model for fast, high-quality motion generation. European Conference Computer Vision, 2024a.
Zixiang Zhou, Yu Wan, and Baoyuan Wang. Avatargpt: All-in-one framework for motion understand- ing planning generation and beyond. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1357-1366, 2024b.
Qiran Zou, Shangyuan Yuan, Shian Du, Yu Wang, Chang Liu, Yi Xu, Jie Chen, and Xiangyang Ji. Parco: Part-coordinating text-to-motion synthesis. In European Conference on Computer Vision, pp. 126-143, 2024.
14


Reviews:
[{"review_id": "NYnX84dsJa", "reviewer": "Reviewer_NpWK", "summary": "This paper presents Motion-Aligned Text Encoding (MATE), a novel framework aimed at improving the semantic alignment between textual descriptions and human motion in text-to-motion (T2M) generation tasks. The authors introduce a lightweight fine-tuning strategy that updates only the word embedding layer of a pre-trained language model, while keeping all other layers frozen. This enables the embedding space to better capture motion-relevant semantics.\n\nMATE consists of two core components:\n- Motion localization, which aligns motion sub-sequences with corresponding text spans through soft attention guidance; and\n- Motion disentanglement, which employs contrastive learning with kinematic prototypes and cross-word negative sampling to disentangle word-specific motion features.\n\nExperimental results show that MATE can serve as a plug-and-play module for various T2M frameworks, significantly enhancing the performance of state-of-the-art models such as MoMask and MotionDiffuse on HumanML3D and KIT benchmarks. Additional ablation, visualization, and user studies further validate the effectiveness and generalizability of the proposed approach.", "questions": "1. If ChatGPT-assisted segmentation is unstable, how does the model handle inconsistent segmentation results? \nAdditionally, When a motion segment corresponds to multiple keywords, how does the system ensure these cues jointly guide generation? Does it produce two motion sequences (each with its own representation $f^m$) or merge them into a unified representation?\n\n2. Line 246 states \"we predefine a set of motion-word prototypes consisting of K learnable\nvectors\". Does “predefine” mean that the prototypes are manually specified or that only the number K is fixed while the prototypes are learned automatically? If the former, how are the prototype words selected?\n\n3. Was any negative transfer observed for non-motion-related words  during word embedding adjustment? Are there mechanisms to mitigate this?\n\n4. Could MATE’s capability of cross-domain generalization be evaluated through few-shot or zero-shot tasks, i.e., testing on unseen verbs, domains, or motion styles, rather than relying solely on the original training distribution?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper effectively identifies a major limitation in existing T2M approaches, word-level semantic misalignment, and addresses it through a simple yet effective embedding-layer fine-tuning mechanism.\nIt is the first to tackle the systemic mismatch between linguistic and motion semantics from the word embedding perspective, offering both scientific motivation and engineering value.\n\n2. By restricting fine-tuning to the word embeddings, the method remains computationally efficient and easily integrable into existing architectures.\n\n3. The proposed motion localization and disentanglement modules enhance the interpretability of semantic learning, and the contrastive objectives are elegantly designed.\n\n4. The experimental section is comprehensive, which These experiments strongly support the method’s efficiency.\n- Evaluation on multiple T2M baselines (e.g., MoMask, MMM, MotionDiffuse) demonstrate consistent improvements on HumanML3D and KIT datasets across metrics such as R-Precision and FID.\n- Visualization of embedding distributions further illustrates improved discrimination of action-related terms such as “left/right” and “quickly/slowly.”\n- The ablation studies thoroughly examine loss design, parameter settings, module compatibility, attention mechanisms, and semantic disentanglement.\n- Supplementary experiments—including integration with different LLMs, user evaluations, and cross-task tests (retrieval, editing, completion)—confirm the stability and versatility of MATE.", "weaknesses": "1. Text segmentation using ChatGPT may introduce stochasticity and hinder replication.\n\n2. The reliance on a fixed, limited set of prototypes could constrain generalization, potentially biasing the learned representations toward dataset-specific distributions.\n\n3. Direct fine-tuning of the text encoder might reduce its capacity to retain rich or nuanced semantics, especially for ambiguous expressions.\n\n4. While embeddings are improved, the inference stage still does not explicitly address temporal misalignment, leaving potential issues with word ordering or motion duration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GnKsIv2BK9", "reviewer": "Reviewer_XYL8", "summary": "This paper addresses a fundamental challenge in text-to-motion (T2M) generation. Existing approaches that rely on pretrained large language models (LLMs) often suffer from a semantic misalignment between motion-related vocabulary—such as “clockwise” or “quickly”—and the corresponding human skeletal movements. This issue stems from the generic nature of text embeddings in LLMs, which are not inherently designed to capture fine-grained kinematic semantics. As a result, the ability of T2M models to accurately interpret and generalize nuanced motion concepts remains limited.\n\nTo tackle this problem, the authors propose a novel framework named Motion-Aligned Text Encoding (MATE), which aims to explicitly inject motion semantics into the word embedding layer of LLMs, thereby enhancing the alignment between linguistic expressions and motion representations.\n\nThe MATE framework consists of two core components:\n\n* A motion localization strategy, which jointly decomposes paired text descriptions and motion sequences into semantically aligned sub-units. This process establishes a soft attention prior that facilitates the temporal grounding of word-level semantics in motion sequences.\n\n* A motion disentanglement module, which isolates word-specific motion semantics through two complementary mechanisms: \n    * self-disentanglement, which extracts shared semantics across related motions using contrastive kinematic prototypes; \n    * cross-disentanglement, which explicitly excludes irrelevant semantics to ensure discriminability between different motion words. \n\nThe disentangled motion semantics are then aligned with their corresponding word embeddings, effectively mitigating the word-level misalignment inherent in conventional LLMs.\n\nExtensive experiments validate the effectiveness of the MATE framework. The authors demonstrate that MATE-enhanced language models can be seamlessly integrated into existing T2M pipelines, leading to significant performance gains over state-of-the-art methods across multiple standard benchmarks.", "questions": "Please refer to the weakness.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The problem this paper focuses is critical for text-to-motion generation task. This paper is well-structured. In addition, the paper provide extensive experiments to prove the effectiveness of the proposed method. The proposed MATE is integrated into several existing T2M pipelines to validate its effectiveness.", "weaknesses": "## Major Concern:\nMy primary concern pertains to the foundational text-to-motion matching strategy outlined in Section 3.2. The proposed method relies on a pre-trained dual encoder to partition motion sequences into sub-clips by minimizing a matching loss. While this approach is intuitively reasonable, its effectiveness is critically dependent on the capacity and quality of the pre-trained encoders.\n\nThe dual encoder adopted from HumanML3D, in my view, may possess inherent limitations for fine-grained semantic alignment. These limitations stem from its relatively simple architecture, limited model scale, and—most importantly—the nature of its pre-training data. \nThe HumanML3D dataset provides only sentence-level motion-text pairs, lacking explicit, fine-grained alignment between sub-motion segments and their corresponding word-level descriptions. Consequently, I have significant doubts regarding the encoder's ability to accurately infer the optimal start and end boundaries for sub-motion clips based solely on weak, sentence-level supervision. \nAn inaccurate partitioning in this initial stage could propagate errors throughout the entire disentanglement and alignment pipeline.\n\nThis challenge is particularly acute in the domain of human motion generation, which is notoriously data-scarce, especially for fine-grained annotations. The situation stands in stark contrast to other domains like action detection or fine-grained action recognition, where similar matching strategies have succeeded. In those fields, methods typically benefit from powerful, large-scale dual encoders (e.g., CLIP) trained on massive, diverse datasets, which provide a robust semantic foundation. \nA similar replication of this paradigm in the data-poor motion generation domain, without a similarly robust backbone, represents a significant and potentially under-addressed risk in the current work.\n\n## Other Concern\n* **Novelty:** As indicated in my major concern, the core technical components—including fine-grained clip recognition, text-segment pairing, and contrastive learning for alignment—have been extensively explored and validated in other research areas (e.g., video-text alignment). While the integration and application of these ideas to the T2M generation task is valuable, the conceptual framework itself does not appear to be fundamentally novel. Therefore, I am inclined to assign a more moderate score regarding methodological innovation.\n* **Discussion of Related Work:** To better position this work within the broader research landscape, I would recommend the authors to include a discussion of contrastive learning-based methods from related fields.\n* **Clarification of Figure 2:** Figure 2 (3) is a bit unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t5r4fhB2IC", "reviewer": "Reviewer_t8pa", "summary": "This paper proposes MATE, a novel framework that explicitly incorporates motion semantics into the word embedding layers of large language models to enhance text-motion alignment for motion generation.", "questions": "1. Do the authors have any solutions for data with annotation errors? Such data may have many negative impacts on the training results.\n\n2. In the demo, I am curious why the generated motion makes a larger circle when \"quickly\" appears in the prompt. Is this because the semantics of some actions and words have not been completely disentangled?\n\n3.Given the failure of the re-weighting strategy for word frequency imbalance, could the authors elaborate on the differences between T2M word embedding alignment and standard long-tail learning? Are there adaptive update strategies (e.g., dynamic weighting based on prototype stability) that could mitigate this issue?\n\n4.Could the authors compare MATE with recent LLM-based T2M methods (e.g., MotionGPT2, MotionGPT) in terms of model complexity, training cost, and fine-grained semantic alignment ability? This would better highlight MATE’s advantages.\n\n5.Author should discuss with other works on the misalignment between motion and language question.\n\nI hope author can increase the presentation quality and answer my questions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.The paper proposes a novel framework (MATE) that explicitly addresses text-motion misalignment rooted in LLM word embeddings, a critical limitation in prior T2M methods.\n\n2.The paper’s structure is logical, with clear problem formulation, method description, and experimental analysis. \n\n3.Several experimental results show the effectiveness of this method.", "weaknesses": "1. The issue of misalignment between textual semantics and motion semantics has been raised in several works, such as LaMP. However, the authors neither compare their method with these works in the baseline nor discuss the differences between their solution and others.\n\n2. There are numerous motion annotation errors in HumanML3D and KIT-ML datasets. For example, the motion shows the left hand being raised, while the textual annotation states the right hand. Such noisy data will have a significant impact on the method proposed by the authors.\n\n3. The baseline lacks comparisons with LLM-based T2M methods, such as MotionGPT and MotionGPT2. The authors also fail to discuss the differences between their work and these methods.\n\n4. The overall writing is slightly rough, and the framework diagram is not clear enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IPipsMf1GN", "reviewer": "Reviewer_wVXx", "summary": "This paper introduces Motion-Aligned Text Encoding (MATE), which improves text-to-motion generation by fine-tuning only the token-embedding layer of a frozen language model to encode motion semantics. The framework has two parts: (1) a text–motion joint segmentation that temporally localizes word meaning within motion sequences, and (2) word-guided motion disentanglement based on dataset-wide kinematic prototypes with self- and cross-disentanglement losses. Plugging the MATE-enhanced encoder into existing T2M backbones yields consistent gains on HumanML3D and KIT. Ablation studies attribute improvements to each loss component and show that restricting updates to embeddings mitigates overfitting compared with tuning deeper layers or using LoRA adapters.", "questions": "1. Relative to training the T2M backbone, what fraction of wall-clock time and compute does training the motion-aligned embeddings add? \n2. The authors decompose each description with ChatGPT, which can introduce noise. Why not use datasets with human action and sentence-level annotations (e.g., BABEL [4])? \n\n>[4] Punnakkal, Abhinanda R., et al. \"BABEL: Bodies, action and behavior with english labels.\" *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*. 2021.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The plug-and-play strategy of injecting motion semantics into word embeddings is simple, effective, and likely useful for future T2M and broader multimodal systems.\n2. The module delivers consistent improvements across multiple T2M backbones and on both HumanML3D and KIT, demonstrating robustness and generalizability.\n3. Comprehensive ablations clearly isolate the effect of each component and provide strong evidence for the overall design.", "weaknesses": "1. The prototype-based contrastive alignment improves stability and semantic discrimination, but may trade off representational consistency against contextual adaptability. The reported generalization to unseen words seems to rely on strong contextual cues and semantic proximity to known terms.\n2. The evaluation centers on CLIP/DistilBERT encoders and does not assess more popular decoder-only language models. Testing MATE with T5-style architectures (e.g., MotionGPT [1], M3-GPT [2]) or LLaMA-based models (e.g., SoLaMi [3]) would provide a stronger case for paradigm-agnostic generalization.\n\n>[1] Jiang, Biao, et al. \"Motiongpt: Human motion as a foreign language.\" *Advances in Neural Information Processing Systems* 36 (2023): 20067-20079.\n\n>[2] Luo, Mingshuang, et al. \"M $^ 3$ GPT: An Advanced Multimodal, Multitask Framework for Motion Comprehension and Generation.\" *arXiv preprint arXiv:2405.16273* (2024).\n\n>[3] Jiang, Jianping, et al. \"Solami: Social vision-language-action modeling for immersive interaction with 3d autonomous characters.\" *Proceedings of the Computer Vision and Pattern Recognition Conference*. 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)