LaMP: Language-Motion Pretraining for Motion Generation, Retrieval, and Captioning
Authors: Zhe Li
Weihao Yuan
Yisheng HE
Lingteng Qiu
Shenhao Zhu
Xiaodong Gu
Weichao Shen
Yuan Dong
Zilong Dong
Laurence Tianruo Yang
Venue: ICLR 2025 Poster (2025)
Keywords: motion generation; motion-language pretraining; multimodal
Primary area: generative models
TL;DR: We propose LaMP, a novel language-motion pretraining model bridging language and motion gaps, enhancing motion sequence relevance and semantics in text-to-motion generation, retrieval, and captioning.

Abstract: Language plays a vital role in the realm of human motion. Existing methods have largely depended on CLIP text embeddings for motion generation, yet they fall short in effectively aligning language and motion due to CLIP’s pretraining on static image-text pairs. This work introduces LaMP, a novel Language-Motion Pretraining model, which transitions from a language-vision to a more suitable language-motion latent space. It addresses key limitations by generating motion-informative text embeddings, significantly enhancing the relevance and semantics of generated motion sequences. With LaMP, we advance three key tasks: text-to-motion generation, motion-text retrieval, and motion captioning through aligned language-motion representation learning. For generation, LaMP instead of CLIP provides the text condition, and an autoregressive masked prediction is designed to achieve mask modeling without rank collapse in transformers. For retrieval, motion features from LaMP’s motion transformer interact with query tokens to retrieve text features from the text transformer, and vice versa. For captioning, we finetune a large language model with the language-informative motion features to develop a strong motion captioning model. In addition, we introduce the LaMP-BertScore metric to assess the alignment of generated motions with textual descriptions. Extensive experimental results on multiple datasets demonstrate substantial improvements over previous methods across all three tasks. Project page: https://aigc3d.github.io/LaMP

Published as a conference paper at ICLR 2025
LAMP: LANGUAGE-MOTION PRETRAINING FOR MOTION GENERATION, RETRIEVAL, AND CAPTIONING
Zhe Li Weihao Yuan Yisheng He?, Lingteng Qiu², Shenhao Zhu Xiaodong Gu?,
Weichao Shen", Yuan Dong", Zilong Dong" Laurence T. Yang"!
1 Huazhong University of Science and Technology
Alibaba Group
3 Nanjing University
ABSTRACT
Language plays a vital role in the realm of human motion. Existing methods have largely depended on CLIP text embeddings for motion generation, yet they fall short in effectively aligning language and motion due to CLIP's pretraining on static image-text pairs. This work introduces LaMP, a novel Language-Motin Pretraining model, which transitions from language- -visin to a more suitable language-motion latent space. It addresses key limitations by generating motion- informative text embeddings, significantly enhancing the relevance and semantics of generated motion sequences. With LaMP, we advance three key tasks: text-to- motion generation, motion-text retrieval, and motion captioning through aligned language-motion representation learning. For generation, we utilize LaMP to provide the text condition instead of CLIP, and an autoregressive masked prediction is designed to achieve mask modeling without rank collapse in transformers. For retrieval, motion features from LaMP's motion transformer interact with query tokens to retrieve text features from the text transformer, and vice versa. For captioning, we finetune large language model with the iinguage-infommatvy motion features to develop strong motion captioning model. In addition, we introduce the LaMP-BertScore metric assess the alignment of generated motions with textual descriptions. Extensive experimental results on multiple datasets demonstrate substantial improvements over previous methods across all three tasks.
INTRODUCTION
Human motion represents an expanding frontier in computer vision, with substantial implications for diverse applications including film production, gaming, virtual reality, and robotics. Traditionally, textual representations have been favored for their simplicity and accessibility, enabling prominent tasks such as text-based motion retrieval, motion-to-text captioning, and text-driven motion generation. Despite the prevalent use of text in human motion tasks, challenges persist in achieving effective alignment between language and motion representations.
Previous methods usually leverage the pretrained CLIP (Radford et al. 2021 text encoder to extract embeddings from textual instructions for text-driven motion generation. Specifically, in many diffusion-basedd frameworks (Chen et al. 2023 Tevet et al. 2023 Lou et al. 2023 Zhang et al. 2023b), CLIP text embeddings are utilized conditional signals control the diffusion process. On the other hand, some transformer-bast
frameworks (Guo et al. 2022a Zhang et
al.) 2023a Zhong
al. 2023 Guo al., 2023) stack the text embeddings with the input guide the motion generation. In these frameworks, the extracted text embeddings serve as rich semantic representations and guide the generation process to produce motion sequences following the text instructions. In this paper, we seek to push the limit of the second path by designing a novel approach.
Although recent methods have achieved impressive results (Guo al. 2023 Zhang al. 2023a b Chen et al. 2023), they share common drawback: relying on text embeddings from CLIP as condition signals. CLIP well pretrained to align language and visual representation in the latent space. However, it is not optimal for aligning language and motion in two aspects. First, CLIP pretrained using text-image pairs, which may capture text embeddings that primarily represent
"Equal Contribution Corresponding Author Project Page: https://aigcdd.. thub. of 1o/laM//

---

Published as a conference paper at ICLR 2025
Learned Queries
Attention mask
Motion Encoder
Motion Transformer
Shared weights
Motien lext Contraction I corning
Motion entt Matching
Matien- Tixt Contrastive
Mution Cenernsiion
Text guiled
Monkon-giided fimurstion
Tex--pided Motion Genention
"A person walks a circlo"
Text Tokembzer
Text
Transformer
Matiu-Tan Maxching
Moticn guided
Figure 1: LaMP overview, We conduct joint training for contrastive learning, matching, and bidirectional lext-motion translation by leveraging the textual features extracted from tokenized text descriptions via the text transformer and the motion features derived from the motion transformer.
image features, potentially overlooking information relevant to motion. Second, since CLIP is pretrained on static images, it focuses predominantly on static characteristics, disregarding the dynamic elements associated with motion. As a result, the text embeddings generated by CLIP may not sufficiently reflect the relationship between text and motion, leading to suboptimal performance in applications that demand a high degree of synchronization between verbal descriptions and corresponding movements.
To address this problem, we propose LaMP, achieving a paradigm shift from a language- vision latent space to more appropriate Linguage-motion latent space. This brings three benefits. First, the text embeddings directly model the dynamics in the motion rather than only modeling the static state. Second, instead of signals, which enhances the quality of generated motion, allowing for more contextually relevant and semantically aligned motion sequences. Third, along with the motion-informative text features, can also acquire Language-informaate motion features. Thanks to these motion features, we achieve better motion-text retrieval and finetune a large language model (LLM) to perform motion captioning.
it can obtain notion-informative. text embeddings condition
In this paper, starting from the language-motion pretraining model LaMP, we improve three human motion tasks, i.e. text-motion retrieval, text-to-mote motion generation, and motion-to captioning. For the LaMP pretraining, we perform language-moting representation learning by constraining the LaMP derive text representations that are most pertinent the corresponding motion. To achieve effective language-moticn alignments, inspired by BLIP2 (Li et al. 2023) for language-vision alignment, four tasks are employed: motion-text contrastive learning, motion-text matching, motion-grounded text generation, and fext-grounded motion generation. After pretraining, utilize the extracted features LaMP-Feat for motion-text retrieval. The motion features from LaMP's motion transformer interact with query tokens retrieve text features from the text transformer, and vice versa. Next, propose a text-to-motion generation model LaMP-T2M. We adopt LaMP's text transformer in place of CLIP as the text encoder to extract text embeddings, which serve as the conditional signal guide motion generation. Following previous works (Devlin 2018 Guo et al.) 2023 Pinyoanuntapong al. 2024), we also employ the masked prediction technique. However, unlike previous methods, we adopt decoder-only architecture, which alleviates the degradation in expressive power caused by low-rank matrices (Dong et al. 202T) during the training process to some extent. Additionally, the causal attention mask enhances information interaction within the masked regions. Subsequently, we propose a motion-to-text captioning model LaMP-M2T. We leverage the motion transformer from LaMP to obtain ainguage-informatiin motion features, which are then fed into LLM. By finetuning this LLM using LoRA (Hu et al., 2022), we develop a LLM capable motion captioning. At last, propose evaluate the quality of generated motion using LaMP- MMM--errt Specifically, input the generated motion into LaMP-MP2 to obtain the corresponding textual description, and then calculate the BertScore (Zhang al.) 2019) between the generated text and the ground truth text. This metric serves to assess how well the generated motions align with the true semantics.
Extensive experiments conducted across various datasets demonstrate the effectiveness of our ap- proach in text-to-motion generation, motion-text retrieval, and motion-to-text captioning, with significant improvements compared previous state-of-the-art methods. For instance, the FID is decreased by 28.9% on the HumanML33 dataset, and 28.0% on the KIT-ML dataset for motion generation. The primary contributions of this work are then summarized as follows:
2

---

Published as a conference paper at ICLR 2025
We novelly propose Language- extract textual embeddings as conditional signals to guide motion generation. It not only ensures that the generated motions align more closely with semantic information but also reduces the gap between modalities. Furthermore, we observe that LaMP demonstrates outstanding capabilities in motion-text retrieval well. To the best of our knowledge, we are the first replace the Sanguage-vision space of CLIP with a language-motion space in motion generation.
Maaguage-MMotin Motion Pretraining model, termed as LaMP, and apply it to
We propose a motion generation model LaMP-12M, where we employ an autoregressive masked prediction mechanism. This approach alleviates the reduction in expressive capability caused by low-rank matrices during training and enhances the information interaction among masked regions.
We propose a motion captioning model LaMP-M2T, where an LLM is fine-tuned from the LaMP motion feature. Based on this, also introduce new metric SereeBertScore for evaluating the extent to which generated motions align with semantic information.
We outperform previous methods in three tasks including motion-text retrieval, motion-to-text captioning, and text-to- tot-tooooooonnnnoo generation.
2 RELATED WORK
Text-guided Human Motion Generation Early research on text-to-motion generation can be divided into two primary approaches: diffusion based continuous regression (eevet et al. 2023
Chen al. 2023 Zhang et al., 2022 Lou a [2023] Zhang et al./ 2023b Yuan al. 2002) and
wranformer-bbsed discrete classification (Zhang et al., 2023a ZZhon et al. 2023 Guo et al. 2023
Zou et al. 2024) Yuan et al. 2024). Among the diffusion- based methods, Motion (Zhang et al.
2022) stands out as the first model to use fine-graines text instructions for body part motion. MDM (Tevet et al. (023)) learns the relationship between motion and input conditions on raw data, while MLD (Chen et al. 2023) reduces computational overhead with latent space diffusion process. In the transformer- based category, motion is encoded with VQ- VAE Van Den Oord et al.) 2017) to create discrete tokens for prediction tasks. T2M-GPT (Zhang et al.] (023a) generates motion sequences from discretized inputs via a transformer. MoMask (Guo et al. (202aa) employs a motion residual VQ-VAE with multiple codebooks, enhancing generation through residual transformer. This paper follows the transformer- based path but simplifies the process by utilizing single codebook without a refinement step, resulting in a simpler and more flexible framework.
Motion-Texx Text Retrieval Motion retrieval remains an under-explored area, which presents challenges due its cross-modal nature, necessitating nearest-neighhor searches between text and motion modalities. Recent work T2M (Guo et al. 2022a) focuses on constructing a retrieval model for evaluation, using margin-based contrastive loss and Euclidean distance for batch pairs. TEMOS (Petrovich et al. 2022) has synthesis branch to generate motions from text and creates cross-modal embedding space but only aligns positive pairs. TMR (Petrovich et al. 2023) builds upon TEMOS, and integrates contrastive training strategy that incorporates negative examples, thus improving retrieval from a diverse set of fine-grained motions. contrast, we adopt a completely different pipeline, using the motion and text transformer from pretrained LaMP directly for retrieval.
Human Motion Captioning To articulate human motion via natural language, (Takano & Nakamura 2015) formulates a mapping between motion sequences and linguistic descriptions utilizing two statistical models. TM2T (Guo et al.) 2022b) proposes a motion representation that condenses continuous motions into concise sequence of discrete variables, leveraging neural translation network for cross-modal mapping. Motiongpt (Jiang et al.] 2023) introduces a motion-language training scheme with instruction tuning, learn from task feedback and generate motion captions through prompts. DLP (Cai et al. (024) constructs new dataset MoCap and finetune a LLM on this dataset for motion captioning. Different from them, we propose simpler approach, which utilizes language-informativ informative motion text features and finetunes an LLM without instruction tuning.
3 METHOD
3.1 LAMP: LANGUAGE MOTION PRETRAINING
We present LaMP, a novel model that aligns motion and language more accurately than CLIP (Radford
al. 2021). Current approaches (Zhang et al. 2023a Chen et al. 2023) Guo et al. 2023 leverage
3

---

Published as a conference paper at ICLR 2025
Matioa Kncader
Motaa Tekar
"A pesce wills tni coce""
Eearned Quirtts
Niskad
LaMP
Cabiook
LaMP Motion
Transfiemer
Masked Transformer
I.M Decoder
MLP
Mation
Prodicted
cadz"
Docuder
FREEED
(0) LaMP-T2M
(b) LaMP-MET
Figure 2: LaMP-T2M and LaMP-M2T frameworks overview. (Leftt) Pretrained LaMP's text transformer is employed to extract condition embedding and autoregressive mask prediction is performed. (Right) Finetuning an LLM to achieve motion captioning.
CLIP pretrained text encoder extract text features as conditions for motion generation. However, the CLIP text encoder maps text features into an image-text aligned latent space, which may not perfectly align with motion features due the domain gap between images and motions. Inspired by (Li it al. 2023), we develop a model that can better capture infiionniformaaive text features and provide more precise conditions for motion generation. Furthermore, we leverage LaMP and learnable query tokens to project motion into the motion- motion-language latent space. With this representation, we finetune a large language model and obtain a motion-to-tex. model.
3.1.1 PRELIMINARY
adopt vanilla VQ-VAE (Van Den Oord et al. 2017) to convert motion sequence into one tuple of discrete tokens. Specifically, we use ID convolutional encoder & to encode the motion sequence MI:N E ND to the latent vectors m1.N E Raxd with downsampling ratio of N/n and latent dimension d. To obtain discrete tokens, the latent vectors M]mm where S is the size of codebook and is the index of embedding in C. Each latent vector is replaced by the nearest codebook embedding in C according the Euclidean distance, which can be formally denoted as: 2(m)) = m, H> 2ş- The quantized code sequence 21... is then projected back into the motion space to reconstruct the motion in via decoder D.
pre-define learnable codebook C := {(s, 2))dd /ges for
3.1.2 MODEL ARCHITECTURE
Our aim is align language and motion better. Similar (Li et al. 2023). LaMP comprises two transformers (Vaswani, 2017) sub-modules that share the same self attentiooo layers: (1) motion transformer,
transformer that interacts with the frozen motion encoder and query tokens, and (2) as in Figure [ The motion transformer learns to extract a set of salient motion features from the input motion, while the text transformer learns generate corresponding set of motion-informative textual outputs. By jointly optimizing these two sub-modules with four auxiliary tasks, LaMP can effectively extract motion-informative text features and language-inforine informative motion features.
We define set of learnable query tokens g, which serve the input to the motion transformer. These queries interact with each other through self-attention layers and with motion features cross-attention layers. Moreover, the queries can engage with the text through the same self-attention mechanism. Following (Li et al. 2023), we initialize LaMP with the pre-trained BERT (Devlin 2018), with the cross-attention layers randomly initialized.
3.1.3 MOTION-LANGUAAK ALIGNMENT
We jointly optimize four distinct objectives that share common input format and model parameters. Each objective employs unique attention masking strategy to modulate the interaction between motion and language.
Motion-Tex Contrastive Learning Contrastive learning is key technique for aligning modalities. Similar to CLIP (Rafford et al. 2021). we perform motion-text contrastive learning in LaMP. aims to bring positive samples closer together and push negative samples further apart in the latent space, thereby maximizing the mutual information between motion and language. We first input the motion embeddings output from the motion encoder into the motion transformer and interact with the
4

---

Published as a conference paper at ICLR 2025
query tokens q, obtaining the motion feature fu, and input the text into the text transformer obtain the text feature ft. Then the pairwise similarity is computed between fm and ft, and we select the highest one the motion-text similarity. Following (Li et al. (023). to avoid information leakage, we employ a unimodal self-attention mask, where motions and texts are not allowed to see each other.
Motion- Toxtonn-Tet Matching Matching tasks can enable finer-grained alignment between modalities. We frame the task as a binary classification, where the model is required to predict whether a given motion-text pair is matched or unmatched. We employ a bi-directional self-attention mechanism, enabling the model capture the interdependent relationships between motion and language. The motion embeddings, text, and query tokens are fed together into the transformer, and the output is passed to a binary linear classifier to obtain logits, which are then averaged across the batch to obtain the matching score. The matching task further increases the mutual information between motion and language, and achieves information interaction.
Motion-grounded Text Generation To further enhance the alignment between the two modalities and get the informative features, we hope achieve cross moosal translation in LaMP. Given a motion sequence as the condition, we intend to generate the corresponding text. We first utilize pretrained tokenizer tokenize the texts and obtain ground truth labels. Then input the motion embedding and query token into the motion transformer to obtain the motion feature fm. which is then fed into the text transformer. To generate texts autoregressively, we employ a causal attention mask and compute the classification loss Lmgt on the output tokens.
Text-grounded Motion Generation Different from BLIP-2 Li et al. 2023), we perform text- grounded motion generation in LaMP, thanks to pretrained motion VQ- TAE stage 1, to enhance the alignment. 2LIP-2 (Li et al.) 2023) primarily focuses on extracting visual features that can be comprehended by LLMs. However, we require not only language-infore informative motion features but also motion-innformatie text features. Consequently,
incorporate text- grounded motion generation during the pretraining phase of LaMP. The ground truth label of motion sequence can be obtained from the pretrained codebook C R in the first stage: (b. = 2(m:))i-1... We input the text into the text transformer and interact with query tokens g via the motion transformer's cross-attention layers. The resulting features are then used for autoregressive generation through the motion transformer. Additionally, we define a motion classifier head F to map the generated results {9.)i=1...n to the space R and compute the classification loss with the ground truth label.
Ligm Σ Cross-Entropy (F(.),
(1)
The joint training across the four aforementioned tasks endows LaMP with strong motion-language alignment capability. After training, the text transformer of LaMP can extract features, making it a better condition encoder than CL.IP for motion generation (Section 3.2). Given the current R-Precision Top-1 metric only achieves around 52% accuracy on real data, LaMP can also serve as a better motion-text retrieval model (Section and an evaluator for the R-Precision and Multimodal Distance metrics, as verified in the experiments. Additionally, the motion transformer can also extract language-infoneg informative motion features. We utilize language-informatin motion features based on pretrained LLM to train a motion- to-text model, and based on this model, we propose using the LaMP-BertScore metric to evaluate the quality of generated motion (Section 3.4
text
3.2 LAMP-T2M: MOTION GENERATION FROM TEXT
Drawing inspiration from Guo et al. motion tokens. As depicted in Figure we first randomly replace a subset of the sequence elements (m1, m......, m with special [M] token. Let m (m1, m2, .... [M], m2- [M], m, denote the sequence after this masking process. Our aim is to accurately predict the masked tokens, given the context text f and the partially masked sequence M thereby endowing the model with generative capabilities. Different from existing approaches, we use pretrained LaMP for extracting text features. The optimization objective is to minimize the negative log- ligelikelihood of prediction:
2023), mask transformer is employed in our work generate
Lmask
Σ [(M)
log p(m |m
5

---

Published as a conference paper at ICLR 2025
3.2.1 MASK STRATEGY
We adopt the same mask strategy MoMask (Guo al. 2023 We employ a cosine function Y to determine the masking ratio. Specifically, the mask ratio is calculated as 7(r) = cos(Tr E [0, 1], where [0. 1] and means fully masked sequence. During the training process, we randomly sample r from uniform distribution U (0, 1), and then we uniformly mask (?()) n tokens of the whole sequence.
Additionally, we also perform the remasking strategy utilized in the BERT (Devlin 2018). Specifically, a token chosen for masking, we replace with [M] token with probability 80%, with a random token with probability 10%, and keep unchanged with probability 10%.
3.2.2 AUTORFGRESSIVE GENERATION
Unlike the bidirectional attention mask in MoMask (Guo et al., 2023), we employ causal attention mask for autoregressive mask prediction tasks. Currently, transformer-be models (Guo et al. 2023 Zhang al. (023a) commonly utilize bidirectional attention masks, which corresoond to encoder-only model architectures. However, during training, the bidirectional attention mask allows the model to simultaneously rely on contextual information, simplifying the mask prediction task and diminishing the model's generative capacity.
motion generation
In addition, this bidirectional masking leads to rank collapse. The attention matrix generated by bidirectional attention mask typically arises from the product of low-rank decomposed matrix and a softmax function; specifically, it results from multiplying an X d matrix with d matrix before applying softmax (where n > d). This form of attention matrix suffers from reduced expressiveness due to low-rank issues (Dong et al., 202T). In contrast, the attention matrix for a causal attention mask is a lower triangular matrix, with its determinant equal to the product of its diagonal elements. Due to the presence of softmax, all diagonal elements must be positive, ensuring that its determinant also positive. Consequently, the attention matrix of the causal attention mask (decoder-only architecture) guaranteed to be full-rank, providing greater expressiveness. Therefore, we predict the masked regions autoregressively:
log Pa(m (t, mz)
max
3.3 LAMP-FEAT: MOTION TEXT RETRIEVAL
Given text query T T-ror instance, "A per- son is walking in a circle"- primary ob- jective is to rank motions from a comprehen- sive database based their semantic alignment with the textual input. The ultimate aim to retrieve the motion that exhibits the highest cor- respondence with the provided textual descrip- tion, effectively bridging the gap between dis- and motion. We hypoth-
T2M
Ours
Figure 3: Heatmap of similarity matrix. The diagonal represents positive sample pairs, with darker colors indi- cating better quality.
parate modalities esize that model exhibiting better alignment between these modalities will inherently possess improved modal retrieval capabilities, and can serve more reasonable R-Precision evaluator.
To facilitate this alignment, we leverage the motion embeddings obtained from a frozen motion encoder, which undergoes processing through the motion transformer component of LaMP. This setup enables the embeddings to interact dynamically with the query tokens yielding refined motion feature representation fm- Such refinement is pivotal for accurately retrieving the corresponding text feature ft, as generated by the text transformer. Conversely, our architecture is designed to support reciprocal interactions; the text embeddings can similarly inform and enhance the motion representations. This bidirectional exchange fosters a richer understanding of the underlying seman- tics, ultimately ensuring that the retrieved motion sequences are not merely similar, but truly relevant and contextually aligned with the input query. As shown in Figure 3 to validate that LaMP can better match motion-text pairs, we present the similarity matrix the form heatmap. The results indicate that LaMP demonstrates more superior retrieval capabilities than T2M (Guo al. 2022h).

---

Published as a conference paper at ICLR 2025
T2M-GPT
grondl x different position
L coes a iitle jump x list walk Reeward
K gi Kenthinng quatcll Walk back gnd forth
toae sumething
MoMask
Xolf ground thfflesen prsition
Eoles fitll walk forward
shelf
X waalle
something.
Outs
SCT ground
does lutle jumi fins walk fervonl
Sputsscething shelf Malk back and (arth
diffcrent position
? juce something
groad place busk grann aighty pe porlist
then Juua."
/oenthing cakal"
Figure 4: Qualitative results of to--tooooooonnnnooo generation on HumanML3D.
3.4 LAMP-M2T: MOTION TEXT CAPTIONING
With the assistance of LaMP, we finetune pretrained LLM (OPT-2.7b (Zhang al. 2023c) to generate corresponding text from motion using LoRA HHu et al. 2021). As depicted Figure 2 we employ a fully connected layer to linearly project the output motion feature fm into the same dimensional space as the text embeddings of the L.M.. We input these language-informatvy motion features from LaMP and a prompt into L.M.. Since LaMP has been pretrained to extract semantically meaningful motion representations, effectively acts as an information bottleneck, passing the most useful information to the language model.
We input the generated motion into LaMP-M2T to obtain the textual description of the generated motion, and compute the BertScore against the ground truth text, termed as serves evaluate the extent which the generated motion aligns with the semantic information.
which
3.5 INFERENCE
During the inference process, we initiate with a fully masked sequence m. (0) and aim to generate the complete sequence
K iterations. At the k-th iteration, given the partially masked token sequence m M (k), the model first estimates the probability distribution for the tokens at the masked positions and samples motion tokens based on this distribution. Subsequently, the tokens with the lowest confidence scores, specifically the lowest [>(r)-n [n] values, are re- re-masked, while the remaining tokens stay unchanged for subsequent iterations. The updated token sequence m. (k: + 1) serves as the basis for predicting the token sequence in the following iteration until A reaches K. Afterward, all tokens are decoded into motion with pretrained VQVAE decoder in stage
Classifier-free Guidance The classifier-fre guidance (CFG) method (Chang et al. 2023 Ho Salimans 2022) is employed to integrate text embeddings into the transformer framework. In the training stage, the transformer trained unconditionally with probability of 10%. During inference, CFG is implemented at the last linear projection layer right before the softmax operation. At this point, the final logits (f, are computed by adjusting the conditional logits 4. relative to the unconditional logits Lnc. using guidance scale denoted as here is set 4:
1 (1 + o.) 1o a Lue
(4)
7

---

Published as a conference paper at ICLR 2025
R Precisiont
LaMP-
MulliModal Dist
Methods
Top
Top
3
FID4
BerSSoore"
Diversity
Ground Truth
0.51 11111:000 2.003
0.703
0.797
0.002
100.00
2.974
9.503
TM2T Guo al.
2022h
0.4244-003
003
0.618
003
0.729 002
1.501 4 017
3.467 011
8.589
Cioo et
0.455#.003 000
0.636= 003
0.736
002
1.087#.022 .211
3.347= 000
9.175
Tevet al.
2023
0.61 0 007
0.544
5.566
022
9.559---086
MLD st
2023
0,481
0.673
0.772
0.4734-001
52.08
3.190
9.72444002
HumanM1 3D
MotionDiftuse
Zhang et 2022
0,491
0,681
0. 0.782
0. 0.630
3.113
9.110
12M-GPT
Zhang et al. 20250
0.4922.003
0.67y=
0.775 002
0.141 2.00
56.21
3,121 - 000
9.761 7.08
PhysDan
Tuan
A1765
0.631
0.433
MationGPT
Zhanng dial
20124
01567
3.775
M2DM
Kong el all
2026)
0.497
0.682
002
0.763
0.352
0.352*t.006
2.974 2.974
9.926 9.926 +.77
Fig T2M
Wang el al
1204
0.4924
002
0.683=
003
0.783 002
0.243
3.109= 007
9.2784.072
AIT2M Zhoong el
2023
0.499* 000
0.690=
000
0.786
002
0.112 2000
3.038 007
9.700
DiverseMotion
Lou
20233
0.496+.000 004
0.687=
000
0.783
008
0.070+t004
3.063=-00 011
4.061
PurCo ez al
[Tb4]
0.515
0.706
0.706
0.801
0.801+-002
0.109
2.927
9.576 9.576
MMM Pinyoamuntapong.
2024
0.504
0.696
0.696
0.794 002
0.080=t.0m
2.998 11998==000 DOT
9.411
ReMoDiffuse (Zhang et al
2023b
0.510+.005
0.698
DOG
0.795
004
0. 0.103+0004
2.974 016
1.018+.070
McMask (uuo al.
+.02
0.713
.002
0,807
3077+.00
0.02
0.0455 +.002
60:40
2.958 -008
Ours
0.557 003
0.751
002
0.843
4 003
0.03244.00
60.81
2.759 2t .007
9.571 4: 1.0
Ground Truth
0.424+0005
0.649
008
0.779
000
0.031 +.00
100.00
2,788
11.080 090
TM2T
Gue al.
2022b
0,280
0,463
0,587 000
*.599*.158
1.591 020
9.473 0117
12M
Guo et
0.3614.005
0.559=
0.681
3.0224.307
2.052
10.724.145
MDM
Tevel et al.
2022
0.396 001
0.4974.0z
9.191 022
10.854.109
MI.D
Chen al.
2023)
0.390 008
0.609= 009
0.734 007
0. 0.404 .077
48.47
3.204 027
10.80
MotionDiniusc
iZhang et al 2022
0.417#
0,621= 0004
0. 0.7392.000 001
1.934 0052
2,958 DOS
11.10
KIT-MI.
T2M-GPT
Zhang ct
20250
0.416 005
0.627=
D06
0. 7.745 008
0.514 g.
46.53
3.007 022
10.864.00p
PhysDiff
Yuan
2025)
0.510 0.0
0.698
.000
0.795 004
0.100-t.004
0.103
t.004
MotionGPPT
Zhaag al.
2024
0.510
0.698=
.006
0.795
U. 0.103=t.004
2.974
10.54
M2DM
Kong al.
2023)
0.4164.004
0.628=.00
0.743
0.5154.022
3,015
11.417=-009 097
Fg-T2M
Wang et a||
2023
0,418 003
0.626=
804
0.745
004
0.571 2:07T
3,111 015
10.93
AITT2M
Zhong et
2033
0.413+.008 00G
0,632
806
1751#.001 000
0.870#.088
3.039=000 000
10.96
DiverseMotson
2023
0.416 003
0.637=
808
0.760
0111
0.4682.098
2,892 041
10.873 101
ParCo (Zo
[1144
0.430
0.649=
0.772
0.4534.00
2.820
10,95
MMM Pinyoanuntapong et
2024
0.3/44.005
0.590=.000
0.718
0.429#.010
3.146=.01
10.633 097
ReMoDDiluse Zhang et al
223
0.427 +014
0.641 804
0.765
0.76550005
0.155 4.006
2.814
10.80 10.80+.105
MoMask
Gua al.
0.433 + 007
0.656
D0S
0.781 4.00
0.204 7011
56.89
2.779
-002
Ours
0.17g 008 0. 0.691
+ 00D 0.826
.00
0.141
57.54
2.704
10.929
Table 1: The quantitative results of text-to-motion generation with evaluator following previous methods the HumanMI.3D dataset and the KIT-MI. dataset.
LaMP-R Precision?
Datasets
Methods
LaMP-R R MultiMudal Dist
Top 2
Top 3
T2M-GPT Zhang
2023a
0.796 002
0.90.4
0.92.1#.001
1.549*.006
HumanML3D
MoMask (uuo et al. 2045
0.82 00e
0.937 one
0.957 001
1.306=.0x
Ours
0.962 t.007
0.981 too
1.187#.008
T2M-GPT Zhang et al MaMask (Gno et al. 2024
2023a
0.710 009
0.8 0.844 one
0. 096 001
1.4934-019
KIT-ML
0.732 002
0.871 000
0.922 0o1
1.289=.0zz 027
Ours
0.784 th.00
0.907 0011
0.969
1.174 #.018
Table Evaluation results of not-toomotion generation with LaMP evaluator on T2M-GPT, MoMask, and ours.
4 EXPERIMENTS
4.1 EXPERIMENTS SETTINGS
Datasets We evaluate our model on HumanML3D (Guo et al. 2022a) and KIT-ML (Plappert et al. 2016) datasets. The H3manML3D dataset comprises 14,616 motion sequences accompanied by 44,970 textual descriptions, while the KIT-ML dataset includes 3,91 movement sequences and 6,278 text inputs. In line with the methodologies established in prior research (Guo et al. 2022a), we allocate 23,384 samples for training, 1,460 for validation, and 4,383 for testing within HL3a and utilize 4,888 for training, 300 for validation, and 830 for testing in KIT-ML. The extracted motion poses yield motion features with dimensionalities of 263 for HumanML3D and 251 for KIT-ML. These motion features encapsulate both global attributes, such as root velocity, root height, and foot contact, well local details including joint positions, velocities, and rotations relative to the root.
8

---

Published as a conference paper at ICLR 2025
Text-Motion Retrievalt
Motion-Ter Maxion-Text Retievalf
Methods
Re1
R
RØ3 m
RØ51 f
R@10
R@11
R@21
R@31 1
R@5 1
R 10
Humnn 3D
TEMOS
Petrowich al.
2022
0.424
63.52
61.14
84.15
39.96
53.49
61.79
86.89
T2M
202
52.48
71.05
80.65
96.58
52.00
71.21
81.11
96.78
Petrowcched
2023
67.16
81.32
$6.81
91.43
95.36
67.97
81.20
86.35
91.70
95.27
67.18
81.9
87.0
t..2
95. 75
68.02
t0.3
82.1
10.08
87.5
t0.3
96.9
T2MOS
Petrovich al.
2022
43.88
58.25
67.00
74.00
84.75
41 .8
$5.88
65.62
75.25
85.75
KIT-ML
T2M
Gao et al.
2024
42:25
62.62
75.12
87.50
96.12
39.75
62:75
73.62
86.88
95.86
TMR
Potroicc cl
2025
49.25
69.75
78.26
87.88
95 .000
50.12
67. 12
76,88
88,88
91.
Ours
52.5=A7
71.849.5
84.74995
92.7403
51.0 97.640.
006
75.340.5
8 81.4
92.2# 97.6
Table 3: Text-motion (left) and motion-text (right) retrieval benchmark the HumanML3D and KIT-ML.
R Precisiont MultiModal Dist,
LaMP- BertScoret
Method
Bleu@
Blcu@4 Rouget Cider
Top I Top 5
T2MT (Guo al.
2022a)
0,516 0.823
2.935
32.2
48.9
7.00
38.1
16.8
Motiongpi Jiang et al. 2023
0.543 0.827
2.821
32.4
48.2
12.47
37.4
29.2
LaMP-MZI M21 (Ours)
0.547 0.831
2.808
32.7
47.8
13.04
37.1
28.9
Table 4: The quantitative results of motion captioning on the HumanML3D, we adhere the evaluation
frameworks outlined in (Jiang ct al. 2023).
The local joint data accounted for corresponds to 22 and 21 joints, respectively, from the SMPL model (Loper et al. 2023) for the HumanML3D and KIT-ML datasets.
Implementation details Our model is implemented on NVIDIA A100 GPU using PyTorch. For the motion VQ- VAE, we employ resblocks for both the encoder and decoder, with a downscale factor of 4. The VQ consists of 6 quantization layers, where each layer's 's codebook contains 512 512-dimensional codes. The quantization dropout ratio p is set to 0.2. The masked transformer composed of 6 transformer layers with casual attention masks, 6 heads, and a latent dimension of 384. The learning rate reaches 2e-4 after 2000 iterations with linear warm-up schedule for the training of all models. During inference, we set the CFG scale of mask transformer as 4 on HumanML3D, and 2
on KIT-ML. Meanwhile, K was set to 10 on both datasets.
4.2 EVALUATION
Evaluation of Text-to-Motion Generation We conduct extensive evaluation of our model against prior text-to-motinn approaches, encompassing both diffusion-based and transformer-based models. Our results, summarized in Table I reveal that our method demonstrably outperforms all previous methods on both the HumanML3D and KIT-ML datasets. Notably, our model achieves an improvement of 6.9%, 5.3%, and 4.7% in R-Precision Top {1, 2, 3) HumanML3D, respectively. Furthermore, we also improve FID by 28.9% on HumanML33, and achieve state-of-the-an LaMP- BertScore, highlighting the remarkable fidelity of our generated motions. Qualitative results presented in Figure corroborate these findings, demonstrating that our method produces motions exhibiting a significantly better alignment with the input text compared to existing techniques.
We also report LaMP-R Precision and MMllimMmlti Distance obtained by extracting motion embeddings and text embeddings using LaMP in Table showing significant improvements over the existing evaluator. This reveals that the LaMP evaluator could better evaluate methods in the motion generation task.
Evaluation of Motion-Text Retrieval To prove the retrieval ability of LaMP, we also evaluate standard retrieval performance for both eext-to-motion and motion-to-tex tasks, using metrics analo- gous to the R-Precision. These metrics include R @ (1,2,3,5110), where a higher value signifies better performance. Recall at rank & represents the proportion of instances where the correct label appears within the top & retrieved results. Importantly, all retrieval evaluations are conducted TMR (Petrovich et al.) 2023) that includes unseen real motion. Our results including text-motion retrieval and motion- text retrieval, are summarized in Table
test set of
Evaluation of Motion-to- Text Captioning Motion-to-tex entails generating textual descriptions based sequences of motion. We perform comparative analysis of the LaMP-M2T against the recent TM2T (Guo et al. 2022a) and Motiongpt Jiang et al.] (2023). Our evaluation is conducted
Jhang
on the Human 3D dataset, utilizing the metrics proposed in (Guo al. 2022a). In Table 4

---

Published as a conference paper at ICLR 2025
Precision
FID4
LaMP
MultiModal Dist.
Top
Top
Top
BenSe
LaMP with contrastive learming
0.519
0.7142-005
0.084 0.8114-002 4.002
57.61
2.876
+ motion-text mathcing
0.5474 002
0.732
0.8244 002
0. 0.067 002
58.90
2.855 007
+ grouuon-gruunee text generation 0.519 002
0.737
0.828
0.062 0.062..002
60.47
2,806 ouT
+ text- text-grounded motion generation 0.557 2.577.000
0.751 1.00.000
0.032##.02 0.843-h.001 001
60.81
2.759
Table 5: Ablation study of the impact of different tasks in LaMP on generative performance on HumanML3D,
R Precision?
LaMP- BertScoref
FID
MuliiModal Distj
Top
Top
Top
Baseline with CLII
0.510
0.701 0.701
0.7964.000
0.086
57.49
2.902
+ LaMP
0.548#.002
0.748
0.840
#.020 .022
0.040
002
59.84
2.704 2.704
+ Query tokens
0.550+-002
0.741 0.741 +000
0.834
+002
0.042
002
60.54
2.9783+.000 000
+ Decoder-oaly
0.5574.003
0.751 4.0
0.843 21 001
0.082
1:0822:000 002
60.81
2.759 007
Table 6: Ablation study of lext-to-motion generation on HumanML3D. We report the impact of LaMP's text
encoder, interactions with query tokens, and the mask prediction manner on the results.
follow the settings in (Jiang et al. 2023) and adopt the raw ground truth text descriptions to facilitate more precise evaluation. The results indicate that LaMP-M2T has competitive performance in generating text descriptions corresponding to motion sequences.
4.3 ABLATION STUDY
In the ablation experiments, we primarily examine the contributions of LaMP, query tokens, and autoregressive mask prediction to the quality of motion generation.
Proxy Tasks in LaMP To validate the significance of each task during the LaMP training process, we report the impact of each task on the generated results in Table The text- soxt-groudded motion generation task aids LaMP in extracting motion-informative text features, making this task the most influential on the generation results.
LaMP as Text Encoder To validate the effectiveness of our proposed LaMP, we first establish a baseline model that utilizes CLIP the text encoder. We then replace CLIP's text encoder with our proposed LaMP text transformer, using the resulting text features as new conditions to guide the motion generation. Based on the results presented in Table 6 we observe significant improvements in the model across all metrics.
Interact with Query Tokens As shown in Figure D during the training process of LaMP, the query tokens do not directly interact with the text features. However, we believe that the query tokens carry significant amount of motion information, which enhances the accuracy of the conditional guidance. Therefore, we further enable interaction between the text features and the query tokens through cross-attention layers, resulting improved generative effects shown in Table
Decoder-only Mask Transformer Unlike MoMask (Guo et al. 2023), our model employs an autoregressive mask transformer. We believe that the bidirectional attention mask suffers from low-rank issues during training (Yang et al.) (2024). which reduces the model's expressive ability and lacks information interaction between the masked regions. We adopt a decoder-only model architecture with a causal attention mask, which mitigates the low-rank problem of the matrix and enhances information interaction between the masked areas. As shown Table the decoder-only model architecture achieves better generative performance.
5 CONCLUSION
In this work, we introduce LaMP, pioneering framework that bridges the gap between language and motion, achieving unprecedented alignment and performance in the process. Our findings elucidate how leveraging a nuanced approach to motion-text relationships can significantly enhance the generation of 3D human poses conditioned on textual descriptions. By integrating decoder-only mask transformer, our LaMP-T2M achieves state-of-the-an results human motion generation. Concurrently, propose the LaMP-M2T model for action description and introduce the LaMP. BertScore metric to evaluate the quality of the generated motions. Comprehensive experiments validate the effectiveness of the proposed approach.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Zhongang Cai, Jianping Jiang, Zhongfei Qing, Xinying Guo, Mingyuan Zhang, Zhengyu Lin, Haiyi Mci, Chen Wei, Ruisi Wang, Wanqi Yin, et al. Digital life project: Autonomous 3d characters with social intelligence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 582-5922999 582-592,22 2024.
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301 00704, 2023.
Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu. Executing your commands via motion diffusion in latent space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18000-18010, 2023.
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 181::1100000 14000405, 2018.
Yihe Dong, lean-Baptinn Cordonnier, and Andreas Loukas, Attention not all you need: Pure attention loses rank doubly exponentially with depth. In International Conference Machine Learning, pp. 2793-2803. PMLR, 2021.
Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5152-5161, 2022a.
Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng. Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts. In European Conference on Computer Vision, pp. 580-597. Springer, 2022b.
Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. Momask: Generative masked modeling of 3d human motions. 2023.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 102.20000006855 2021.
Biao Jiang. Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as a foreign language. Advances in Neural Information Processing Systems, 36:20067-20077 2023.
Hanyang Kong, Kehong Gong, Dongze Lian, Michael Bi Mi, and Xinchao Wang. Priority-centric human motion generation in discrete latent space. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11806-14818 2023.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael Black. Smpl: A skinned multi-person linear model. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pp. 851-866. 2023.
Yunhong Lou, Linchao Zhu, Yaxiong Wang, Xiaohan Wang, and Yi Yang. Diversemotion: Towards diverse human motion generation via discrete diffusion. arXiv preprint arXiv:2309.01377 2023.
Shunlin Lu, Ling-Hao Chen, Ailing Zeng, Jing Lin, Ruimao Zhang, Lei Zhang, and Heung- Yeung Shum. Humantomato: Text-aligned whole-body motion generation. arXiv preprint 1eXiv:2310.12977 2023.
Mathis Petrovich, Michael Black, and Gül Varol. Temos: Generating diverse human motions from textual descriptions. In European Conference Computer Vision, pp. 480-497. Springer, 2022.
11

---

Published as a conference paper at ICLR 2025
Mathis Petrovich, Michael J Black, and Giil Varol. Tmr: Text-to-motion retrieval using contrastive 3d human motion synthesis. In Proceedings of the IEEE/CVP International Conference on Computer Vision, pp.
2023.
Ekkasit Pinyoanuntapong, Pu Wang, Minwoo Lee, and Chen Chen. Mmm: Generative masked motion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1546-1555, 2024.
Matthias Plappert, Christian Mandery, and Tamim Asfour. The kit motion-language dataset. Big data, 2(4):236-252, 2016.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PMLR, 2021.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
Wataru Takano and Yoshihiko Nakamura. Statistical mutual conversion between whole body motion primitives and linguistic sentences for human motions. The International Journal of Robotics Research, 14(10):1114-1325 2015.
Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffusion model. In The Eleventh International Conference on Learning Represen- fations, 2023.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Yin Wang, Zhiying Leng. Frederick WB Li, Shun-Cheng Wu, and Xiaohui Liang. Fg-t2m: Fine- grained text-driven human motion generation via diffusion model. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 22035-22044 2023.
Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng. Haoming Jiang, Shaochen Zhong, Bing Yin, and Xia Hu. Harnessing the power llms in practice: survey on chatgpt and beyond. ACM Transactions Knowledge Discovery from Data, 18(6):1-32, 2024.
Weihao Yuan, Yisheng He, Weichao Shen, Yuan Dong, Xiaodong Gu, Zilong Dong, Liefeng Bo, and Qixing Huang. Mogents: Motion generation based on tpatial-temporal joint modeling. Advances in Neural Information Processing Systems, 37:130739 1307993,1 2024.
Ye Yuan, Jiaming Song, Umar lqbal, Arash Vahdat, and Jan Kautz. Physdiff: Physics-guide human motion diffusion model. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 16010-16021, 2023.
Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xi Shen, and Ying Shan. Generating human motion from textual descriptions with discrete representations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14730- 14740, 2023a.
Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208. 15001, 2022.
Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, and Ziwei Liu, Remodiffuse: eectievallauummentee motion diffusion model. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 364-373, 2023b.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre- pre-trained trained transformer language models, 2022. 3:19-0, 2023c.
12

---

Published as a conference paper at ICLR 2025
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:190..0967,, 2019.
Yaqi Zhang, Di Huang, Bin Liu, Shixiang Tang, Yan Lu, Lu Chen, Lei Bai, Qi Chu, Nenghai Yu, and Wanli Ouyang. Motiongpt: Finetuned Ilms general-purpose motion generators. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 7368-7376, 2024.
Chongyang Zhong, Lei Hu, Zihao Zhang, and Shihong Xia. Att22m: Text-driven human motion gen- eration with mustieperseecttve attention mechanism. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 509-519, 2023.
Qiran Zou, Shangyuan Yuan, Shian Du Yu Wang, Chang Liu, Yi Xu, Jie Chen, and Xiangyang Ji. Parco: Part-coordinating text-to-motion synthesis. arXiv preprint arXiv:2403. 18512, 2024.
13


Reviews:
[{"review_id": "jIyAJUYp7k", "reviewer": "Reviewer_PasF", "summary": "In this work, the authors point out that existing text-motion related methods largely depend on the CLIP text embeddings, yet will fall short in effectively aligning the language and motion features since CLIP is pretrained on static image-text pairs. The motion information and dynamic characteristics are overlooked. Thus, this work introduces LaMP, i.e., Language-Motion Pretraining model to generate motion-informative text embeddings, significantly enhance the relevance and semantics of generated motion sequences. With LaMP, text-to-motion generation, motion-text retrieval and motion captioning are advanced through aligned language-motion representation learning.", "questions": "My main concern lie in that LaMP seems to be trained on small datasets, thus the generalization ability, the proposed BertScore may be not that reliable. Meanwhile, I also want to know how to deal with the simple text descriptions and errors for LaMP?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work points out a significant problem for the current motion generative tasks, where the CLIP text embeddings are insufficient for motion generation;\n2. The authors have conducted extensive experiments on text-to-motion generation, motion-text retrieval and motion captioning and show substantial improvements over baseline methods;\n3. The paper writing is good and easy to follow.", "weaknesses": "1. CLIP is trained on large-scale image-text pair data, thus it can serve as a strong text feature extractor with strong generalization ability. However, it seems that LaMP is pretrained on a small text-motion dataset, thus the generalization ability of the LaMP features may be worse than CLIP’s. This ablation study should be provided to further verify the robustness and effectiveness of LaMP.\n2. LaMP-BertScore is introduced as an evaluation metric to better measure the alignment between motion and text. However, since LaMP is trained on small-scale datasets, the metric may be not reliable and convincing.\n3. In current text-motion datasets such as HumanML3D and KIT-ML, the text descriptions are very simple and may include errors, such as left-right body part mistakes. How can LaMP handle these dataset problems to produce a good text-motion alignment model?\n4. Some typos: L082, pertaining → pretraining, L096, 097, an LLM → a LLM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "THtM1OZ3pk", "reviewer": "Reviewer_S3ZK", "summary": "This manuscript presents LaMP which aims at improving the alignment of language and human motion. Unlike traditional methods relying on image-text models like CLIP, LaMP directly aligns language with motion data through a dedicated language-motion latent space. This approach enhances tasks such as text-to-motion generation, motion-text retrieval, and motion captioning. LaMP leverages a transformer-based framework with four key objectives, including contrastive learning and cross-modal generation, resulting in improved performance across various datasets. The model also introduces a new metric, LaMP-BertScore, to evaluate alignment quality between text and generated motions.", "questions": "Please see the weakness part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. LaMP replaces traditional image-based text embeddings with motion-focused ones, effectively improving the semantic alignment between language and motion data.\n2. The model successfully tackles three critical tasks—text-to-motion generation, motion-text retrieval, and motion captioning—demonstrating versatility and effectiveness across each task.\n3. The introduction of LaMP-BertScore offers a valuable metric for assessing the quality of generated motions, particularly the semantic fidelity of textual descriptions. This enhances the evaluation process in the domain of motion generation", "weaknesses": "1. Comparison with DLP[1] on motion-to-text generation.\n2. Failure cases. Please provide some failure cases for further analysis.\n3. The details about the Text encoder-decoder. How about the reconstruction metrics on the KIT and HumanML3D datasets.\n\n[1] Cai et al. Digital Life Project: Autonomous 3D Characters with Social Intelligence. CVPR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lm2nvlh6Mu", "reviewer": "Reviewer_ZgrR", "summary": "The paper presents a language-motion pretraining approach which can address motion generation, retrieval, and caption problem. The proposed approach targets the problem of enhancing the relevancce and semantics of generated motion sequences. Experiments on multiple datasets have validated the effectiveness of the proposed approach. More specifically, the proposed approach has reasonable performance on the tasks of motion generation, retrieval, and caption.", "questions": "Please address the questions raised in the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The task of language-motion pretraining is of great importance to the community. \n2. The paper obtains reasonable performance on several benchmarks like HumanML3D, KIT-ML datasets with different evaluation tasks.", "weaknesses": "1. The novelty of the paper is not clear. As LLM model is already been utilized for the motion understanding and generation, for example, MotionGPT, what are the major contribution of the paper against the previous work based LLM?\n\n2. The algorithm is based on BERT. How about the performance if the approach is based on a large LLM, for example, LLAMA-7B?\n\n3. For the text caption, the approach attaches another pretrained LLM model (e.g., OPT-2.7B) to generate captions. This may be due to the limitations from the BERT model. It would be better if the motion caption generation can be integrated into one model. \n\n4. In the implementation details, what the CFG scale is set different for humanKL3D and KIT-ML datasets? (Also, what is the value for humanML3D dataset)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UT17gzuyVM", "reviewer": "Reviewer_yD2w", "summary": "The paper introduces LaMP, a novel pretraining model that aligns language and motion to enhance the performance of multiple text-motion tasks including text-to-motion generation, motion-text retrieval, and motion captioning. The authors propose a new paradigm that transitions from language-vision to language-motion latent spaces, addressing the limitations of relying on CLIP for motion-related tasks. LaMP generates motion-informative text embeddings and utilizes an autoregressive masked prediction mechanism to improve the generation process. The paper reports substantial improvements over previous methods on multiple datasets and introduces a new metric, LaMP-BertScore, for assessing the alignment of generated motions with textual descriptions.", "questions": "None", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-organized and easy to follow.\n2. The authors have achieved a paradigm shift from language-vision alignment to language-motion alignment. Motivated by BLIP-2, they employ four text-motion tasks for joint pretraining, leading to a more enriched latent space that encapsulates rich motion dynamics information and informative embeddings for both motion and text.\n3. Through extensive experiments, the authors showcase significant improvements over state-of-the-art methods across three key tasks. Notably, their proposed method exhibits substantial performance enhancements in human motion generation tasks on the HumanML3D and KIT-ML datasets compared to previous approaches.\n4. The authors introduce the LaMP-BertScore metric as a novel evaluation measure to assess the quality of generated motions in relation to textual descriptions.", "weaknesses": "1. Some existing studies have also emphasized the downside of aligning static images and text in CLIP, and have suggested directly aligning motion and text in the latent space, as proposed in [1]. The authors should clarify the unique distinctiveness of their method compared to existing appro aches.\n\n2. The authors should include a computational complexity comparison between their proposed method and previous works to provide insights into resource consumption.\n\n[1] HumanTOMATO: Text-aligned Whole-body Motion Generation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)