HGM³: Hierarchical Generative Masked Motion Modeling with Hard Token Mining
Authors: Minjae Jeong
Yechan Hwang
Jaejin Lee
Sungyoon Jung
Won Hwa Kim
Venue: ICLR 2025 Poster (2025)
Keywords: text-to-motion generation
generative masked model
hard token mining
hierarchical semantic graph
Primary area: generative models

Abstract: Text-to-motion generation has significant potential in a wide range of applications including animation, robotics, and AR/VR. While recent works on masked motion models are promising, the task remains challenging due to the inherent ambiguity in text and the complexity of human motion dynamics. To overcome the issues, we propose a novel text-to-motion generation framework that integrates two key components: Hard Token Mining (HTM) and a Hierarchical Generative Masked Motion Model (HGM³). Our HTM identifies and masks challenging regions in motion sequences and directs the model to focus on hard-to-learn components for efficacy. Concurrently, the hierarchical model uses a semantic graph to represent sentences at different granularity, allowing the model to learn contextually feasible motions. By leveraging a shared-weight masked motion model, it reconstructs the same sequence under different conditioning levels and facilitates comprehensive learning of complex motion patterns. During inference, the model progressively generates motions by incrementally building up coarse-to-fine details. Extensive experiments on benchmark datasets, including HumanML3D and KIT-ML, demonstrate that our method outperforms existing methods in both qualitative and quantitative measures for generating context-aware motions.

Published as a conference paper at ICLR 2025
HGM HIERARCHICAL GENERATIVE MASKED Mo- TION MODELING WITH HARD TOKEN MINING
Minjae Jeong" Yechan Hwang' Jaejin Lee Sungyoon Jung
Won Hwa Kim
Pohang University Science and Technology (POSTECH), Pohang, South Korea fmin jminjaetidtid, yechan99, jlee4923, sy syjung,
ABSTRACT
Text-to-motion generation has significant potential in wide range of applications including animation, robotics, and AR/VR. While recent works on masked motion models are promising, the task remains challenging due to the inherent ambiguity in text and the complexity of human motion dynamics. To overcome the issues, we propose novel text-to-motion generation framework that integrates two key components: Hard Token Mining (HTM) and a Hierarchical Generative Masked Motion Model (HGMS). Our HTM identifies and masks challenging regions in motion sequences and directs the model to focus on hard-to-learn components for efficacy. Concurrently, the hierarchical model uses semantic graph rep- resent sentences at different granularity, allowing the model to learn contextually feasible motions. By leveraging shared-weight masked motion model, it re- constructs the same sequence under different conditioning levels and facilitates comprehensive learning of complex motion patterns. During inference, the model progressively generates motions by incrementally building up coarse-to-fine de- tails. Extensive experiments on benchmark datasets, including HumanML3D and KIT-ML, demonstrate that method outperforms existing methods in both qual- itative and quantitative measures for generating context-aware motions.
INTRODUCTION
Text-to-motion generation, which creates 3D human motion from textual descriptions, has recently emerged as an important task for a wide range of applications such as animation, graphics, robotics, and AR/VR (Azadi et al., 2023) Guo et al. 2024). As creating dynamic scenes with various motions from scratch costly and time-consuming. text-to-motion generation offers efficient solution by integrating natural language interfaces into the motion creation. Despite its potential, the task is still challenging because of the intrinsic ambiguity of human language and the complicated structure of human motion dynamics. To tackle these challenges, various deep generative models have been proposed, including methods based language- motion latent space alignment (Ahuja & Morency 2019), diffusion models (Tevet et al. 20233. and autoregressive models (Zhang et al. 2023).
Recently, generative masked models have become particularly popular for human motion generation (Pinyoamuntapong et al.] 2024b; Guo et al. 2024). as they offer a powerful approach for generat- ing realistic motion sequences from tokenized motion representations. These models first convert continuous motion data into discrete motion tokens, where certain tokens in the sequence are ran- domly masked. The models are then trained to infer the missing tokens based on the unmasked ones, enhancing their understanding of spatial and temporal relationships of action-specifics within the motion and ensuring the smooth continuity of the generated sequences. Such frameworks have proven effective in generating human motion, even from incomplete or noisy input data.
These masked motion generation models, however, rely on random masking strategies, which do not effectively target the most informative components in the data. Such approaches do not consider the varying levels of difficulty within motion sequences, Icading to suboptimal learning where both simple and challenging parts treated evenly. Indiscriminate masking may result in the model being insufficiently challenged during training, limiting its capacity to develop a deep understanding of difficult motion patterns. Despite the success of advanced masking techniques in other domains
These authors contributed equally this work.

---

Published as a conference paper at ICLR 2025
such masking hard patches in images (Wang al. 2023a), there has been limited exploration of how these techniques could be adapted to improve the learning process in human motion generation.
Moreover, previous eexttto-motion generative models typically rely on extracting a single sentence embedding from language processing methods such as CLIP (Radiord et al., (02I) to represent motion-from-tee, which can be potentially problematic for generating complex and sequential mo- tions. While these embeddings are suitable for static data such as images, human motion inherently involves dynamic sequence of actions and interactions, which cannot be fully captured by single vector representation. This results in a lack of detailed control and expressiveness in the generated motions, particularly for complex or nuanced sequences.
To overcome the limitations listed above, we propose 1) Hard "Token" Mining (HTM) strategy and 2) integrating it with hierarchical semantic graph-based textual conditioning. Inspired by image patch mining (Wang et al., (023a), our HTMM aims to selectively mask the most challenging tokens within a motion sequence. Implemented as a teacher-student scheme, the teacher model guides the student model to learn from the most challenging motion tokens by selective masking. HTM pushes the limits beyond random masking methods by focusing on challenging regions, thereby signifi- cantly improving the model's s abilitty to capture complex temporal (i.e., the progression and timing of actions) and spatial (i.e., how different joints move in relation to each other) dependencies. Ad- ditionally, we introduce Hierarchical Generative Masked Motion Model (HGM*), which employs a hierarchical semantic graph from Shi & Lin (2019) for a masked motion model to represent input text at various semantic granularities. Originally developed for natural language understanding, this representation allows our model to incorporate contextual information into the motion generation, improving the details and relevance within generated motion sequences.
To this end, implementation of our key ideas as an integrated framework leads to the following con- tributions: 1) We propose Hard Token Mining (HTM) strategy, which selectively masks challenging regions in motion sequences to enhance generation performance over traditional random masking approaches, 2) We introduce a hierarchical semantic graph representation into generative masked motion model that organizes text semantics into various granularity, which allows the model to in- corporate multiple layers of context and lead to more accurate and context-award motion generation. 3) We showcase the effectiveness of our model through both qualitative and quantitative evaluations, which demonstrate state-of-the-are results in standard texl-to-motion generation tasks.
2 RELATED WORK
2.1 TEXT-DRIVEN MOTION GENERATION
Early methods for text-driven human motion generation foeus on aligning the latent representa- tions of motion and text by minimizing the Kullback-Leible (KL) divergence or contrastive loss between their distributions (Ahuja & Morency 2019 Tevet al., 2022 Petrovich et al.) 2022 Ghosh et al. 202T] Guo et al., (022c). However, aforementioned methods suffer from unnatural motion generations due to the implicit discrepancy between the representation of the motion and text. To address this, learning the stochastic mapping between motions and texts was proposed. T2M (Guo et al. (002bb) utilized a temporal VAE to learn the mapping. and diffusion models such as MDM (Tevet et al.) (023) trained a transformer (Vaswani 2017) encoder to reconstruct noised raw motion sequences, while MLD (Chen et al. 2023) leveraged latent motion representations to enhance the computational efficiency of MDM. GraphMotion (Jin et al. 2024) introduced hierarchi- cal text conditionings defined in three semantic levels to gain finer control over motion generation.
Other works (Kong al. 2023 Wang et al. 2023b; Dabral al. 2023 Zhang et al., 2024 Huang 2024 Dai et al. 2025) also used the diffusion model for text-driven motion generation. et al.
Meanwhile, motion generation with masked transformers have shown better efficiency. T2M- GPT (Zhang et al., 2023) and MotionGPT (Jiang et al.] (2023) generated motion tokens autore- gressively from masked tokens, while MMM (Pinyoanuntapong et al., 2024b) employed random masking on input motion tokens. MoMask (Guo et al. (024) used residual VQ-VAE to recon- struct the final motion sequence using residual transformer. BAMM (Pinyoanuntapong et al.) 2024a) adopted a bidirectional causal masking to complement autoregressive transformers.
2.2 MASKING STRATEGIES IN MASKED MODELING
Masking strategies have been widely explored for masked modeling to obtain more generalizable representation. BERT (Devlin et al. 2019) introduces a random masking strategy on the input text tokens, where bidirectional transformers predict the masked tokens. VIT (Dosovitskiy et al. 2021)
2

---

Published as a conference paper at ICLR 2025
Forward Pass
+
Stop Gradient
Level wise concatemation
Weight Shared
Mortne--ened Tedt enbvaldng
Spertfioneewe embookling
Packling
Teput Text
person takes huge
Panysau-ford Mask Generation
loap forvard.
HTM
Graph Reavoning
Transformer Goue (Teacher)
PEMA Update
Iully randov masis
Handom mankka
HIM Transformer
Geym (Student)
Masked Mosien Tokens F
Illerarciical Text Graph
MQ
Recundrruciinn
Masked
+
Transformer (Student)
Predictio Les Gpodd
Recondraction
Leo
Inpur Moclon Sequence
Original
Reconstructed Tokem go
Mation Tokeas 1d
Overview of Training HGM'S The input motion X is encoded into latent features using pre- trained residual VQ VAE Encoder ε, which are then quantized into discrete motion tokens Y while the input text is processed through CLIP and GAT to generate a hierarchical graph embedding with motion-, action-, and specifics-leve level nodes as conditioning inputs. The framework uses (eacher-student paradigm, where the teacher Geus is updated by the student Ge and identifies challenging tokens to guide the masked transformer Fg. The Fe reconstructs the motion tokens oo from which the token-wise reconstruction loss E is computed. The loss f, along with the student- student-predicted loss then used to obtain the prediction loss Lpeed.
Figure 1:
extends similar random masking strategy to image patches. BETT (Bao et al. 2022) separately masks image patches and tokenizer tokens, using blockwise masking for the patches and random masking for the tokens. MAE (He et al.) 2022) showed that masking a large portion the entire image
patches helps reduce the spatial redundancy between them, and SimMIM (Xie al.) 2022) used
larger masks to learn good representations regardless of the masking ratio. Other works adopting random masking include MaskGIT (Chang et al., 2022) and Muse (Chang et al., (003)).
Rather than applying masks at totally random, some approaches leverage the semantic patterns of the input or learned strategies. SemMAE (Li al., 2022) uses a part segmentation map guide the masking process, which is derived by selecting the most prominent features from the self-attention maps. ADIOS (Shi et al. (022) learns how to mask using U-Net model with an adversarial training objective. Hard Patch Mining (HPM) (Wang et al. (023a) employs teacher-student model that transitions from completely random masking to focusing on difficult image patches identified by high reconstruction loss predictions from the teacher model. Inspired by HPM, propose effective masking strategy, i.c. HTM, that can be applied to motion tokens.
METHOD
Our goal is to develop a comprehensive text-to-motion synthesis framework that generates high- quality and contextually accurate human motion based on textual descriptions. To realize this, our approach integrates multiple components that jointly enhance both the representation and generation of motion data, as illustrated Fig. First, we utilize motion tokenizer based on a residual VQ- VAE (Van Den Oord et al., 2017 Guo et al.] 2024). which converts raw 3D human motion data into sequence of discrete tokens through a hierarchical quantization process (Sec. 3.1. Next, implement a masked transformer with a HTM strategy, which learns to mask discriminative parts of the motion and helps the model better recognize significant features the data (Sec. (.2). Furthermore, our framework introduces hierarchical semantic graph-based based textual conditioning for masked transformer, which structures semantic information into multiple levels to enhance the hierarchical inference process for motion generation, where the model iteratively refines the generated motion and employs pre-trained residual transformer correct quantization errors (Sec. 3.4).
model's comprehension of textual descriptions (Sec. 3.3) Finally,
3.1 MOTION TOKENIZER: RESIDUAL VQ-VAE
VQ-VVQ-VVVVVVV VAE conventionally used to encode motion sequences into a discrete latent space, which produces motion tokens that serve inputs for Transformer model (Vaswani 2017). VQ-VAE

---

Published as a conference paper at ICLR 2025
quantizes the continuous output of the encoder into discrete latent space through single vector quantization layer, which leads to information loss. To better approximate the encoder output, we used a residual VQ-VAE to quantize the lost information across additional V quantization layers. We train a separate single residual transformer following Guo et al. (2024) and Pinyoanuntapong et al. (2024a) predict the token sequences at the additional layers. The details are as follows.
Given a motion sequence X = II,N E N of length N with a frame dimension of D, let E(X) - 2 Raxd denote the output latent feature of the VAE encoder E, where N/n is the downsampling rate and d is the latent dimension. The input and output of the w-th quantizer, z", 2 E 12 are defined as follows:
2 Q(Z"),
2"
where Q(-) represents the quantization operation, which maps each vector 2 of the continuous latent feature Z" to its nearest entry in the codebook of e-th quantization layer. The reconstructed
motion sequence through VAE decoder D is then given by X = D (Ev=o ) and the training objective of the residual VQ-VAE is formulated as:
Cry = |X - X11 + 3 _ 1 - - 11
where /8 is hyperparameter for the commitment loss, and sg(-) denotes the stop-gradient which blocks the gradient flow during backpropagation. The first term of Liv is optimized using a straight- through gradient estimator (Van Den Oord et al. (017)). and our codebooks are updated using expo-
nential moving averages and codebook resets, as in Zhang et al. (2023) and Guo et al. (2024).
As result, each motion sequence X Ti:N is quantized into V + sequences [ via the residual VQ-VAE, and the sequences are converted into indices [y'.n]y=o _ with yi... E R", where each element of vi... corresponds to index in the e-th learned codebook. The first sequence contains the most information about the input motion and is used to train masked transformer, while the remaining sequences progressively contain less but still essential information and are used to train residual transformer. Detailed descriptions of the training and inference procedures for the residual VQ-VAE and the residual transformer are provided in Appendix A
3.2 MASKED TRANSFORMER WITH HARD TOKEN MINING STRATEGY
Given a sequence of motion tokens yo obtained from a pre-trained residual VQ- we apply masking to a subset of the tokens and train a masked transformer to reconstruct the original sequence. While existing methods such as Guo et al. (2024) and Pinyoanuntapong et al. (2024b) randomly select tokens to mask, we strategically choose the tokens to mask with criteria.
Our framework employs teacher-student scheme, where the teacher model dynamically generates challenging masked token sequences to guide the student model. As shown in Fig. I the proposed model consists of student model (Fe. Ge...) and a teacher model (Got.). ). Here, F-(-) repre- sents the masked transformer parameterized by 0, that reconstructs the masked tokens, while Go(.) an auxiliary transformer parameterized by d, which identifies challenging tokens for Fo(-) by predicting the relative ordering of reconstruction losses.
Token Reconstructor. Let y gl. be the masked motion token sequence. In this sequence, the tokens identified as challenging by the loss predictor G are replaced by special [Mask] token, while all other tokens remain unchanged from y We denote the set of indices of these masked tokens as M, which is dynamically updated at every epoch l. A masked transformer Fa(+) is trained on the masked sequence 4 with contextual text embeddings C,, which capture semantic information from the input text. Using both y and C, the masked transformer predicts the corrupted parts of the motion tokens. The reconstruction loss for each masked token is defined as:
la - log Fo Y .C),
(3)
where k M denotes the index of each masked token. The total reconstruction loss is then com- puted as:
free
Σ 1==) Σ
log Fo ,C).
(4)
NEM
KEM
4

---

Published as a conference paper at ICLR 2025
Loss Predictor. To identify motion tokens that more difficult to reconstruct, loss predictor is trained to predict argsort(( using a relative loss (Wang et al. 2023a). The model is trained to rank the reconstruction difficulty by predicting which token has the higher reconstruction loss for each pair of tokens ji, Given y and C, Go... is optimized by computing the loss over only the tokens that are masked, i.e., those in MM. The prediction loss Lpeed is defined as:
Spred ΣΣ REM JEM 121
1, log (ol()- E 1,) log (1-
(5)
FEM REM
where l GG (Y-0, C) represents the predicted relative reconstruction loss for each token from the student and (-) is sigmoid function. The indicator 1;j represents whether the value of & is greater or smaller than lj. i.c. Lij 1 if 2, > E and 0 otherwise. Here, Lree is detached from the gradient, serving as the ground truth for loss prediction. To ensure stable predictions from the teacher model, a momentum update (He et al. 2020) is applied as:
Orea < Adrea + (1 - A)dstu
(6)
where À represents the momentum coefficient. The student model is trained with two objectives: reconstruction loss and prediction loss. These two objectives are combined as:
C= Lree + Lpred:
where they alternate and complement each other. This interaction gradually encourages the student model to reconstruct challenging motion tokens, which leads to desired feature representation.
Easy-to-Hard Mask Generation. To help the model gradually adapt to more complex patterns and improve its ability to handle difficult tokens, we employ an Easy-to-Hard Mask Generation strategy. The masked regions are initially selected completely at random, gradually increasing the intensity by masking demanding tokens as training advances. This strategy ensures that the model starts by learning basic patterns and gradually adapts to more challenging parts of the data. In each iteration, the teacher model Geve. predicts the relative reconstruction loss for each token, which then ranked in descending order of difficulty using an a argsort()) operation. Tokens with the highest reconstruction loss are then selected for masking. For each epoch t, proportion a of the tokens to be masked is selected based on this ranking, while the remaining - a tokens are chosen randomly. Throughout the training, the proportion of hard tokens increases linearly as:
0t = 00 + 7 (OT - on).
(8)
where ao and aT are within the range [0, 1]. representing the initial and final proportion of hard tokens, respectively, and T is the total number of training epochs. Consequently, at each epoch, ae (((T)) n tokens with the highest predicted reconstruction loss masked, while the remaining (1 (1-ax) a-))))77 - ((r)) tokens are masked randomly. Here, we adopt a cosine function to schedule the masking ratio following Chang et al. (2022), defined as 7(T)) = cos
[0, 1]. where Ti is
2
randomly sampled from a uniform distribution 41 1) during training.
3.3 HIERARCHICAL SEMANTIC TEXTUAL CONDITIONING FOR MASKED TRANSFORMER
Existing works that generate motions from discrete tokens using transformers (Zhang et al. 2023
Guo et al. 2024 Pinyoanuntapong et al. (024a) rely on single implicit embedding as a condition token for given text. This approach overlooks the fine-grained details of the text. To address this, we propose the Hierarchical Generative Masked Motion Model (HGM²), which leverages enhanced text embeddings as input conditions with HTM. Inspired Chen al. (2020) and Din al. (2024), we decompose the text into semantically fine-grained components to construct hierarchical graph and obtain enhanced text embeddings through a Graph Attention Network (GAT) (Veličković al.) 2018). These embeddings are fed into the HGM facilitating a deeper understanding of the text.
We employ a semantic role parsing toolkit (Shi & Lin, 2019) to extract a hierarchical graph with three types of nodes, i.e., 1) motions, 2) actions, and 3) specifics, and twelve types of edges repre- senting various relationships between nodes. The sentence is treated as a global motion node, with verbs action nodes and attribute phrases specific nodes. The motion node is connected to all action nodes, while each action node is linked to its corresponding specific nodes. For further details on the semantic role parsing process, please refer to Appendix B
5

---

Published as a conference paper at ICLR 2025
Graph Reasoning, To obtain text embeddings for each node, we pre-trained CLIP, and attention based message passing is performed between neighboring nodes using GAT. From CLIP, obtain node embeddings c= [cmh, e",, ,'', where c" Rd represents the embedding of a single motion node, c < IR Xde denotes the embeddings of na action-level nodes, and c E R corresponds to the embeddings of n, specific-level nodes, each with dimension of de. Each node embedding is then transformed into h = { h" h through shared weight W E Rd Xde as:
=" h² c'W.
(9)
For connected nodes i and j E Nr, where N₁ represents the set of neighbor nodes for node we concatenate the embeddings of the two nodes to obtain hug h,; shared transformation vector for all edge types, and M, E = be relationship embedding matrix with distinct weights for E edge types. A one-hot vector riy E R represents the edge type between nodes i and j. The attention coefficient ēig is then obtained as:
hy] E R2dd. Let M R2d, be
exp(ei))
Eij LUM hig) + Leaky hig), Ei ΣnEN; exp(ex))
(10)
Finally, the output node embedding C: from GAT is computed as follows:
G sb
Σ @oghis
Ci.
(11)
JEN
where z denotes non-linear activation function.
Textual Conditioning for Masked Transformer. Eq. (4) and Eq. (5) are derived using CM" as the only condition. Leveraging the three levels of text embeddings obtained from Eq. (TT). we construct three types of conditions for each sentence: C" [Cth; C*). and [CTh; Ca; C*). where [:] denotes concatenation. The lengths the condition tokens are aligned with padding tokens, and position encoding (Vaswani 2017) is applied to the entire input sequence, including the condition tokens. These conditions are prepended to the motion token sequence and provided as input to the models, as illustrated in Fig. [ Instead of using only |CTm, C,m C*), providing all three types of conditions ensures that as the number of nodes increases from the root (motion level) to the leaves (specific level) of the semantic graph, the global meaning encapsulated by C" is not diminished, thereby maintaining balance between global and fine-grained information.
These three types of conditions are used not only to train Fg. but also G3- This is because we assume that the tokens that are difficult for Fg to predict will change depending on the type of condition given. Taking this into account, the losses are redefined follows. Let the token-wise reconstruction losses predicted by Go... with CmM, [Cmh; C*), and [Cmh; C?; C+] be denoted ln (o, and ش respectively. Using these losses, Lpred is computed separately as peed Lipred and L and the overall Ereed redefined as: Lpeed
Spre + Lpred fined-
(12)
Let 70,m, y0,a, and y0,. represent the masked token sequences, generated by masking the tokens identified as difficult by G for each level of conditions. Using each condition and masked se- quence, free is redefined as:
Lree Σ (-lg Fo (omm (m") log Fo Folyilyua (1"""; - log Fo """"""""""""", [C"h "0,; C"))))
REM
(13)
Throughout the overall process, a single model with shared weights is used, rather than training separate models for each of the three types of conditions. This approach serves two purposes. First, it improves efficiency by reducing the number of parameters, making the training process faster and more memory-efficien Second, weight sharing promotes the learning of consistent and generalizable features across all conditions. Without shared weights, the model may learn disjointed representations for each condition, leading incoherent or misaligned motion predictions.
3.4 INFERENCE: HIERARCHICAL PROCESS FOR MOTION GENERATION
As shown in Fig. the inference process involves two stages. In the first stage, we generate the base- layer token sequence using hierarchical approach. Starting from empty token sequence (O)),,

---

Published as a conference paper at ICLR 2025
Iaput
Mutiondledd Text unbablinng
Anrisnddesd Tedt anbaakling
Speefiesiewl Teas anbolding
Concalaation
knop Nexuet
Graph Reasiing
Metiun Token
Misked Mution Token
Palding
O0 Ea Gooder D
Masked Transformer F.
ger
Masked Transformer Fu
Repeet Revalon
Masked Transformer F
Generated
Repast
Molion Sequence
X Overview the hierarchical inference process for motion generation. Given an empty mo- tion sequence (with all tokens masked), the input text is transformed into hierarchical semantic features and concatenated with the masked motion tokens. The masked transformer F'g is applied iteratively: first utilizing motion-level features for LM iterations, then adding action-level features until LA iterations, and finally using all three levels until iterations. The generated tokens y are then iteratively passed through the pre- pre-trained residual transformer to produce yoovv which is decoded by VQ- VAE generate the motion sequence X.
Figure 2:
se
where all tokens are masked, the model generates the base-layer token sequence y = (L) of length n over L iterations. each iteration I, the masked transformer predicts the probability distri- bution for each possible token index at the masked positions, which reflects the model's 's confidence in each prediction. The [> ().m) [n]] tokens with the lowest confidence are masked again. This process repeats until the base-layer tokens are fully generated after L iterations.
During the iterations, the model is conditioned hierarchically at different levels to progressively refine the motion tokens. This allows the model to capture both general and detailed information from the input text. the initial iterations, the model is conditioned on the motion level semantic feature, which provides high-level guidance for the overall structure of the motion sequence. As the iterations proceed, the model incorporates the action-level condition, adding more precise infor- mation about the actions involved in the motion. Finally, in the later iterations, the specific level condition is applied, introducing finer details about the agents or objects interacting in the scene Each condition (t) for the I-th iteration is provided as:
m
if 1 S LM
C(I) = ICm
Ca Icm: C"; C*
if LM < <I< LA
(14)
LA < <
where LM and LA are the transition points between the levels of conditioning during the L iterations.
After the base-layer tokens are generated through the hierarchical process, the final motion sequence is reconstructed by the pre-trained residual transformer. The residual transformer progressively predicts V token sequences containing residual information, with each sequence refining details lost during quantization. Finally, the predicted residual tokens are combined with the base-layer tokens, which is passed through the VQ- VQ-VAE decoder to produce the motion sequence.
EXPERIMENT
In this section, present various experimental results for our model, HGM Sec. 4.1 covers the overall experimental setup. In Sec. 4.2) we demonstrate the superior quantitative and qualitative re- sults of our model compared to other state-of-the-at methods. Sec. [rresetsstts an ablation analysis and discussion, showing the effectiveness of the components that make up our model.
4.1 EXPERIMENTAL SETUP
Datasets. We evaluate HGM' on widely-used text-to-motion datasets: HumannML3D (Guo et al. 2022a) and KIT Motion-Languae (KIT-ML) (Plappert et al.] 2016). HumanML3D is cur- rently the largest publicly available dataset for 3D human motion with textual annotations. It con- tains 14.616 motion sequences sourced from the AMASS MMhmmood et al. 2019) and HumanAct12 (Guo et al. 2020) datasets, paired with 44,970 textual deseriptions. Hach motion accompanied by at least three detailed text annotations, with an average description length of approximately 12 words. The motions cover a wide range of activities such as exercising, dancing, and everyday actions. The motion sequences are processed at 20 frames per second (IPS) and have durations between 2 and 10 seconds. KIT-ML consists of 3,911 human motion sequences and 6,278 textual descriptions. Each motion is annotated with one to four textual descriptions, with average of 8 words per description. Both datasets are split into training, validation, and test sets with proportions of 80%, 5%, and 15%, respectively.
7

---

Published as a conference paper at ICLR 2025
Table 1: Quantitative results the HumanML.3D test set. All evaluations were replicated 20 times and the average with a 95% confidence interval are reported. (Bold: best result, Underlined: second-best)
R. Precision f
Mathods
FTD 4
MM-Dist 4
Diversity t
MModality 1
Top-11
Top-21 1
Top-3 1
Hier
0.301 = 002
0.425
0.552
6,523 024
5.012
8,332
TEMOS
0,424 002
0.612
0.722
3.734
3,703
8,973
0.368
TM2T'
0.424 003
0.618 003
0.729 003
1,501 2 017
3,4672.011 011
8,589 .76
2.424 083
T2M
0.455 002
0.636 003
0.736 003
1.087 021
3,347 008
9.175 .03
2.219 o74
MDM
0.320 0005
0.498 0004
0.611 - OUT
0.544 0444
5,566 t.566tt0027
9.559.t.00e
2.799 .072
MotionDifluse
0.491 001
0.681 002
0.782 002
0.630 011
3.113 001
9.410 ..19
1.5532.00 0012
MLD
0,481 -
0.673
0.772
0.473 01.3
3.1964.011
9,724
2.4134
Fg-T2M
0,492 002
0.683
0.783 003
0.243
3.109+.007
9,278 .072
1.614
M2DM
0.497 0003
0.682
0.763 0003
0.352
3.134th.010
9.926 .773
3,587
T2M- GPT
0.491#-002 -002
0.680 002
0.775 002
0.116 004
3.118-20011
9.761-0081
1.856 o0
GrsphMMttio
0.504 003
0.699 002
0.785 002
0.116 007
3.070 008
9.692 007
2,766 OBG
MMM
0.515 002
0.708 002
0,804 003
0.089 00G
2.926+.007
9.577 000
1,226 oas
MuMask
0,521
0.713
0.807
0.045
2,95g2.000
1.241
BAMM
0.525 0002
0.720
003 0.814
0.055
2.9194.000 005
9.7174 .099
1.887
HGM (ours)
13355h.00
0.726
4.002
0.822 3.22..000 .02
0.036
0.036#.002
2.904
9.545 0011
1,206 001
Table 2: Quantitative results on the KIT-ML test set. All evaluations were replicated 20 times and the average with 95% confidence interval are reported. (Bold: best result, Underlined: second- bbstt
Methods
R-Precision f
FID 4
MM-Dis 4
Divensity 1
MModality t
Top-1
Top-2 1
Top-3 f
[lier
0,255 006
0.432 007
0,531 007
5.203 107
4.986 027
9.563 072
TEMOS
0.353 000
0,561 21.00000 .0r
0.6874.00s ons
3.717 051
3.417 019
10.84 100
0.532##084
TMZTT
0,280 000
0,463 +.00 000
t.06
3.599 168
4.591 +.02
t.17
3.242
T2M
0.361 +-000
0,559 +..0
0.681 +000
1002--110 107
3.488
028
10.72
2.052*-107
MDM
0,164
0.291
0.396 t..9
0.497
9.191 9.191
10.85tt.100
1.907
MotionDiffuse
0.417 000
0 1.00
0.739
1.9542.004 064
2.958
056
11. 11.10221443
0,730=2003
MLD
0,390 pos
0.609 000
0,734 .077
0.404 097
3 3.204 017
10.804
2. 29122-0071
Fg-T2M
0.418 000
0.626#.000 .044
0.745 .044
0.571 047
3.114#.013 015
10.93 osa
1.019+.028
M2DM
0.416 000
0,628 004
0.743 -004
0.515
3.015 017
11.47
3.325
T2M-GPT
0,402
0.6194.omm
0,7374 .00
0,7174.041
3.053
10.86 091
1.9124.0036
GraphMotion
0.429
0.648
0,769
0,313
3.076*.022
11.12
3.627
MMM
0,404 000
0.621 2.00
0,744 0004
0.316
2,977
10.910
1.232
MoMask
0.4334.007 007
0.6564.omm oos
0.781 178814-00-
0.204
2.779 022
1.1314-0443
BAMM
0.438 4: 008
0.661 4 000
0.788 .05
0.183 013
2.723 026
11.008 0004
1.609 000
HGM (ours)
0.414 007
0.664 007
0.791 000
0.176 010
2.710
10.882+.081 081
1.152
Baselines and evaluation metrics. We evaluate HGM using the baselines referenced in the Sec. 2.1 which include state-of-the-ar text-to-motion generation methods, as listed in Tab. and Tab. The evaluation is conducted using five commonly adopted metrics in text-to-motion genera- tion tasks. All metrics use embeddings extracted from pre-trained models following the evaluation protocol from Guo et al. (2022a). (1) R-Precision assesses the alignment between input text and generated motion by ranking the Euclidean distances between the motion and 32 text descriptions (1 ground-truth and 31 mismatched). We report Top-1, Top-2, and Top-3 retrieval accuracy. (2) Frechet Inception Distance (FID) (Heusel et al. 2017) measures the quality of generated motions by comparing the distribution of their features with real motions. (3) Multimodal Distance (MM- Dist) calculates the Euclidean distance between each text feature and the corresponding generated motion feature, evaluating how well the generated motion aligns with the text. (4) Diversity com- putes the average Euclidean distance between randomly sampled pairs of generated motions. (5) Multimodality (MMooalityy) measures the model's ability generate diverse motions from the same text by averaging Fuclidean distances between motions generated from a single description.
Implementation Details. The residual VQ- contains quantization layers, each with a code- book of 512 codes of 512 dimensions. The downsampling rate N/n of VAE encoder is set to 4 and latent dimension is set to 384. 3 in Crog is set to 0.02. We use the ViT-B-32 CLIP model, where the dimension of the text representation is set to 512. All transformers consist of 6 transformer layers with 6 attention heads. For the HTM implementation, au and or are set to and 0.5, respectively. Our models are trained using the AdamW optimizer for 500 epochs. The learning rate is linearly warmed up to 2 over 2000 iterations. The batch size is set to 512 for training residual VQE, and 256 for training the masked transformer and the transformer predicting the reconstruction loss.
8

---

Published as a conference paper at ICLR 2025
HGM (ours)
MoMask
T2M-GPT
MLD
man walking around in counterclockwise circle with left arm hovering over left shoulder as walks.
walks forward, then squats to pick something up with both hands, stands back up, and resumes walking.
Figure 3: Qualitative comparisons. We present the motions generated by various models for two different
text prompts. where darker colors indicate later timestamps. Our model not only comprehends the overall semantics of given text but also captures detailed information effectively compared other models.
a person walks forward. is pushed backward, and then resumes walking forward.
walks backwards and then steps up stairs buckwards well.
the person trying to bug.
Figure 4: Visualization of text-to-motion generation results by HGM The red boxes highlight regions in each motion sequence where the HTM model Gegg predicts the highest reconstruction loss. These areas correspond sudden changes specifie, detailed actions, such being pushed backward, stepping up stairs backward, or attempting hit bug. By focusing on these challenging regions during training, model achieves high level consistency between the generated motions and the given textual descriptions.
For training the residual transformer, the batch size set to 64 for HumanML3D and 32 for KIT- ML. During the inference process, Las- LA. and L set 2. and 10. respectively, All our experiments are conducted on single NVIDIA RTX 6000 Ada Generation GPU.
4.2 COMPARISON TO TTATE-OF-THE-ARY APPROACHES
Quantitative Results. The test results for the HumanMI.3D and KIT-MI. datasets are presented in Tab. and Tab respectively. Following standard practices in text-to-motion generation (Guo al. 2022a), each experiment was repeated 20 times, and the reported values represent the mean with 95% confidence interval. Our model consistently achieves superior performance compared recent state-of-the--rt methods across multiple metrics, including R R-Precision, FID, and MM-Distance. While diversity and MModality are important. they should be viewed as secondary metrics along primary measures, i.e., FID and R- R-Precision (Guo al. 2024). HGM achieves superior FID of 0.036, much lower than M2DM's 0.352 on HumanM L3D. indicating higher output quality.
Qualitative Results. Fig. presents qualitative comparisons between our model, MoMask, T2M- GPT, and MLD. In the first row, only the motion generated by our model accurately exhibits the left arm hovering over the left shoulder while walking, whereas the others walk in counterclockwing direction but fail to depict this detail. In the second row, the motions from the other models do not resume walking after standing back up, while model correctly performs the full sequence. These results demonstrate that our model effectively captures fine-grained details and excels in generating motions that other models struggle reproduce.

---

Published as a conference paper at ICLR 2025
In Fig. 4 we visually demonstrate the validity of HTM, by highlighting the regions where Geven predicts the highest reconstruction loss. These regions typically correspond to parts of the motion involving sudden changes very specific actions. By focusing the learning process on such regions, our model is able to generate motions that are highly consistent with the given text. For additional qualitative results, including heatmaps illustrating the output of Goreo. please refer to Appendix C
4.3 ABLATION STUDY
Text Conditioning and Associated Training and Inference Strategies. An ablation study is per- formed to investigate the effects of different types of text conditioning, as well as the use of hier- archical inference and weight sharing among models. The results are presented in Tab. 3 and the experiments are conducted in the same manner as the evaluation in Tab. [on the HLmanMM3D..
First, when providing three types of text conditions, i.e., [C'"], [Cmh; C*), [CTh, C C*), the perfor- mance is higher in all cases compared to using only [C"h; C C*] We believe that this contributes to our model effectively balancing global and specific information. Moreover, utilizing a hicrar- chical process during inference yields improved motion generation, suggesting that generating an overall structure first and subsequently refining details more effective than attempting to produce a specific motion at one shot. Lastly, we found that using a single model with shared weights during training outperforms using three separate models for each condition type. We attribute this to the ability of the unified model to learn more generalizable features across all three types of conditions.
Table 3: Ablation study on text conditioning and associated training and inference strategics. The results are obtained from the HumanML33D experiment. "-" represents that the strategy cannot be implemented, while v and indicate whether the strategy is executed not, respectively.
Text conditioning
Hierarchical inference
Weight shared
R-Precision Top-1
FID 4
IC C* C*]
0.523 003
0.051 + 003
[C"",, [ 3 Co1 c C9; C
[C C*), J. [em; C* C*
0.526 + .084
0.042
0.531 002
0.040 002
1
; C* c 00,00 C*
0.535 .022
0.036 002
Mask Strategies. To identify an effective strategy for HTM, different values of aq and ar are evaluated on the HumanML3D. As training progresses, the ratio of hard tokens within the masked tokens increases from 0g or,, whereas the proportion of randomly masked tokens decreases ac- cordingly. Thus, when both ap and aT are set to 0, only random masking is applied throughout the training process, representing the easiest strategy. As ar increases, the proportion of hard tokens in the same epoch increases, leading to a higher level of difficulty. We found that our model achieved the best performance when ag and or are set to 0 and 0.5, respectively, as shown in Tab. 4 This demonstrates that learning which tokens to mask is effective, while maintaining a certain degree of randomness is necessary to prevent the task from becoming overly difficult. In particular, when both values are set to 1, the performance significantly drops, as the model struggles to learn effectively when it is consistently required to predict challenging parts based solely on less meaningful tokens.
Table 4: Ablation study masking strategies with varying dag and or values. As training progresses, the proportion hard tokens among the masked tokens increases from ag to ar.
Case
Difficulty
Randomness
ao
0-
R-Precision Top-1
FID 4
random
easy
strong
0
0.529 003
0.044 002
learn to mask
0
0.5
0.535 002
0.036
learn to mask
4
0
0.533 002
0.041
learn to mask
hard
weak
I
0.519 002
0.054 003
5 CONCLUSION
We propose a novel framework for human motion generation that integrates HTM and a hicrarchical semantic text into a generative masked motion model, i.c., HGM By selectively focusing on the most challenging motion tokens during training, HTM enhances the model's ability to capture complex dependencies, thereby generating smoother and more coherent motions. Additionally, the hierarchical semantic graph approach effectively organizes text into embeddings at various seales, allowing the model to incorporate multi-layered contextual information for precise and context- aware motion synthesis. Extensive experiments demonstrate that our model achieves state-of-the. art performance multiple benchmark datasets, significantly improving both motion quality and semantic alignment compared to existing methods. These results emphasize the effectiveness of our approach and provide potential for applications with realistic motion generation.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY
To ensure the reproducibility of our work, we present a detailed illustration of the training and inference process of our Hierarchical Generative Masked Motion Model with HTM in Fig. and Fig. 2 Furthermore, we provide the implementation details for HGM and the experimental results replicated 20 times and the average with 95% confidence interval on HumanML3D and KIT-ML datasets. We will relcase the full code and setup to facilitate reproducibility of our work.
ACKNOWLEDGEMENTS
This research was supported by RS-2022-11220290 (40%), NRF-2022R1422209333 (30%). 10240508-09825-01 (Samsung Electronics Co., Ltd, 20%), and RS-2019-11191906 (AI Graduate Program at POSTECH, 10%).
REFERENCES
Chaitanya Ahuja and Louus-PPiliieh Moreney. Language2pose: Natural language grounded pose forecasting. In International Conference on 3D Vision, 2019.
Samaneh Azadi, Akbar Shah, Thomas Hayes, Devi Parikh, and Sonal Gupta. Make-an iece--an-nnimation Large-scale text-conditional 3d human motion generation. In IEEE/CVF International Confer- ence on Computer Vision, 2023.
Hangbo Bao, Li Dong. Songhao Piao, and Furu Wei. BEÎT: BERT pre-training of image transform- ers. In International Conference on Learning Representations, 2022.
Huiwen Chang, Han Zhang. Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang. Kevin Murphy, William T Freeman, al. Muse: Text-to-image generation via masked generative transformers. In International Conference on Machine Learning, 2023.
Shizhe Chen, Yida Zhao, Qin Jin, and Qi Wu. Fine-grained video-text retrieval with hierarchical graph reasoning. In JEEE/CVF Conference Computer Vision and Pattern Recognition, 2020.
Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu. Executing your commands via motion diffusion latent space. In JEEF/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Rishabh Dabral, Muhammad Hamza Mughal, Vladislav Golyanik, and Christian Theobalt. Mofu- sion: A framework for denoising- dit-ddiiiuooobbseeee motion synthesis. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Wenxun Dai, Ling- Hao Chen, Jingbo Wang, Jinpeng Liu, Bo Dai, and Yansong Tang. Motionlem: Real-time controllable motion generation via latent consistency model. In European Conference on Computer Vision, 2025.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Associ- ation for Computational Linguistics, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, et al. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021.
Anindita Ghosh, Noshaba Cheema, Cennet Oguz, Christian Thcobalt, and Philipp Slusallek. Synthe- sis of compositional animations from textual descriptions. /EEF/CVF International Conference Computer Vision, 2021.
Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. Action2motion: Conditioned generation of 3d human motions. In ACM International Conference Multimedia, 2020.
11

---

Published as a conference paper at ICLR 2025
Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022a.
Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang. Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2022b.
Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng, Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts. In European Conference on Computer Vision, 2022c.
Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. Momask: Generative masked modeling of 3d human motions. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In IEEE/CVF Conference on Computer Vision and Patlern Recognition, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Neural Infor- mation Processing Systems, 2017.
Yiheng Huang, Hui Yang, Chuanchen Luo, Yuxi Wang, Shibiao Xu, Zhaoxiang Zhang, Man Zhang, and Junran Peng. Stablemofusion: Towards robust and efficient diffusion- motion genera- tion framework. In ACM International Conference on Multimedia, 2024.
Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as foreign language. Neural Information Processing Systems, 36, 2023.
Peng Jin, Yang Wu, Yanbo Fan, Zhongqian Sun, Wei Yang, and Yuan. as you wish: Fine- grained control of motion diffusion model with hierarchical semantic graphs. Neural Information Processing Systems, 2024.
Hanyang Kong, Kehong Gong, Dongze Lian, Michael Bi Mi, and Xinchao Wang. Priority-centric human motion generation in discrete latent space. IEEE/CVF International Conference on Computer Vision, 2023.
Gang Li, Heliang Zheng, Daqing Liu, Chaoyue Wang, Bing Su, and Changwen Zheng. Semmae: Semantic-guidee masking for learning masked autoencoders. Neural Information Processing Systems, 2022.
Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael Black. Amass: Archive of motion capture surface shapes. In IEEE/CVF International Conference on Computer Vision, 2019.
Mathis Petrovich, Michael J Black, and Gül Varol. Temos: Generating diverse human motions from textual descriptions. In European Conference on Computer Vision, 2022.
Ekkasit Pinyoanuntapong, Muhammad Usama Saleem, Pu Wang, Minwoo Lec, Srijan Das, and Chen Chen. Bamm: Bidirectional autoregressive motion model. In European Conference Computer Vision, 2024a.
Ekkasit Pinyoanuntapong, Pu Wang, Minwoo Lee, and Chen Chen. Mmm: Generative masked motion model. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024b.
Matthias Plappert, Christian Mandery, and Tamim Asfour. The kit motion -anguuge dataset. Big
data, 2016.
12

---

Published as a conference paper at ICLR 2025
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
Peng Shi and Jimmy Lin. Simple bert models for relation extraction and semantic role labeling. arXiv preprint 2XXiv:900.002200 2019.
Yuge Shi, N Siddharth, Philip Torr, and Adam R Kosiorek. Adversarial masking for self-supervised learning. International Conference on Machine Learning, 2022.
Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-On Motionclip: Ex- posing human motion generation clip space. In European Conference Computer Vision, 2022.
Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffusion model. In International Conference on Learning Representations, 2023.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Neural Informa- lion Processing Systems, 2017.
A Vaswani. Attention is all you need. Neural Information Processing Systems, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018.
Haochen Wang, Kaiyou Song, Junsong Fan, Yuxi Wang, Jin Xie, and Zhaoxiang Zhang. Hard patches mining for masked image modeling. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023a.
Yin Wang, Zhiying Leng, Frederick WB Li, Shun-Cheng Wu, and Xiaohui Liang, Fg-t2m: Fine- grained text-driven human motion generation via diffusion model. In IEEF/CVF International Conference on Computer Vision, 2023b.
Zhenda Xie, Zheng Zhang. Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: simple framework for masked image modeling. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xi Shen, and Ying Shan. Generating human motion from textual descriptions with discrete rep- resentations. In IEEE/CVF Conference Computer Vision and Pattern Recognition, 2023.
Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang. and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. IEEE Transac- fions Pattern Analysis and Machine Intelligence, 2024.
13

---

Published as a conference paper at ICLR 2025
A ARCHITECTURES OF THE RESIDUAL VQ- VAE AND RESIDUAL TRANSFORMER
A.1 TRAINING SCHEME OF RESIDUAL VQ-VAE
We illustrate the training scheme of the Residual -Q-VAE for motion tokenization in Fig. which is described in Section 3.1 of the main manuscript. 1) VQ-VAE Encoder: The VQ-VAE encoder encodes the input motion sequence X, generating continuous latent feature Z This feature is then quantized into discrete latent representations 20 through vector quantization operation Q(.), 2) Residual Processing: To effectively capture lost information during quantization of 7 20, residual learning approach is employed. Specifically, the lost information is retained for every layer and used as input for the next layer, i.e., Zu+1 Z 2o. 3) Quantization Layers: The architecture includes V residual quantization layers. Each layer quantizes the continuous latent features 2" into discrete features z". Here, 2 obtained by mapping each vector Z" to the nearest entry in the corresponding learned codebook. VQ-VAE Decoder: Finally, the aggregated quantized latent features from all layers are fed into the VQ-VAE decoder, which reconstructs the original motion sequence X.
Paiia Lncder
g DD
10x
Decdlr
Recastreee Supeare
Soquine 3
Figure 1: Training Scheme of the Residual VQ-VAE
A.2 TRAINING SCHEME THE RESIDUAL TRANSFORMER
As shown in Fig. 2, the residual transformer is trained to predict the next layer's token yi+1 ÿr++1 based on the tokens from the previous layers (Y" ,y..,, Y*} and text embedding e obtained from the CLIP encoder. The latent representations from the Residual VQ-VAE, denoted as {20...., 2 indexed into discrete token sequences {Y" ,..., Y'J. Fig. 2 illustrates this single residual trans- former, which is applied iteratively across layers. The reconstruction loss defined as:
Lrss Σ
log R. (yod cc.
where si is the j-th token of Y and Y is obtained by indexing 211
Input Text
person Iagge
leep fowwrrd
Resicked TrurstemEE
Embodding c
Reczeddaaiion Loss f...
Codebook Indexing
yas
31-
Indexing
20
21
>!
Figure Training Scheme the Residual transformer
14

---

Published as a conference paper at ICLR 2025
A.3 MOTION SEQUENCE GENERATION WITH RESIDUAL VQ- VAE AND RESIDUAL TRANSFORMER
The residual transformer is used predict the residual layer tokens Y ,.... YV from the generated base layer token y during the motion generation phase outlined in Section 3.4 of the main paper, as illustrated in Fig. 3. This single residual transformer operates iteratively across V layers, with each iteration predicting intermediate motion tokens based on the previous layer's output and the text embedding. As the process progresses through V iterations, the model incrementally incorpo- rates residual information, ensuring that each successive layer captures more detailed aspects motion. Ultimately, the outputs from each iteration the residual transformer, denoted as yov passed to the VQ-VAE Decoder. This decoder transforms the aggregated motion tokens into final generated motion sequence X, effectively translating the learned representations back into coherent motion that aligns with the input text description.
Residual Transformet
Residual Transformer
Residual Transformor
Residiall Transformer
VQ-VAE UQ-VADeeeode Decnder
Residual
Text Embedding
pez
Recandracted Median
Suquance
Generated Motian
Figure 3: Motion Sequence Generation with Residual VQ- VAE and Residual Transformer
B DETAILS OF HIERALRCHICAL SEMANTIC GRAPHS
Following (Jin et al. 2024). we employ semantic role parsing toolkit (Shi & Lin 2019) to develop Hierarchical Semantic Graphs, enabling us to identify actions along with their attributes and the roles those attributes play within motion descriptions. The parser analyzes the given motion description to extract verbs that signify actions and related attribute phrases, determining the semantic relationships of these phrases concerning the actions. In the graph, the full sentence is represented motion node, while the verbs are modeled as action nodes that connect to the motion node. Attribute phrases are represented as specific nodes associated with the action nodes, with the nature of their connection dictated by the semantic roles of these specifics. The types of nodes and edges in the graph are summarized in Tab. 1.
global
Table 1: Node and Edge Type Descriptions Hierarchical Semantic Graphs
Type
Description
global motion description
Node type
Action
verb
Specific
altribute of action
agent
ARG]
patient
ARG2
instrument, benefactive
ARG3
start point
ARG4
end point
ARGM-LOC
location (where)
Edge type
ARGM-MNR MNR
ARGM-
manner (how)
ARGM-TMP
time (when)
ARGM-DIR
direction (where to/from)
ARGM-ADY
miscellaneous
ARGM MA
motion action dependencies
OTHERS
other argument types, egg., action
15

---

Published as a conference paper at ICLR 2025
C MORE QUALITATIVE RESULTS
Additional qualitative results generated using HumanML33 text prompts are shown in Fig. 4. These results demonstrates that HGM successfully captures and reflects broad spectrum of motion quences corresponding to various textual descriptions, ensuring strong semantic alignment be- tween the text and the generated motions.
To further illustrate the model's 's effectiveness, we include heatmaps of the output of Gieen for se- lected motion sequences Fig. 5. The heatmaps visualize the predicted reconstruction difficulty across different parts of the motion sequence, with darker regions indicating more challenging areas. This visualization provides deeper insights into how the model identifies and focuses on complex patterns during motion generation.
person walks forward, and then shuffles quickly to the right and then walks diagonally forward.
the person walks downstairs, gripping railing with their left hand, and stops at the bottom of the stairs.
quarterback will throw a footboll another team member in hopes to score a touchdown.
person grubbed something with right hand and put somewhere
man kicks something with right foot.
person got down and is crawling across the floor.
a person walks on beum,
person is performing dance and spinning around
person is walking drunk circle clockwise
loses his balance and fall off to his right.
Figure 4: Additional text-to-motion generation results HGM Darker colors indicate later timestamps.
16

---

Published as a conference paper at ICLR 2025
person walks forward, pushed bockward. and then resumes walking forward.
walks backwards steps up stairs backwards as well.
the person trying hit bug.
walking forword and kioking
man steps forward handstand.
a persen jumps forward
Figure 5: Heatmaps of the output of Gagan for various motion sequences. The reconstruction loss normalized range of O to with darker colors indicating regions predicted be more challenging to reconstruct.
17

---

Published as a conference paper at ICLR 2025
D ADDITIONAL ABLATION STUDY
An ablation study is conducted to analyze the impact of varying the number of iterations at each stage during the hierarchical inference process according different overall iteration counts L and transition points LM and LA- The experiments are performed on HumanML3D, and are reported in Tab. 2. Each condition C()) for the /-th iteration provided as:
if S S LM if LM < <ISTA
C(I) = ccm:ca C9) 1CTH Ca C*]
(2)
LA <ISL
For the initial setup, fixed to 10 and evaluated the performance under different conditions at each stage. When only single stage is used for inference, providing conditions at the action or specific details improved performance compared to using motion-level condition alone, but the results are still lower than those achieved with hierarchical approach. In the hierarchical setting, performance improvement is observed when more iterations allocated the final stage compared to the initial stage. This suggests that while all condition types contain motion-level information, specific details can only effectively utilized in the final stage. As result, placing more emphasis on the final stage allows the model to better integrate coarse motion structure with fine- fine-grained details, resulting in a more coherent generation. Additionally, we extended the experiments by increasing L to 15 and 20, but found that further increasing the total number of iterations did not significantly impact the model's performance.
Table 2: Ablation study on the number of iterations during inference.
of iterations
R-Precision Top-1 f
Total L
Motion Ls
Action (LA- LM) Specific (L LA)
10
0.522
0040=0003
10
0.525#.000
0.046
10
0
10
0.527
0.043
10
3
2
0.532
0.038 40000
3
0.535 t. 002
0.036:.002
0.535 t: 008
0.037
20
6
10
0.534
0.036
18


Reviews:
[{"review_id": "7NZTKORmIu", "reviewer": "Reviewer_RKiq", "summary": "This work suggests a text-to-motion generation model, using the combination of three existing techniques:\n1. Masked VQ-VAE in the motion domain:\n    1. Hierarchical (MoMask by Guo et al., 2024)\n    2. Masking based on confidence (Pinyoanuntapong et al. 2024b)\n2. Hard token mining (HTM) in the imaging domain (Wang et al., 2023a)\n3. hierarchical semantic graph representation in the language domain (Shi & Lin, 2019) + Graph Attention Network (GAT) (Velickovic et al., 2018)\n\n------------------------------------\n**Post rebuttal comment:**\n\nFollowing the rebuttal, I am raising my score toward acceptance.", "questions": "Questions and Comments:\n\n- Masked and residual transformers: Do they predict all tokens together (i.e., predict $\\mathcal{M}$ tokens for the masked transformer and $n$ tokens for the residual one)? Or, are they causal? (by causal I mean that for the residual transformer, predict tokens according to their temporal order where each prediction is conditioned on the previously predicted tokens; for the masked transformer it probably means conditioning masked tokens on those already predicted). If the prediction is causal, then a prediction of an $n$ length motion requires an n-step loop, hence each iteration in Sec. 3.4 has an internal loop within it. Please explain.\n- L209.5 Eq. 4): Rephrase. it seems you wanted to describe k and M, but described M only.\n- L220 (Eq. 5):  The L_pred objective focuses on ranking losses by relative magnitude rather than exact values. Therefore, $\\hat{\\ell}$ (and G) should be interpreted as predicting the values with the topological ordering of token losses, not as estimating the precise reconstruction loss. \n- L249: \"At each step\": do you mean \"at each epoch\"? The index k is related to epochs (L242), not to steps.\n- L 249: Does it mean $\\mathcal{M} = \\gamma(\\tau_k)\\cdot n$ ? I don't see anywhere in the paper how $\\mathcal{M}$ is defined.\n- L339: where is the probability distribution taken from?\n- Sec. A.2, A.3: Fig. 2 depicts the residual transformer in one block, in blue Fig. 3 depicts the same blue block multiple times, but relates to it as sub-blocks of the residual transformer from Fig. 2. This is misleading. Please clarify the differences and similarities between the sub-blocks and make their notation more consistent.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Impressive and thoughtful choice of SOTA works.\n- Clear writing (mostly).\n- Reproducibility: implementation details are given and, more importantly, the authors plan to release their full code and setup.", "weaknesses": "**Major weaknesses**\n\n- Novelty: This work combines existing SOTA works (see \"Summary\" above). While this work demonstrates solid engineering execution in integrating existing techniques, its novel contribution to the field is questionable. Please discuss any new theoretical insights or algorithmic innovations beyond the integration of existing techniques. \n- Technical soundness: How is the loss of the residual transformer (L_res) incorporated into the overall network loss? Can you clarify the summation range in Eq. 1 of Sec. A.2? I believe it should be from i=1 to V. Could you provide a diagram showing the architectural flow between the masked and residual transformers?\"\n- Qualitative results: To fully assess the quality and naturalness of the generated motions, I recommend including a supplementary video. While the paper includes qualitative figures, they cannot be validated for dynamic motion artifacts such as jitteriness and foot sliding.\n- Quantitative results: For most metrics, results are only marginally better and sometimes marginally worse. I would call that comparable. There is, however, a notable improvement for HML3D FID. \n\n**More weaknesses**\n\n- Some technical descriptions need more clarification or need to be corrected (see \"Questions\" below).\n- I suggest concatenating the supp to the main paper to allow mutual references and better reader experience. This is allowed in ICLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sEtfTUKMLW", "reviewer": "Reviewer_z3vH", "summary": "The authors investigate the hard token learning challenge within MoMask's native training scheme and enhance the model's text comprehension by integrating hierarchical textual conditions. Experiments and ablations demonstrate that this implementation improves MoMask's performance.", "questions": "Could Figure 4 show more examples and include **heatmaps of the distribution of the predicted reconstruction loss**, like in Hard Patches Mining (Wang et al., CVPR 23)?\n\nMissing cites:\n- MotionGPT: Human Motion as a Foreign Language\n- MotionLCM: Real-time Controllable Motion Generation via Latent Consistency Model\n- StableMoFusion: Towards Robust and Efficient Diffusion-based Motion Generation Framework\n- Mofusion: A framework for denoising-diffusion-based motion synthesis", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 1, "strengths": "This paper has the following strengths:\n\n- Introduce the Hard Token Mining (HTM) into the motion generation task for the first time and prove its effectiveness.\n- Design a Hierarchical Generative Masked Motion Model (HGM$^3$) and use the text conditions with different granularity to enhance the text-motion matching performance.\n- Qualitative and quantitative results demonstrate the effectiveness of the proposed method.", "weaknesses": "For weaknesses, I have the following comments:\n\n- From an innovation perspective, it’s quite limited. The overall pipeline of the paper is essentially a **replica** of Hard Patches Mining (Wang et al., CVPR 23), merely validated for effectiveness in motion generation.\n- As for the proposed Hierarchical Text Graph, while providing more textual information obviously enhances the model's text-motion alignment, a more straightforward approach like using CLIP token-level features would likely be more effective and simpler than this **complex** method (parsing, CLIP, graph, etc.). If the performance of using CLIP token-level features is inferior to the Hierarchical Text Graph, please provide a comparative experiment to demonstrate this.\n-  This paper is a well-executed A+B **technical report** with clear and complete experiments, but it offers limited practical value for real-world applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sWX2eSpIgX", "reviewer": "Reviewer_oLX4", "summary": "This paper works on the text-to-motion task and proposes two modules to specifically enhance the generative masked modeling-based motion generation. The first contribution is to apply a hard-mining training strategy to replace random masking when training the masked transformer, which facilitates the model to better generate the hard motions. Another contribution is using a graph-based hierarchical text embedding to replace the CLIP text embedding, which provides a better semantic understanding and leads to improved generation quality. Experiments show that the proposed two modules lead to better quantitative and qualitative results compared to the baselines.", "questions": "1. The citations in the text of the submitted PDF do not contain clickable links, which makes it hard to match the in-context citation with the reference works. This is not a factor for grading but can make the life of reviewers easier.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper proposes a hard-token mining training strategy that uses additional networks to predict masks of the hard-to-predict tokens. Compared to the random masking strategy used in previous generative masked modeling-based motion generation works, this hard-mining strategy effectively facilitates the models to learn the hard motions.\n\n2. The proposed hierarchical graph-based semantic embedding provides an enhanced text embedding compared to the CLIP text encoder embedding used in previous works. The proposed hierarchical semantic embedding well captures both the fin-grained and global semantics, leading to improved generation results.\n\n3. Experiments show that the proposed two components can effectively enhance the generation quality of generative masked modeling-based methods, and outperforms the baselines both qualitatively and quantitatively.", "weaknesses": "1. This paper proposes two components that can specifically enhance generative masked modeling-based motion generation. Although the proposed components are effective, there is not much new knowledge from this paper.  Hard mining is used in the cited reference Hard Patch Mining (HPM) (Wang et al., 2023a) for the image domain and it employs a teacher-student model that transitions from completely random masking to focusing on difficult image patches, in the same spirit as this paper. The hierarchical graph-based embedding is proposed in the reference Act As You Wish, Jin et al. (2024) and proved effective for diffusion-based text-to-motion. This paper applies these two methods to the niche problem of masked modeling-based text-to-motion and achieves enhanced performance.\n\n2. As a motion generation work, this submission does not include any video or animation visualization of motion results. I would strongly suggest including some video results, at least for the sequences presented in Figures 3, 4, and Appendix C.\n\n3. Some technical details regarding text conditioning require further explanation. Line 205 states that the text embedding is from CLIP,  but from other sections I infer that it should be the proposed hierarchical semantic embedding. Moreover, Appendix Figure 2 indicates that the residual transformer uses the text embedding directly from CLIP instead of the hierarchical embedding. What is the reason for not using the hierarchical semantic embedding? I would appreciate additional explanations of the text conditioning used in this work.\n\nI'm on the borderline with this paper given the currently presented manuscript.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7BjDuaOBxm", "reviewer": "Reviewer_3ms7", "summary": "The paper introduces a novel framework with two main components: Hard Token Mining (HTM) and a Hierarchical Generative Masked Motion Model (HGM3). HTM focuses on difficult-to-learn motion areas, while HGM3 represents sentences with different levels of granularity, enabling contextually accurate motion generation by decomposing motion descriptions into hierarchical semantic graphs with three levels: motions, actions, and specifics. This global-to-local structure enables a deep understanding of motion descriptions and provides fine-grained control over motion generation. Experiments show that the propose method out perform state of the art methods for both HumanML3D and KIT dataset.", "questions": "- HTM?\n  - What is the output of HTM? What is the different between G(student) and F(student) models?\n  - What is the motivation of student-teacher architecture? To prevent model collapse? If yes, I think it's better to have ablation study on this.\n- $HGM^3$\n  - How the integration of Graph Reasoning to masked model is different from GraphMotion?\n  - Does Motion-Action-Specific embeddings applied to Residual layers?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- This work is the first to integrate hierarchical semantic graphs into a masked motion model, offering an innovative approach in this area.\n- The literature review effectively outlines the background on text-to-motion, masked motion models, and the challenges in existing methods.\n- Comprehensive experiments with detailed comparisons.\n- The experiments demonstrate that the proposed method achieves state-of-the-art (SoTA) performance across multiple datasets.", "weaknesses": "- It is a bit confusing that $HGM^3$ is used both as the name of a component and as the name of the whole model ($HGM^3 + HTM$).\n- $HGM^3$ component:\n  - Since the main idea of $HGM^3$ is inspired by GraphMotion [1], it would strengthen the contribution if the authors could demonstrate the differences in applying GAT to a masked model.\n  - The $HGM^3$ component is not well described. It is unclear how 1) motions, 2) actions, and 3) specific elements work within the Graph Attention Network.\n  - There are no details provided on the \"twelve types of edges representing various relationships between nodes\" mentioned in Section 3.3.\n- The visualization in Figure 4 is unclear. A few more samples (or a video supplement, if easier) may help clarify the results.\n- It is unclear if the model truly requires a student-teacher architecture. (I don’t fully understand this part—I’ve also addressed this in the questions section.)\n\n\n[1] Peng Jin, Yang Wu, Yanbo Fan, Zhongqian Sun, Wei Yang, and Li Yuan. Act as you wish: Finegrained control of motion diffusion model with hierarchical semantic graphs. Neural Information\nProcessing Systems, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)