HGM³: Hierarchical Generative Masked Motion Modeling with Hard Token Mining

Authors: Minjae Jeong, Yechan Hwang, Jaejin Lee, Sungyoon Jung, Won Hwa Kim
Primary area: generative models
Keywords: text-to-motion generation, generative masked model, hard token mining, hierarchical semantic graph

Abstract:
Text-to-motion generation has significant potential in a wide range of applications including animation, robotics, and AR/VR. While recent works on masked motion models are promising, the task remains challenging due to the inherent ambiguity in text and the complexity of human motion dynamics. To overcome the issues, we propose a novel text-to-motion generation framework that integrates two key components: Hard Token Mining (HTM) and a Hierarchical Generative Masked Motion Model (HGM³). Our HTM identifies and masks challenging regions in motion sequences and directs the model to focus on hard-to-learn components for efficacy. Concurrently, the hierarchical model uses a semantic graph to represent sentences at different granularity, allowing the model to learn contextually feasible motions. By leveraging a shared-weight masked motion model, it reconstructs the same sequence under different conditioning levels and facilitates comprehensive learning of complex motion patterns. During inference, the model progressively generates motions by incrementally building up coarse-to-fine details. Extensive experiments on benchmark datasets, including HumanML3D and KIT-ML, demonstrate that our method outperforms existing methods in both qualitative and quantitative measures for generating context-aware motions.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730040207447

thread_type:
official_review

<thread_object>
creator: reviewer_3ms7

<content>
summary:
The paper introduces a novel framework with two main components: Hard Token Mining (HTM) and a Hierarchical Generative Masked Motion Model (HGM3). HTM focuses on difficult-to-learn motion areas, while HGM3 represents sentences with different levels of granularity, enabling contextually accurate motion generation by decomposing motion descriptions into hierarchical semantic graphs with three levels: motions, actions, and specifics. This global-to-local structure enables a deep understanding of motion descriptions and provides fine-grained control over motion generation. Experiments show that the propose method out perform state of the art methods for both HumanML3D and KIT dataset.

strengths:
- This work is the first to integrate hierarchical semantic graphs into a masked motion model, offering an innovative approach in this area.
- The literature review effectively outlines the background on text-to-motion, masked motion models, and the challenges in existing methods.
- Comprehensive experiments with detailed comparisons.
- The experiments demonstrate that the proposed method achieves state-of-the-art (SoTA) performance across multiple datasets.

weaknesses:
- It is a bit confusing that $HGM^3$ is used both as the name of a component and as the name of the whole model ($HGM^3 + HTM$).
- $HGM^3$ component:
  - Since the main idea of $HGM^3$ is inspired by GraphMotion [1], it would strengthen the contribution if the authors could demonstrate the differences in applying GAT to a masked model.
  - The $HGM^3$ component is not well described. It is unclear how 1) motions, 2) actions, and 3) specific elements work within the Graph Attention Network.
  - There are no details provided on the "twelve types of edges representing various relationships between nodes" mentioned in Section 3.3.
- The visualization in Figure 4 is unclear. A few more samples (or a video supplement, if easier) may help clarify the results.
- It is unclear if the model truly requires a student-teacher architecture. (I don’t fully understand this part—I’ve also addressed this in the questions section.)


[1] Peng Jin, Yang Wu, Yanbo Fan, Zhongqian Sun, Wei Yang, and Li Yuan. Act as you wish: Finegrained control of motion diffusion model with hierarchical semantic graphs. Neural Information
Processing Systems, 2024.

questions:
- HTM?
  - What is the output of HTM? What is the different between G(student) and F(student) models?
  - What is the motivation of student-teacher architecture? To prevent model collapse? If yes, I think it's better to have ablation study on this.
- $HGM^3$
  - How the integration of Graph Reasoning to masked model is different from GraphMotion?
  - Does Motion-Action-Specific embeddings applied to Residual layers?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730411382303

thread_type:
official_review

<thread_object>
creator: reviewer_olx4

<content>
summary:
This paper works on the text-to-motion task and proposes two modules to specifically enhance the generative masked modeling-based motion generation. The first contribution is to apply a hard-mining training strategy to replace random masking when training the masked transformer, which facilitates the model to better generate the hard motions. Another contribution is using a graph-based hierarchical text embedding to replace the CLIP text embedding, which provides a better semantic understanding and leads to improved generation quality. Experiments show that the proposed two modules lead to better quantitative and qualitative results compared to the baselines.

strengths:
1. This paper proposes a hard-token mining training strategy that uses additional networks to predict masks of the hard-to-predict tokens. Compared to the random masking strategy used in previous generative masked modeling-based motion generation works, this hard-mining strategy effectively facilitates the models to learn the hard motions.

2. The proposed hierarchical graph-based semantic embedding provides an enhanced text embedding compared to the CLIP text encoder embedding used in previous works. The proposed hierarchical semantic embedding well captures both the fin-grained and global semantics, leading to improved generation results.

3. Experiments show that the proposed two components can effectively enhance the generation quality of generative masked modeling-based methods, and outperforms the baselines both qualitatively and quantitatively.

weaknesses:
1. This paper proposes two components that can specifically enhance generative masked modeling-based motion generation. Although the proposed components are effective, there is not much new knowledge from this paper.  Hard mining is used in the cited reference Hard Patch Mining (HPM) (Wang et al., 2023a) for the image domain and it employs a teacher-student model that transitions from completely random masking to focusing on difficult image patches, in the same spirit as this paper. The hierarchical graph-based embedding is proposed in the reference Act As You Wish, Jin et al. (2024) and proved effective for diffusion-based text-to-motion. This paper applies these two methods to the niche problem of masked modeling-based text-to-motion and achieves enhanced performance.

2. As a motion generation work, this submission does not include any video or animation visualization of motion results. I would strongly suggest including some video results, at least for the sequences presented in Figures 3, 4, and Appendix C.

3. Some technical details regarding text conditioning require further explanation. Line 205 states that the text embedding is from CLIP,  but from other sections I infer that it should be the proposed hierarchical semantic embedding. Moreover, Appendix Figure 2 indicates that the residual transformer uses the text embedding directly from CLIP instead of the hierarchical embedding. What is the reason for not using the hierarchical semantic embedding? I would appreciate additional explanations of the text conditioning used in this work.

I'm on the borderline with this paper given the currently presented manuscript.

questions:
1. The citations in the text of the submitted PDF do not contain clickable links, which makes it hard to match the in-context citation with the reference works. This is not a factor for grading but can make the life of reviewers easier.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730463866127

thread_type:
official_review

<thread_object>
creator: reviewer_z3vh

<content>
summary:
The authors investigate the hard token learning challenge within MoMask's native training scheme and enhance the model's text comprehension by integrating hierarchical textual conditions. Experiments and ablations demonstrate that this implementation improves MoMask's performance.

strengths:
This paper has the following strengths:

- Introduce the Hard Token Mining (HTM) into the motion generation task for the first time and prove its effectiveness.
- Design a Hierarchical Generative Masked Motion Model (HGM$^3$) and use the text conditions with different granularity to enhance the text-motion matching performance.
- Qualitative and quantitative results demonstrate the effectiveness of the proposed method.

weaknesses:
For weaknesses, I have the following comments:

- From an innovation perspective, it’s quite limited. The overall pipeline of the paper is essentially a **replica** of Hard Patches Mining (Wang et al., CVPR 23), merely validated for effectiveness in motion generation.
- As for the proposed Hierarchical Text Graph, while providing more textual information obviously enhances the model's text-motion alignment, a more straightforward approach like using CLIP token-level features would likely be more effective and simpler than this **complex** method (parsing, CLIP, graph, etc.). If the performance of using CLIP token-level features is inferior to the Hierarchical Text Graph, please provide a comparative experiment to demonstrate this.
-  This paper is a well-executed A+B **technical report** with clear and complete experiments, but it offers limited practical value for real-world applications.

questions:
Could Figure 4 show more examples and include **heatmaps of the distribution of the predicted reconstruction loss**, like in Hard Patches Mining (Wang et al., CVPR 23)?

Missing cites:
- MotionGPT: Human Motion as a Foreign Language
- MotionLCM: Real-time Controllable Motion Generation via Latent Consistency Model
- StableMoFusion: Towards Robust and Efficient Diffusion-based Motion Generation Framework
- Mofusion: A framework for denoising-diffusion-based motion synthesis
</content>
</thread_object>

</thread>
<thread>
creation date:
1730624533094

thread_type:
official_review

<thread_object>
creator: reviewer_rkiq

<content>
summary:
This work suggests a text-to-motion generation model, using the combination of three existing techniques:
1. Masked VQ-VAE in the motion domain:
    1. Hierarchical (MoMask by Guo et al., 2024)
    2. Masking based on confidence (Pinyoanuntapong et al. 2024b)
2. Hard token mining (HTM) in the imaging domain (Wang et al., 2023a)
3. hierarchical semantic graph representation in the language domain (Shi & Lin, 2019) + Graph Attention Network (GAT) (Velickovic et al., 2018)

------------------------------------
**Post rebuttal comment:**

Following the rebuttal, I am raising my score toward acceptance.

strengths:
- Impressive and thoughtful choice of SOTA works.
- Clear writing (mostly).
- Reproducibility: implementation details are given and, more importantly, the authors plan to release their full code and setup.

weaknesses:
**Major weaknesses**

- Novelty: This work combines existing SOTA works (see "Summary" above). While this work demonstrates solid engineering execution in integrating existing techniques, its novel contribution to the field is questionable. Please discuss any new theoretical insights or algorithmic innovations beyond the integration of existing techniques. 
- Technical soundness: How is the loss of the residual transformer (L_res) incorporated into the overall network loss? Can you clarify the summation range in Eq. 1 of Sec. A.2? I believe it should be from i=1 to V. Could you provide a diagram showing the architectural flow between the masked and residual transformers?"
- Qualitative results: To fully assess the quality and naturalness of the generated motions, I recommend including a supplementary video. While the paper includes qualitative figures, they cannot be validated for dynamic motion artifacts such as jitteriness and foot sliding.
- Quantitative results: For most metrics, results are only marginally better and sometimes marginally worse. I would call that comparable. There is, however, a notable improvement for HML3D FID. 

**More weaknesses**

- Some technical descriptions need more clarification or need to be corrected (see "Questions" below).
- I suggest concatenating the supp to the main paper to allow mutual references and better reader experience. This is allowed in ICLR.

questions:
Questions and Comments:

- Masked and residual transformers: Do they predict all tokens together (i.e., predict $\mathcal{M}$ tokens for the masked transformer and $n$ tokens for the residual one)? Or, are they causal? (by causal I mean that for the residual transformer, predict tokens according to their temporal order where each prediction is conditioned on the previously predicted tokens; for the masked transformer it probably means conditioning masked tokens on those already predicted). If the prediction is causal, then a prediction of an $n$ length motion requires an n-step loop, hence each iteration in Sec. 3.4 has an internal loop within it. Please explain.
- L209.5 Eq. 4): Rephrase. it seems you wanted to describe k and M, but described M only.
- L220 (Eq. 5):  The L_pred objective focuses on ranking losses by relative magnitude rather than exact values. Therefore, $\hat{\ell}$ (and G) should be interpreted as predicting the values with the topological ordering of token losses, not as estimating the precise reconstruction loss. 
- L249: "At each step": do you mean "at each epoch"? The index k is related to epochs (L242), not to steps.
- L 249: Does it mean $\mathcal{M} = \gamma(\tau_k)\cdot n$ ? I don't see anywhere in the paper how $\mathcal{M}$ is defined.
- L339: where is the probability distribution taken from?
- Sec. A.2, A.3: Fig. 2 depicts the residual transformer in one block, in blue Fig. 3 depicts the same blue block multiple times, but relates to it as sub-blocks of the residual transformer from Fig. 2. This is misleading. Please clarify the differences and similarities between the sub-blocks and make their notation more consistent.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732034112399

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to all Reviewers

comment:
Dear Reviewers,

We sincerely thank the reviewers for their thoughtful and constructive feedback. We appreciate the recognition of our contributions, practical value, and experimental rigor. Notably, the reviewers highlighted:

**Innovative Approach:**
- *“...the first to integrate hierarchical semantic graphs into a masked motion model, offering an innovative approach in this area.”* (**Reviewer 3ms7**)
- *“Introduce the Hard Token Mining (HTM) into the motion generation task for the first time and prove its effectiveness.”* (**Reviewer z3vH**)

**Strong Empirical Performance and Comprehensive Experiments:**
- *“...outperforms the baselines both qualitatively and quantitatively.”* (**Reviewer oLX4**)
- *“...state-of-the-art (SoTA) performance across multiple datasets.”* (**Reviewer 3ms7**)
- *“Qualitative and quantitative results demonstrate the effectiveness of the proposed method.”* (**Reviewer z3vH**)
- *“Comprehensive experiments with detailed comparisons.”* (**Reviewer 3ms7**)

While we are grateful for the positive feedback, we also seriously take the reviewers’ concerns especially on the novelty of our work. Below, we provide a detailed response to address these points, and revised texts are given in red in the newly attached manuscript.

We truly hope that our rebuttal clarifies all the critical issues raised by the reviewers and helps the reviewers with a better understanding of our work, possibly leading to higher chances for the acceptance of this paper. We are open for any discussion during the rebuttal period.

---

### Clarifying Novelty: addressing Reviewers’ common concerns

**Hard Token Mining (HTM) vs. Hard Patch Mining (HPM) (Wang et al., 2023a)**

While HTM draws inspiration from the targeted masking approach of HPM, it has been fundamentally redesigned to address the unique challenges of motion data. The key differences are as follows:

1) **Purpose and Architecture**

In HPM, the primary goal is to learn novel representation by training an encoder-decoder architecture to produce mask-invariant features from the encoder with two separate decoders (i.e., reconstructor and loss predictor). In contrast, our HTM aims to enhance reconstruction quality rather than representation learning. By focusing on improving the masking strategy for motion sequence reconstruction, HTM eliminates the need to train an encoder for representation extraction. Instead, it consists of two components with distinct purposes:
- The **HTM Transformer (Loss Predictor)**: Predicts reconstruction difficulty for each token in an unmasked motion sequence, guiding the masking strategy.
- The **Masked Transformer (Reconstructor)**: Reconstructs the masked tokens, concentrating on the most difficult segments identified by the loss predictor.

Unlike HPM’s emphasis on representation learning, HTM focuses entirely on improving reconstruction quality by selectively masking complex dynamics in motion data.

2) **Input Consistency in the Student-Teacher Model**

In HPM, the loss predictors in the teacher and student models operate on different inputs, resulting in input inconsistency:
- During training, the student model’s loss predictor learns from the encoded representation of masked images.
- The teacher model, however, uses the encoded representation of unmasked images to determine which patches to mask.

This can lead to suboptimal masking guidance, as the teacher's decisions are based on unmasked inputs that differ from the student's context.

However, HTM eliminates this issue by using the same unmasked token sequences for both teacher and student models of the HTM transformer, ensuring consistent masking guidance throughout the process. By maintaining consistent inputs, HTM simplifies its design and enables a more robust and reliable masking strategy.


**HGM³: Novel Contributions Beyond Existing Hierarchical Approaches**

HGM³ builds on hierarchical semantic graph reasoning but introduces key innovations that improve text-to-motion generation:

1) **Unified Weight Sharing Across Hierarchical Levels**

HGM³ employs a single model with shared weights across all hierarchical levels (motion, action, specifics):
- This significantly reduces the number of parameters, making the model faster and more memory-efficient.
- Weight sharing ensures that the model learns consistent, generalizable features across levels, enhancing coherence in the generated motions. Without shared weights, models often develop disjointed representations, leading to inconsistencies.

2) **Single Latent Space for Hierarchical Conditioning**

While GraphMotion (Jin et al., 2024) maintains separate latent spaces for different semantic levels, HGM³ unifies these into a single latent space. This simplifies the conditioning process, reducing computational overhead while preserving fine-grained semantic representation.

We believe these tailored innovations not only demonstrate originality but also significantly contribute to the field of motion generation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732103034520

thread_type:
official_comment

<thread_object>
creator: reviewer_rkiq

<content>
title:
Some supplementary video files cannot be opened

comment:
Dear authors,

Thank you for adding new supplementary video files.

Some of them cannot be opened. 

Can you kindly re-upload them?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This is a borderline paper. It exhibits strengths in writing, methodology, and experimental performance. However, there are still significant concerns regarding its relatively modest technical contributions and innovations, which is pointed out by all reviewers. The proposed method builds upon several existing techniques—for instance, hard token mining extends hard patch mining (Wang et al., 2023a), and the hierarchical graph representation is an extension of GraphMotion (Jin et al., 2024). Although the authors have expanded these modules and implemented detailed modifications to better address the problems presented, which is explained in their rebuttal, the overall technical contributions still appear marginal.

The Area Chair shares the reviewers' concerns about the paper's novelty. Nevertheless, considering its strengths in writing, methodology (its well-designed methodology that effectively extends and integrates existing methods), and comprehensive experiments, this paper falls within the borderline acceptance range but does not merit a clear acceptance.