InterMask: 3D Human Interaction Generation via Collaborative Masked Modeling

Authors: Muhammad Gohar Javed, chuan guo, Li cheng, Xingyu Li
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Motion Synthesis, Human Interaction Generation, Masked Generative Transformer, Text-driven Generation, Vector Quantized VAE

Abstract:
Generating realistic 3D human-human interactions from textual descriptions remains a challenging task. Existing approaches, typically based on diffusion models, often produce results lacking realism and fidelity. In this work, we introduce *InterMask*, a novel framework for generating human interactions using collaborative masked modeling in discrete space. InterMask first employs a VQ-VAE to transform each motion sequence into a 2D discrete motion token map. Unlike traditional 1D VQ token maps, it better preserves fine-grained spatio-temporal details and promotes *spatial awareness* within each token. Building on this representation, InterMask utilizes a generative masked modeling framework to collaboratively model the tokens of two interacting individuals. This is achieved by employing a transformer architecture specifically designed to capture complex spatio-temporal inter-dependencies. During training, it randomly masks the motion tokens of both individuals and learns to predict them. For inference, starting from fully masked sequences, it progressively fills in the tokens for both individuals. With its enhanced motion representation, dedicated architecture, and effective learning strategy, InterMask achieves state-of-the-art results, producing high-fidelity and diverse human interactions. It outperforms previous methods, achieving an FID of $5.154$ (vs $5.535$ of in2IN) on the InterHuman dataset and $0.399$ (vs $5.207$ of InterGen) on the InterX dataset. Additionally, InterMask seamlessly supports reaction generation without the need for model redesign or fine-tuning.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730557475928

thread_type:
official_review

<thread_object>
creator: reviewer_yc4h

<content>
summary:
This paper addresses the problem of generating human-human interaction from textual descriptions. The authors introduce InterMask, a novel framework that employs 2D VQ-VAE to encode interactions into a discrete space and generative masked modeling to capture complex spatio-temporal interdependencies.

strengths:
1. This paper introduces a novel 2D VQ-VAE that quantizes motion into a 2D discrete map, which can more effectively preserve fine-grained spatio-temporal details than the 1D version.

2. The proposed method achieves SOTA performance on the InterHuman and InterX benchmarks.

weaknesses:
1. The method does not appear substantially different from prior work. Overall, the paper feels somewhat underwhelming, lacking distinctive insights, though there are no major issues with the experiments or writing. I would rate it as borderline.

2. The proposed approach faces challenges in generating human reactions in an online manner (although this isn’t the primary focus of the current setting). This capability would be more practical for some virtual reality applications.

questions:
I have no other questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730653434047

thread_type:
official_review

<thread_object>
creator: reviewer_lgj2

<content>
summary:
The paper presents InterMask, a framework for generating 3D human interactions using masked modeling in discrete space. By encoding motion as 2D token maps and modeling intra- and inter-person dependencies with a custom transformer, InterMask achieves impressive results on the InterHuman and InterX datasets. The model shows strong performance in terms of realism and alignment with text prompts, outperforming existing methods in controlled settings.

strengths:
1. The use of 2D token maps and spatial-temporal attention layers in the Inter-M transformer enables more nuanced modeling of human interactions, demonstrating improved fidelity in motion generation.

2. InterMask achieves state-of-the-art results on key metrics in standard datasets, indicating its effectiveness in generating realistic, text-aligned interactions.

3. The model is relatively lightweight and demonstrates reduced inference time.

weaknesses:
1. The paper’s innovative contribution is not well-defined. Although the authors introduce a transformer module to model intra- and inter-person spatial-temporal dependencies, this approach is widely used in various generative tasks. The ablation studies, which compare only with/without this module, do not sufficiently demonstrate the model’s unique contribution. If this module is intended as a core innovation, the authors should focus on explaining why its internal structure, arrangement, or sequence specifically suits the 3D interaction modeling problem. Merely showing the presence or absence of the module does not justify its relevance, as it is a fairly standard approach.

2. The process of converting each motion sequence into a 2D discrete motion token map lacks detailed description. The main innovation of this paper appears to lie in the combination of the 2D discrete motion token map with MAE; however, there is insufficient explanation of how the 2D token map is constructed. Providing a more thorough description, ideally with a detailed diagram, would enhance clarity and better communicate this aspect of the method.

3. User Study. The user study includes only three participants, which is insufficient to draw robust conclusions on user preference. A larger sample size and detailed descriptions of the study’s methodology would add reliability and provide stronger validation for claims of user preference for InterMask-generated interactions.

4. Upon reviewing the supplementary videos, the generated motions appear to lack the fluidity seen in InterGen’s outputs. A discussion on this aspect would be valuable, particularly to understand any limitations in InterMask’s approach to generating smooth, natural motion.

questions:
1. Could the authors comment on the suitability of the current framework for multi-person interactions? Given the focus on two-person setups, it would be useful to understand any potential modifications or challenges in adapting this approach for more complex, multi-person interaction scenarios.

2. How does the model handle longer, more complex instructions or instructions that are less structured (in-the-wild)? 

3. Do the authors see potential for combining InterMask with language models to improve interpretability or to handle a wider range of textual prompts?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730675413070

thread_type:
official_review

<thread_object>
creator: reviewer_js9r

<content>
summary:
This article introduces a new framework called InterMask for generating 3D human interaction actions from text descriptions. The proposed framework appears to be based on the T2M-GPT framework, modeling in discrete space using generative masks, while also incorporating the concept of interhand to collaborate within the mask modeling framework by utilizing tokens from two interacting individuals. During the inference process, it starts with a fully masked sequence and gradually fills in the tokens for both individuals.

strengths:
1. A Transformer structure combined with VQ-VAE has been proposed for generating 3D human interaction actions. 
2. It utilizes a 2D discrete motion token map to represent motion, distinguishing itself from traditional 1D representation methods. This representation better preserves spatiotemporal details and spatial awareness, which is crucial for accurately capturing complex motion relationships in human interactions. 
3. The framework has achieved state-of-the-art results on the InterHuman and InterX datasets, performing exceptionally well on various metrics. 
4. The average inference time is only 0.77 seconds, significantly lower than InterGen's 1.63 seconds, providing a considerable advantage in practical applications by enabling faster result generation.

weaknesses:
1. In the visualization results, issues such as body penetration between interacting individuals and unnatural sliding movements still occur.
2. The article also mentions the masking strategy, but I believe its effectiveness may be influenced by various factors, such as the setting of the mask ratio and the dynamic adjustment of masks at different stages. The article does not delve deeply into how these factors affect model performance or how to optimally set these parameters.

questions:
1. I noticed that in the second video of the visualization "Everyday Actions," the character on the right makes a sudden arm-raising motion, which seems somewhat disconnected. Additionally, is there a 10-second video available for reference? I saw that the authors mentioned the ability to generate interactive results up to 10 seconds long.

2. In terms of quantitative experiments, it seems there is no comparison with PriorMDM, which also produces results for two-person interactions. Similarly, there doesn't appear to be a comparison with other VQ-VAE methods like T2M-GPT. I'm curious about their results, especially since they perform very well on single-person datasets. If T2M-GPT were given a reference action sequence for one individual as an additional condition, how would it perform in generating interactive individuals, particularly on interactive datasets?

3. Regarding the Inter-M Transformer structure, although there is some evidence in the ablation studies concerning the relative importance of different attention mechanisms, it may not fully clarify their specific roles and trade-offs in different interaction scenarios and data distributions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732247437485

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All Reviewers of Paper 12356

comment:
We thank the reviewers for their valuable feedback and for highlighting both the strengths and areas of improvement in our work. We greatly appreciate the encouraging comments about our framework, particularly regarding the **novel generative masked modeling framework for human interactions** (Reviewer jS9r), the **2D motion token map** and its ability to better preserve spatio-temporal details in the discrete representation (Reviewer jS9r, LgJ2, YC4H), **state-of-the-art results on interaction generation** (Reviewer jS9r, LgJ2, YC4H), and the **lower inference time and relatively lightweight architecture** (Reviewer jS9r, LgJ2).  We have individually addressed each reviewer’s concerns, providing detailed responses and clarifications. Here, we provide a general response to summarize the key points raised and the changes made to the paper and supplementary materials as a consequence.  

We would like to re-emphasize the novelty and contributions of this work: 
1. **InterMask Framework**: To the best of our knowledge, we are the first to introduce generative masked modeling for human interactions in the discrete space, which provides valuable insights into this framework regarding its effectiveness, efficiency and flexibility, and sets a new direction for the field.  
2. **2D discrete motion token map**: A unique 2d discrete motion representation that better captures spatio-temporal details than previous 1D token maps.  
3. **Inter-M Transformer**: While not novel in architecture design, it features custom attention modules and masking techniques that are shown to be crucial for learning collaborative modeling of interactions and intra- and inter-person spatio-temporal dependencies using 2D motion tokens.  
4. All the above together results in a strong, flexible and lightweight generative framework that achieves state-of-the-art results on the human interaction generation task, with a lower inference time compared to previous state-of-the-art methods. Moreover, it supports the reaction generation task seamlessly and could potentially extend to multi-person interactions with minimal modifications to only one module.  

We have sincerely taken into consideration the comments and suggestions from reviewers and have incorporated the following changes in our revised manuscript.  
1. Added a section on the 2D motion token map construction as Appendix J, including a detailed figure and explanation of the 2D encoder architecture and quantization process, as suggested by reviewer **LgJ2**.  
2. Rectified confusion about user study participants in Section 4.1 of the main text and Appendix I in the supplementary text, which was raised by reviewer **LgJ2**.
3. Added a section on the two-stage masking technique used during training as Appendix K, with a detailed figure, explanation, and the experiment determining $p_r$, as suggested by reviewer **jS9r**.
4. Included the value of the random masking to interaction masking ratio ($p_r$) in Table 5, Appendix C.2.  
5. Added more visual samples for ablations on the attention modules in Inter-M Transformer to Appendix F, to address concerns raised by reviewers  **jS9r**, **LgJ2** and **YC4H**.
6. Changed the spelling of "modelling" to "modeling" in the title for consistency with American English, as used throughout the paper.  

We have also made the following additions to the supplementary video webpage.  
1. Explanation of non-fluid and sudden movements in visualization results, attributed to SMPL conversion rather than the model, in Section A, as suggested by reviewers **jS9r** and **LgJ2**.
2. Included Longer generation results (10 seconds) in Section B, as suggested by reviewer **jS9r**.
3. Added ablation results of the Inter-M Transformer for each attention module in Section C, to address concerns raised by reviewers  **jS9r**, **LgJ2** and **YC4H**. 
4. Results on complex and in-the-wild instructions in Section D, as suggested by reviewer **LgJ2**.  

We have sincerely made our best effort to address all concerns raised by the reviewers. We hope that the additional results, insights, and clarifications strengthen our contributions and demonstrate the capability of our model as the current state-of-the-art for human interaction generation. Please do not hesitate to let us know if you have any additional questions or concerns. *If our responses have adequately addressed your concerns, we would appreciate it so much if you could re-evaluate our work, and possibly raise your score.* Thanks again for all your effort in evaluating our work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces InterMask, a novel framework for generating 3D human-human interactions from textual descriptions. It employs 2D VQ-VAE to encode motion into a 2D discrete token map and uses masked generative modeling to model spatio-temporal interdependencies. While both the masked modeling approach and transformer-based design are not invented by the authors, the use of them for human interaction modeling is creative. By carefully integrating different components, the proposed method is able to effectively capture 4D structure of humans, achieve state-of-the-art results, and maintain high efficiency. The reviewers are unanimously positive about the paper. While there are initially some concerns about the experimental evaluations and analyses, the authors did a great job during the rebuttal phase and addressed most of them. The ACs urge the authors to incorporate the feedback from the reviewers into the final version of the paper. The ACs recommend acceptance.