InterMask: 3D Human Interaction Generation via Collaborative Masked Modelling

## Abstract

Abstract Generating realistic 3D human-human interactions from textual descriptions remains a challenging task. Existing approaches, typically based on diffusion models, often generate unnatural and unrealistic results. In this work, we introduce InterMask , a novel framework for generating human interactions using collaborative masked modeling in discrete space. InterMask first employs a VQ-VAE to transform each motion sequence into a 2D discrete motion token map. Unlike traditional 1D VQ token maps, it better preserves fine-grained spatio-temporal details and promotes spatial awareness within each token. Building on this representation, InterMask utilizes a generative masked modeling framework to collaboratively model the tokens of two interacting individuals. This is achieved by employing a transformer architecture specifically designed to capture complex spatio-temporal interdependencies. During training, it randomly masks the motion tokens of both individuals and learns to predict them. In inference, starting from fully masked sequences, it progressively fills in the tokens for both individuals. With its enhanced motion representation, dedicated architecture, and effective learning strategy, InterMask achieves state-of-the-art results, producing high-fidelity and diverse human interactions. It outperforms previous methods, achieving an FID of 5.154 5.154 5.154 5.154 (vs 5.535 5.535 5.535 5.535 for in2IN) on the InterHuman dataset and 0.399 0.399 0.399 0.399 (vs 5.207 5.207 5.207 5.207 for InterGen) on the InterX dataset. Additionally, InterMask seamlessly supports reaction generation without the need for model redesign or fine-tuning.

## 1 Introduction

3D human interaction generation is a fundamental task in computer vision and graphics, with applications in animation, virtual reality, robotics, and sports. Although significant progress has been made in generating single-person motion sequences from text descriptions (Tevet et al., ; Guo et al., ; ; Pinyoanuntapong et al., ; Zhang et al., ; Petrovich et al., ), generating two-person interactions remains a significant challenge. In two-person interactions, each individual’s motion depends not only on their own state but also on the state of their interacting partner. Additionally, precise spatial positioning and orientation becomes crucial, specially in close-contact cases. These factors add to the complexity and nuances of human interactions resulting in a higher degree of spatial and temporal interdependence compared to single-person motion.

Figure: Figure 1: InterMask generates high fidelity text-conditioned 3D human interactions, with accurate spatial and temporal coordination, including synchronized dance moves, realistic reaction timings in boxing, and correct proximity, while maintaining high-quality poses.
Refer to caption: x1.png

Existing works on human interaction generation commonly rely on diffusion models. ComMDM (Shafir et al., ) bridges two pretrained single-person motion diffusion models using a small neural layer with limited interaction data. In contrast, InterGen (Liang et al., ) proposes a specifically tailored diffusion model for two-person interactions, which uses cooperative transformers to model each individual’s motion conditioned on the latent features of their partner’s motion. in2IN (Ruiz-Ponce et al., ) then extends it by conditioning the diffusion model on LLM-generated individual descriptions in addition to the overall interaction descriptions, and leveraging motion priors learned from single-person motion datasets like HumanML3D (Guo et al., ). MoMat-MoGen Cai et al. () retrieves motions from supplementary datasets, based on textual inputs, and refines them for interaction quality. Some very recent efforts (Shan et al., ; Fan et al., ) extend these methods to more than two individuals. Despite these efforts, the generated two-person interactions still fall short from achieving satisfactory realism and fidelity.

In this work, we present *InterMask*, a novel framework based on generative masked modeling in discrete space (Chang et al., ). InterMask consists of two stages. First, a shared VQ-VAE is trained to transform each individual’s motion into a 2D discrete token map using a learned codebook $\mathcal{C}$ (section 3.1). Unlike previous motion VQ-VAEs (Guo et al., ; Zhang et al., ; Jiang et al., ; Guo et al., ; Pinyoanuntapong et al., ), which represent motion as a temporal sequence of 1D pose vectors and use 1D convolutions to capture only temporal context in each token, we preserve both temporal and spatial dimensions. Each motion is represented as $\mathbb{R}^{N\times J\times d}$, where $N$ is the number of poses, $J$ is the number of joints, and $d$ is the joint feature dimension. Using 2D convolutions, we generate a 2D token map, which captures body part context over short time ranges in each token, allowing for better spatial awareness. This is crucial for interactions, where relative body part positioning (e.g., hands) of two individuals holds significant importance.

In the second stage, a specialized Inter-M Transformer (section 3.2) is trained to collaboratively model the tokens of both individuals using a generative masked modeling framework. During training, a random proportion of tokens from either both individuals or only one individual is masked and predicted. The masking proportion follows a scheduling function to ensure its tractability during inference. The inference process begins with all tokens masked, which are progressively generated over a pre-defined number of iterations. At each iteration, the model predicts all masked tokens, but only the most confident predictions are retained, while the rest are re-masked and re-predicted in subsequent iterations. In addition to the standard self attention module, Inter-M Transformer features a shared spatio-temporal attention module to capture fine-grained dependencies within each individual’s motion, and a cross-attention module to capture dependencies between the two individuals’ motion. This design effectively the dynamics of spatial and temporal relationships with and between the two individuals, allowing enhanced interaction generation capabilities.

The main contributions of this work are as follows. First, we introduce InterMask, a novel masked generative framework for human-human interaction generation from textual descriptions. InterMask utilizes a 2D VQ encoding that transforms motion sequences into discrete 2D token maps and features a dedicated transformer trained with a generative masked modeling framework which effectively captures both intra- and inter-person spatial-temporal dependencies. Second, InterMask achieves state-of-the-art results in generating high-fidelity, text-conditioned human interactions. Empirically, it achieves an FID of 5.15 (vs. 5.54 in Ruiz-Ponce et al. ()) on InterHuman and 0.399 (vs. 5.207 in Liang et al. ()) on InterX. Third, InterMask seamlessly supports tasks such as reaction generation without requiring any task-specific fine-tuning or architectural changes.

## 2 Related Work

Quantized Motion Representation Quantized latent representations transform continuous data into discrete tokens, which has been widely explored in human motion generation. Deep Motion Signatures (Aristidou et al., ) adopted the use of contrastive learning to create discrete motif words, while TM2T (Guo et al., ) was one of the first to apply VQ-VAE (Van Den Oord et al., ) to human motion data, producing discrete motion tokens. T2M-GPT (Zhang et al., ) used Exponential Moving Average (EMA) and code reset techniques to improve the VQ-VAE, later adopted in PoseGPT (Lucas et al., ) and MotionGPT (Jiang et al., ; Zhang et al., ). MoMask (Guo et al., ) reduced quantization errors via residual quantization, while MMM (Pinyoanuntapong et al., ) utilized a large codebook with the factorized codes technique from Yu et al. (). Nevertheless, these works often ignore the spatial dimension of motions, where each token encapsulates the whole information of poses in a short range. This limits the locality of each token, which is important in applications like interactions and motion editing.

Generative Human Motion Modeling Synthesizing single-person motion has gained interest, driven by the availability of large motion capture datasets and advancements in generative modeling techniques like Diffusions (Tevet et al., ; Kim et al., ; Zhang et al., ; Tseng et al., ; Chen et al., ; Kong et al., ; Lou et al., ) and Autoregressive models (Guo et al., ; Zhang et al., ; Jiang et al., ; Zhang et al., ; Gong et al., ; Lucas et al., ; Gong et al., ). While these models produce high-quality motions, they require numerous sampling steps during inference. Techniques like the DDIM sampler (Song et al., ) aim to address this for diffusion models but are still evolving. Additionally, autoregressive models may have limited expressivity since the model only considers past tokens to generate the next one. MoMask (Guo et al., ) and MMM (Pinyoanuntapong et al., ) attempted to address both these issues by using a masked generative bidirectional transformer to predict all masked tokens simultaneously. These works inspired the base structure of our InterMask.

Generative Human-Human Interaction Modeling Most research on human interaction modeling has targeted two-person interactions, primarily addressing two major tasks: 1) reaction generation, where the reactor’s motion is generated based on the actor’s motion, and 2) interaction generation, where both individuals’ motions are generated simultaneously. While reaction generation has seen increasing interest Chopin et al., ; Liu et al., ; Xu et al., ; Ghosh et al., ; Ren et al., ; Liu et al., , interaction generation is less explored. ComMDM (Shafir et al., ) trained a small neural network to bridge two single-person motion diffusion model (MDM (Tevet et al., )) copies on a limited interaction dataset. On the other hand, RIG (Tanaka & Fujiwara, ) and InterGen (Liang et al., ) introduced interaction diffusion models that simultaneously denoise both individuals’ motions, conditioned on their partner’s latent representation. Subsequent works, like  Cai et al. () and Ruiz-Ponce et al. () extended interaction diffusion models by using additional single-person annotations and supplementary motion datasets, whereas Shan et al. () and Fan et al. () extended them to handle more than two individuals. While these methods achieve impressive results, there remains significant room for improvement in the quality and realism of two-person interactions. In this paper, we address this by explicitly modeling the spatio-temporal dependencies between interacting individuals’ using a masked generative framework.

## 3 Methodology

Our objective is to generate two-person interaction $\{\mathbf{m}_{p}\}_{p\in\{a,b\}}$ given a textual description, where $\mathbf{m}_{p}\in\mathbb{R}^{N\times J\times d}$ represents the motion sequences of one individual (either $a$ or $b$), consisting of $N$ poses, each with $J$ joints and $d$-dimensional joint features. As shown in Figure 2, our methodology consists of two stages. First, we learn a discrete representation of individual motions using a VQ-VAE (Section 3.1), which maps motion sequences to a 2-dimensional token map $\{t_{p}\}_{p\in\{a,b\}}\in\{0,1,\cdots,|\mathcal{C}|-1\}^{n\times j}$, where $n$ and $j$ are down-sampled temporal and spatial dimensions, and $\mathcal{|C|}$ is the number of codes in the codebook. Then we learn an interaction masked generative transformer (Inter-M Transformer), (Section 3.2) to collaboratively model the discrete tokens of both individuals.

Figure: Figure 2: Overview of InterMask. (a) Individual motions are quantized through vector quantization (VQ) to obtain 2D tokens $\{t_{a},t_{b}\}$ for each. (b) Motion tokens from both individuals are flattened, concatenated, masked and predicted collaboratively by the Inter-M Transformer. (c) Each block in Inter-M Transformer consists of Self, Spatio-Temporal and Cross Attention modules to learn complex spatio-temporal dependencies within and between both interacting individuals.
Refer to caption: x2.png

### 3.1 2D Discrete Motion Token Map

We start by learning a discrete latent representation space for individual motions using a VQ-VAE. We tokenize individual motions instead of the entire interaction because the complexity of modeling interactions is significantly higher. This allows both individuals to share the same discrete token space, simplifying the representation and improving expressivity, as empirically shown in Table 2. Additionally, this provides greater flexibility for complementary tasks such as reaction generation.

We create a 2D token map with both spatial and temporal dimensions by representing motion as $\mathbf{m}_{p}\in\mathbb{R}^{N\times J\times d}$ and using 2D convolutions. As illustrated in Figure 2(a), our VQ-VAE downsamples the input sequence $\mathbf{m}_{p}$ into a latent representation $\mathbf{\tilde{t}_{p}}\in\mathbb{R}^{n\times j\times d^{\prime}}$, with downsampling ratios $n/N$ and $j/J$. Each $d^{\prime}$-dimensional vector is then quantized by replacing it with the nearest entry from a learnable codebook $\mathcal{C}=\{\mathbf{c}_{k}\}_{k=0}^{|\mathcal{C}|-1}\subset\mathbb{R}^{d^{
\prime}}$, producing a quantized sequence $\mathbf{t}_{p}=\mathrm{Q}(\mathbf{\tilde{t}}_{p})\in\mathbb{R}^{n\times j
\times d^{\prime}}$. This is then decoded to reconstruct the motion $\mathbf{\hat{m}_{p}=\mathrm{D}(\mathbf{t}_{p})}$. After training, each vector in $\mathbf{t}_{p}$ can be replaced by its respective index $k$ in $\mathcal{C}$ to obtain the 2d discrete representation of motion, namely *motion tokens* $\{t_{p}\}_{p\in\{a,b\}}\in\{0,1,\cdots,|\mathcal{C}|-1\}^{n\times j}$, which is modeled in our Inter-M Transformer.

#### 3.1.1 Training Objective

Our primary VQ-VAE training objective consists of a motion reconstruction loss and the commitment loss (Van Den Oord et al., ):

$$ $\displaystyle\mathcal{L}_{vq}=\|\mathbf{m}_{p}-\mathbf{\hat{m}}_{p}\|_{1}+ \beta\|\mathbf{\tilde{t}}_{p}-\mathrm{sg}(\mathbf{t}_{p})\|_{2}^{2},$ (1) $$

where $\mathrm{sg}(\cdot)$ denotes the stop-gradient operation, and $\beta$ a weighting factor. We use Exponential Moving Average (EMA) and codebook reset to update $\mathcal{C}$, following  Zhang et al. ().

We also incorporate geometric losses from Liang et al. () to impose additional constraints, as shown in eq. 2. The velocity loss $\mathcal{L}_{vel}$ encourages the reconstructed motion sequences to obey the velocity of joints in the ground truth sequences and the bone length loss $\mathcal{L}_{bl}$ the distance between adjacent joints. The foot contact loss $\mathcal{L}_{f}c$ encourages the feet to have zero velocity whenever they are in contact with the ground.

$$ $\displaystyle\mathcal{L}_{vel}=\frac{1}{N-1}\sum_{i_{n}=1}^{N}\|(m_{i_{n}+1}-m _{i_{n}})-(\hat{m}_{i_{n}+1}-\hat{m}_{i_{n}})\|_{1}$ (2) $\displaystyle\mathcal{L}_{fc}=\frac{1}{N-1}\sum_{i_{n}=1}^{N}\|(\hat{m}_{i_{n} +1}-\hat{m}_{i_{n}})\cdot f_{i_{n}}\|_{1}$ $\displaystyle\mathcal{L}_{bl}=\frac{1}{N-1}\sum_{i_{n}=1}^{N}\|B(m_{i_{n}})-B( \hat{m}_{i_{n}})\|_{1}$ $$

Here, $m_{i_{n}}$ represents the ground pose, $\hat{m}_{i_{n}}$ represents the reconstructed pose at time step $i_{n}$, $N$ represents the total time steps in the sequence, $f_{i_{n}}\in\{0,1\}$ represents the binary foot contact label for the heel and toe joints for each pose $m_{i_{n}}$, and $B(\cdot)$ denotes the bone lengths joining adjacent joints.

The overall loss function $\mathcal{L}_{vqvae}$ is a weighted sum of the primary VQ loss and geometric losses:

$$ $\mathcal{L}_{vqvae}=\mathcal{L}_{vq}+\lambda_{vel}\mathcal{L}_{vel}+\lambda_{ fc}\mathcal{L}_{fc}+\lambda_{bl}\mathcal{L}_{bl}$ (3) $$

### 3.2 Interaction Masked Generative Transformer

Next, the motion tokens for both individuals {$t_{a},t_{b}$} are collaboratively modeled by our Inter-M Transformer, capturing complex spatial and temporal dependencies within and between them. Importantly, there is no order to the actions of interacting individuals in this task. Therefore we make sure our data sampling, transformer architecture and embedding design are all permutation invariant.

#### 3.2.1 Token Masking and Embedding

As shown in Figure 2(b), we first flatten and concatenate $t_{a}$ and $t_{b}$, separated by a special $[\mathrm{SEP}]$ token, to form a consolidated sequence $t\in\mathbb{Z}^{2nj+1}$. A subset of these tokens is then masked by replacing with a learnable special-purpose token $[\mathrm{MASK}]$ following a two-stage scheme. In the first stage, we apply *random masking* with probability $p_{r}$ or *interaction masking* with $1-p_{r}$. In random masking, both $t_{a}$ and $t_{b}$ are randomly masked with ratio $\gamma(\tau_{i})$, controlled by a cosine scheduling function (Chang et al., ), as shown in eq. 4. In interaction masking, one of $\{t_{a},t_{b}\}$ is kept fully unmasked, encouraging the model not to rely on the tokens from the same individual and learn inter-individual dependencies.

$$ $\displaystyle\gamma(\tau_{i})=\cos\left(\frac{\pi\tau_{i}}{2}\right)\in[0,1]$ (4) $\displaystyle\tau_{i}\sim\mathcal{U}(0,1)$ $$

In the second stage, we employ *step unroll masking*, adopted from Kim et al. (), where we remask a portion ($\gamma(\tau_{i+1})$) of the predicted tokens with the lowest confidence scores and predict them again. This refines the predictions iteratively, aligning with the inference process, where tokens are progressively revealed. The masked tokens $\tilde{t}$ are then embedded using the *input process* module, consisting of a learnable embedding layer and a linear transformation. Then, 2D positional encodings, from  Wang & Liu (), are added to impose spatio-temporal structure on the embeddings. The updated embeddings, $\mathbf{e}^{0}\in\mathbb{R}^{(2nj+1)\times\tilde{d}}$, where $\tilde{d}$ is the transformer embedding dimension, are then passed through $\mathbf{L}$ blocks of the Inter-M transformer.

#### 3.2.2 Transformer block design

As illustrated in  Figure 2(c), each Inter-M block starts with a self-attention module (Vaswani et al., ) to model long-range, global dependencies within and between individuals. Then we introduce a novel shared spatio-temporal attention layer to model spatial and temporal dependencies within each individual’s motion tokens. Finally, we employ a shared cross attention layer to model inter-individual dependencies. This design enables rich and dynamic modeling of spatio-temporal interactions, ensuring effective learning of both intra- and inter-individual relationships.

Self Attention Let $\mathbf{e}^{l-1}\in\mathbb{R}^{(2nj+1)\times\tilde{d}}$ represent the input token embeddings to block $l$, where $l\in\{1,2,\dots\mathbf{L}\}$. The block architecture begins with a self-attention module which computes attention scores for all tokens in the sequence using the standard scaled dot-product attention mechanism:

$$ $\mathrm{Attn}(\mathbf{Q},\mathbf{K},\mathbf{V})=\mathrm{softmax}(\mathbf{QK}^{ \top}/\sqrt{\tilde{d}})\mathbf{V}$ (5) $$

where the query, key, and value matrices, $\mathbf{Q}$, $\mathbf{K}$, and $\mathbf{V}$, are linear projections of $\mathbf{e}_{l-1}$. The self attention module is followed by a feedforward network (FFN).

Spatio-Temporal Attention Next, we split the embeddings of the two individuals and pass them through a shared spatio-temporal attention module. This module consists of two separate attention mechanisms: spatial attention and temporal attention. In the spatial attention module, each token attends only to other spatial tokens within the same temporal instance; and in the temporal attention module, each token attends only to tokens across the temporal dimension at the same spatial location. Let $\mathbf{e}_{p}^{l-1}$ represent the updated embeddings of individual $p\in\{a,b\}$ in block $l$ after the self attention module. We drop the $l-1$ superscript to simplify the notation. Then $\mathbf{e}_{p}(i_{n})\in\mathbb{R}^{j\times\tilde{d}}$ represent the spatial tokens of individual $p$ at temporal instance $i_{n}$, and $\mathbf{e}_{p}(i_{j})\in\mathbb{R}^{n\times\tilde{d}}$ represent the temporal tokens of individual $p$ at spatial location $i_{j}$. The spatial and temporal attention is defined as:

$$ $\mathbf{e}_{p}^{\prime}(i_{n})=\mathrm{Attn}(\mathbf{Q}_{j},\mathbf{K}_{j}, \mathbf{V}_{j})\quad\mathbf{e}_{p}^{\prime\prime}(i_{j})=\mathrm{Attn}(\mathbf {Q}_{n},\mathbf{K}_{n},\mathbf{V}_{n})$ (6) $$

where $\mathbf{Q}_{j},\mathbf{K}_{j},\mathbf{V}_{j}$, and $\mathbf{Q}_{n},\mathbf{K}_{n},\mathbf{V}_{n}$ are the spatial queries, keys and values and temporal queries, keys, and values, obtained from $\mathbf{e}_{p}(i_{n})$ and $\mathbf{e}_{p}(i_{j})$ respectively. After updating the embeddings with spatial and temporal attention, the outputs of these two modules are added together for each token:

$$ $\mathbf{e}_{p}=\mathbf{e}_{p}^{\prime}(i_{n})+\mathbf{e}_{p}^{\prime\prime}(i_ {j})\quad\forall\quad 0<i_{n}<n,0<i_{j}<j$ (7) $$

Cross Attention Next, we apply a shared cross attention module to model dependencies between the two individuals. Each token of individual $a$ attends to all tokens of individual $b$ and vice versa:

$$ $\mathbf{e}_{a}^{\prime}=\mathrm{Attn}(\mathbf{Q}_{a},\mathbf{K}_{b},\mathbf{V} _{b})\quad\mathbf{e}_{b}^{\prime}=\mathrm{Attn}(\mathbf{Q}_{b},\mathbf{K}_{a}, \mathbf{V}_{a})$ (8) $$

where $\mathbf{Q}_{a},\mathbf{K}_{a},\mathbf{V}_{a}$, and $\mathbf{Q}_{b},\mathbf{K}_{b},\mathbf{V}_{b}$ are the queries, keys and values obtained from embeddings $\mathbf{e}_{a},\mathbf{e}_{b}$ of each individual. Finally, each individual’s embeddings are passed through another FFN and concatenated to form the sequence $\mathbf{e}^{l}\in\mathbb{R}^{(2nj+1)\times\tilde{d}}$, which is passed to the next block.

Text Conditioning We condition our transformer blocks on the input text by replacing the standard layer normalization in each attention and FFN module with modulated adaptive layer normalization (AdaLN-mod) as proposed by Peebles & Xie (). In AdaLN-mod, the dimension-wise scale and shift parameters are regressed using a multi-layer perceptron (MLP), from the conditional vector $c$, obtained from a frozen, pre-trained CLIP network (Radford et al., ). This modulation allows the model to dynamically adapt its normalization based on the content of $c$. Additionally, we initialize each residual connection in the transformer block as an identity function by regressing another scale parameter $\alpha$ from $c$, which is initialized to output a zero vector. This ensures that the network can adapt the modulation during learning without imposing strong biases at initialization.

#### 3.2.3 Output Process

After the final block, we
use a standard linear projector with AdaLN-mod to map the output $\mathbf{e}^{\mathbf{L}}$ to the indices of codebook $\mathcal{C}$, $\mathbf{e}_{out}\in\mathbb{R}^{2nj\times|\mathcal{C}|}$, where $|\mathcal{C}|$ represents the size of $\mathcal{C}$. As defined in eq. 9, the transformer’s training objective minimizes the negative log-likelihood of the masked tokens, computed as the cross-entropy between the one-hot encoded ground truth and model predictions.

$$ $\displaystyle\mathcal{L}_{mask}=\sum_{\tilde{t}_{k}=[\mathrm{MASK}]}-\log p_{ \theta}(t_{k}|\tilde{t},c).$ (9) $$

### 3.3 Inference

Figure: Figure 3: Inference process. Starting from completely masked token sequences of both individuals $\{t_{a}(0),t_{b}(0)\}$, the Inter-M transformer generates all tokens in $I$ iterations. Next, the tokens are dequantized and decoded to generate motion sequences $\{\mathbf{m}_{a},\mathbf{m}_{b}\}$ using the VQ-VAE decoder.
Refer to caption: x3.png

As illustrated in Figure 3, the inference process consists of two stages. Starting with a fully masked sequence $t(0)$, the Inter-M transformer generates tokens for both individuals over $I$ iterations. At each iteration $i$, the transformer predicts token probabilities at masked locations, then samples tokens and remasks those with the lowest $\lceil\gamma\left(\frac{i}{I}\right)\cdot 2nj\rceil$ confidence scores, repeating until $i$ reaches $I$. The final token sequence is decoded back into motion using the VQ-VAE decoder. A cosine schedule $\gamma\left(\frac{i}{I}\right)$ controls the number of retained tokens, increasing as $i$ progresses. Classifier-free guidance (Ho & Salimans, ) is also applied for refinement, following Chang et al. ().

## 4 Experiments

Datasets We adopt two datasets to evaluate InterMask for the text-conditioned human interaction generation task: InterHuman (Liang et al., ) and InterX (Xu et al., ). The InterHuman dataset contains 7,779 interaction sequences and InterX contains 11,388, each paired with 3 distinct textual annotations. InterHuman follows the AMASS (Mahmood et al., ) skeleton representation with 22 joints, including the root joint. Each joint is represented by $\{p_{g},v_{g},r_{6d}\}$, where $p_{g}\in\mathbb{R}^{3}$ is the global position, $v_{g}\in\mathbb{R}^{3}$ is the global velocity, and $r_{6d}\in\mathbb{R}^{6}$ is the local 6D rotation of each joint, rendering $\mathbf{m}_{p}\in\mathbb{R}^{N\times 22\times 12}$. InterX follows the SMPL-X (Pavlakos et al., ) skeleton representation, comprising 54 body, hands and face joints, accompanied by root orientation and translation. Each joint and root orientation is represented by $\{r_{6d}\}$ and root translation by $\{p_{g}\}$, to which we incorporate root velocity rendering root $\{p_{g},v_{g}\}$ and $\mathbf{m}_{p}\in\mathbb{R}^{N\times 56\times 6}$. We adhere to the respective body representations of both datasets to demonstrate that our framework is compatible with different representation formats and joint counts. Other implementation details including architecture, training and inference hyper-parameters are provided in appendix A.

Evaluation Metrics Following Liang et al. (), we adopt several feature-space evaluation metrics to assess the performance of our model. These include the Frechet Inception Distance (*FID*), which measures the fidelity of the generated interactions by calculating similarity between the generated and real interactions feature distributions. Additionally we employ *R-precision* and *MMDist*, which evaluate how well the generated interactions align with the corresponding texts. Furthermore, we use *Diversity* to measure the overall variation in generated interactions, and multimodality (*MModality*) to quantify the ability to generate multiple distinct interactions for the same text.

### 4.1 Comparison with state-of-the-art approaches

**Table 1: Quantitative evaluation on the InterHuman and InterX test sets. $\pm$ indicates a 95% confidence interval and $\rightarrow$ means the closer to ground truth the better. Bold face indicates the best result, while underscore refers to the second best.**
| Dataset | Method | R Precision$\uparrow$ | FID$\downarrow$ | MM Dist$\downarrow$ | Diversity$\rightarrow$ | MModality$\uparrow$ |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Top 1 | Top 2 | Top 3 |  |  |  |  |  |  |
| Inter<br>Human | Ground Truth | ${0.452}^{\pm{.008}}$ | ${0.610}^{\pm{.009}}$ | ${0.701}^{\pm{.008}}$ | ${0.273}^{\pm{.007}}$ | ${3.755}^{\pm{.008}}$ | ${7.948}^{\pm{.064}}$ | - |
| T2M (Guo et al., ) | ${0.238}^{\pm{.012}}$ | ${0.325}^{\pm{.010}}$ | ${0.464}^{\pm{.014}}$ | ${13.769}^{\pm{.072}}$ | ${5.731}^{\pm{.013}}$ | ${7.046}^{\pm{.022}}$ | ${1.387}^{\pm{.076}}$ |  |
| MDM (Tevet et al., ) | ${0.153}^{\pm{.012}}$ | ${0.260}^{\pm{.009}}$ | ${0.339}^{\pm{.012}}$ | ${9.167}^{\pm{.056}}$ | ${7.125}^{\pm{.018}}$ | ${7.602}^{\pm{.045}}$ | ${\textbf{2.350}}^{\pm{.080}}$ |  |
| ComMDM (Shafir et al., ) | ${0.223}^{\pm{.009}}$ | ${0.334}^{\pm{.008}}$ | ${0.466}^{\pm{.010}}$ | ${7.069}^{\pm{.054}}$ | ${6.212}^{\pm{.021}}$ | ${7.244}^{\pm{.038}}$ | ${1.822}^{\pm{.052}}$ |  |
| InterGen (Liang et al., ) | ${0.371}^{\pm{.010}}$ | ${0.515}^{\pm{.012}}$ | ${0.624}^{\pm{.010}}$ | ${5.918}^{\pm{.079}}$ | ${5.108}^{\pm{.014}}$ | ${7.387}^{\pm{.029}}$ | ${\underline{2.141}}^{\pm{.063}}$ |  |
| MoMat-MoGen (Cai et al., ) | ${\textbf{0.449}}^{\pm{.004}}$ | ${\underline{0.591}}^{\pm{.003}}$ | ${\underline{0.666}}^{\pm{.004}}$ | ${5.674}^{\pm{.085}}$ | ${\textbf{3.790}}^{\pm{.001}}$ | ${8.021}^{\pm{.35}}$ | ${1.295}^{\pm{.023}}$ |  |
| in2IN (Ruiz-Ponce et al., ) | ${\underline{0.425}}^{\pm{.008}}$ | ${0.576}^{\pm{.008}}$ | ${0.662}^{\pm{.009}}$ | ${\underline{5.535}}^{\pm{.120}}$ | ${\underline{3.803}}^{\pm{.002}}$ | ${\underline{7.953}}^{\pm{.047}}$ | ${1.215}^{\pm{.023}}$ |  |
| InterMask | ${\textbf{0.449}}^{\pm{.004}}$ | ${\textbf{0.599}}^{\pm{.005}}$ | ${\textbf{0.683}}^{\pm{004}}$ | ${\textbf{5.154}}^{\pm{.061}}$ | ${\textbf{3.790}}^{\pm{.002}}$ | ${\textbf{7.944}}^{\pm{.033}}$ | ${1.737}^{\pm{.020}}$ |  |
| InterX | Ground Truth | ${0.429}^{\pm{.004}}$ | ${0.626}^{\pm{.003}}$ | ${0.736}^{\pm{.003}}$ | ${0.002}^{\pm{.0002}}$ | ${3.536}^{\pm{.013}}$ | ${9.734}^{\pm{.078}}$ | - |
| T2M (Guo et al., ) | ${0.184}^{\pm{.010}}$ | ${0.298}^{\pm{.006}}$ | ${0.396}^{\pm{.005}}$ | ${5.481}^{\pm{.382}}$ | ${9.576}^{\pm{.006}}$ | ${2.771}^{\pm{.151}}$ | ${2.761}^{\pm{.042}}$ |  |
| MDM (Tevet et al., ) | ${0.203}^{\pm{.009}}$ | ${0.329}^{\pm{.007}}$ | ${0.426}^{\pm{.005}}$ | ${23.701}^{\pm{.057}}$ | ${\underline{9.548}}^{\pm{.014}}$ | ${5.856}^{\pm{.077}}$ | ${\underline{3.490}}^{\pm{.061}}$ |  |
| ComMDM (Shafir et al., ) | ${0.090}^{\pm{.002}}$ | ${0.165}^{\pm{.004}}$ | ${0.236}^{\pm{.004}}$ | ${29.266}^{\pm{.067}}$ | ${6.870}^{\pm{.017}}$ | ${4.734}^{\pm{.067}}$ | ${0.771}^{\pm{.053}}$ |  |
| InterGen (Liang et al., ) | ${\underline{0.207}}^{\pm{.004}}$ | ${\underline{0.335}}^{\pm{.005}}$ | ${\underline{0.429}}^{\pm{.005}}$ | ${\underline{5.207}}^{\pm{.216}}$ | ${9.580}^{\pm{.011}}$ | ${\underline{7.788}}^{\pm{.208}}$ | ${\textbf{3.686}}^{\pm{.052}}$ |  |
| InterMask | ${\textbf{0.403}}^{\pm{.005}}$ | ${\textbf{0.595}}^{\pm{.004}}$ | ${\textbf{0.705}}^{\pm{005}}$ | ${\textbf{0.399}}^{\pm{.013}}$ | ${\textbf{3.705}}^{\pm{.017}}$ | ${\textbf{9.046}}^{\pm{.073}}$ | ${2.261}^{\pm{.081}}$ |  |

Quantitative Comparison Table 1 shows quantitative comparison of our InterMask with previous methods. All evaluations are run 20 times (with the exception of MModality, which is run 5 times), and we report the averaged results along with a 95% confidence interval. InterMask achieves state-of-the-art (SOTA) results on both InterHuman and InterX datasets. It records the lowest FID scores (5.154 on InterHuman and 0.399 on InterX), indicating superior realism and quality of generated interactions, and leads in R-Precision and MMdist, showing excellent text alignment. With different body representations in each dataset, these results highlight the robustness of our method, proving it is not dependent on a specific body pose structure. Furthermore, the larger performance gap on the larger InterX dataset suggests that InterMask scales effectively with more data. While our MModality is slightly lower than some methods, the high R-Precision and low MMdist emphasize that InterMask prioritizes adherence to text over extreme diversity, yet still achieves high Diversity scores, demonstrating its ability to generate a broad range of interactions. Please refer to appendix B for visual demonstration of InterMask’s ability to generate diverse results from the same text prompt.

Qualitative Comparison
In Figure 4, we provide a qualitative comparison of interaction sequences generated by our InterMask and InterGen Liang et al. (), trained on the InterHuman dataset, for the same text descriptions. For the prompt ”*The first person raises the right leg aggressively towards the second*”, InterGen incorrectly raises both legs, while InterMask accurately raises only the right leg. For ”*Two people bow to each other*”, InterGen introduces an unnecessary fighting gesture after the bow, indicating overfitting to implicit biases in the training data, whereas InterMask stays true to the simple bow. Lastly, for ”*One person is sitting and waving their hands at the other person, while the other drifts away*”, InterGen generates a crouching pose instead of sitting and misplaces the waving action, while InterMask faithfully generates the described scene. These examples demonstrate that InterMask produces more higher quality and more reliable interactions compared to InterGen.

Figure: Figure 4: Qualitative comparison between InterMask and InterGen (Liang et al., ), highlighting InterMask’s superior interaction quality, text adherence and avoidance of implicit biases.
Refer to caption: x4.png

Figure: Table 2: Ablation Study results on InterHuman test set to verify key components of the proposed Motion VQ-VAE. Bold face indicates the best result.
Refer to caption: x5.png

User Study User studies are essential in generative AI, complementing metrics like FID by incorporating human perception to evaluate the realism and quality of generated content. To compare our InterMask with InterGen (Liang et al., ), we conducted a user study where 30 interaction sequences were generated by both models with identical motion lengths. Each sequence pair was rated by 3 users on interaction quality and adherence to text (scores out of 5), and users selected their preferred sequence based on both factors. More details are provided in appendix F. As shown in Figure 5, InterMask outperforms InterGen in interaction quality (4.089 vs. 3.296) and text adherence (3.938 vs. 3.198), with 69.14% of users preferring InterMask over InterGen.

Computation and Time Cost In addition to superior generation fidelity and text adherence, InterMask demonstrates better computational efficiency when compared to InterGen (Liang et al., ). InterMask contains only 74 M inference parameters, compared to 182 M for InterGen, resulting in a considerably smaller model footprint. Furthermore, InterMask achieves an average inference time of 0.77 seconds, less than half of InterGen’s 1.63 seconds, highlighting its low latency.

### 4.2 Ablation Studies

**Table 3: Ablation Study results on the InterHuman test set to verify key components of the proposed Inter-M Transformer. Bold face indicates the best result.**
| Method | R Precision$\uparrow$ | FID$\downarrow$ | MM Dist$\downarrow$ | Diversity$\rightarrow$ | MModality$\uparrow$ |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Top 1 | Top 2 | Top 3 |  |  |  |  |  |
| Ground Truth | ${0.452}^{\pm{.008}}$ | ${0.610}^{\pm{.009}}$ | ${0.701}^{\pm{.008}}$ | ${0.273}^{\pm{.007}}$ | ${3.755}^{\pm{.008}}$ | ${7.948}^{\pm{.064}}$ | - |
| Alternative Modeling | ${0.340}^{\pm{.002}}$ | ${0.425}^{\pm{.009}}$ | ${0.514}^{\pm{.006}}$ | ${7.637}^{\pm{.072}}$ | ${3.937}^{\pm{.003}}$ | ${8.424}^{\pm{.022}}$ | ${\textbf{2.192}}^{\pm{.076}}$ |
| Collaborative Modeling |  |  |  |  |  |  |  |
| w/o Self Attention | ${0.393}^{\pm{.006}}$ | ${0.527}^{\pm{.005}}$ | ${0.596}^{\pm{007}}$ | ${5.715}^{\pm{.067}}$ | ${3.827}^{\pm{.003}}$ | ${7.873}^{\pm{.034}}$ | ${1.683}^{\pm{.076}}$ |
| w/o Cross Attention | ${0.328}^{\pm{.008}}$ | ${0.416}^{\pm{.009}}$ | ${0.508}^{\pm{007}}$ | ${8.461}^{\pm{.076}}$ | ${3.972}^{\pm{.005}}$ | ${\textbf{7.951}}^{\pm{.085}}$ | ${1.942}^{\pm{.076}}$ |
| w/o Spatio-Temporal Attention | ${0.296}^{\pm{.004}}$ | ${0.391}^{\pm{.007}}$ | ${0.470}^{\pm{005}}$ | ${10.968}^{\pm{.093}}$ | ${4.219}^{\pm{.006}}$ | ${7.659}^{\pm{.023}}$ | ${1.607}^{\pm{.076}}$ |
| w/o Step Unroll and Interaction Masking | ${0.402}^{\pm{.003}}$ | ${0.548}^{\pm{.004}}$ | ${0.633}^{\pm{002}}$ | ${5.629}^{\pm{.053}}$ | ${3.806}^{\pm{.002}}$ | ${7.942}^{\pm{.096}}$ | ${1.741}^{\pm{.076}}$ |
| InterMask | ${\textbf{0.449}}^{\pm{.004}}$ | ${\textbf{0.599}}^{\pm{.005}}$ | ${\textbf{0.683}}^{\pm{004}}$ | ${\textbf{5.154}}^{\pm{.061}}$ | ${\textbf{3.790}}^{\pm{.002}}$ | ${7.944}^{\pm{.033}}$ | ${1.737}^{\pm{.020}}$ |

Figure: Figure 6: Qualitative results for the ablation study on Motion VQ-VAE to verify the proposed 2D token map.
Refer to caption: x6.png

Figure: Figure 7: Qualitative results for the ablation study on Inter-M Transformer to verify the proposed Spatio-Temporal Attention module.
Refer to caption: x7.png

Motion VQ-VAE Table 2 presents an ablation study to evaluate key components of our proposed *Motion VQ-VAE*. We investigated two primary factors: the input modality (individual motions vs two-person interactions) and the token map dimensions (1d vs 2d), using Reconstruction FID and MPJPE (Mean Per Joint Position Error). The results demonstrate that using individual motions as input with a 2D token map outperforms both the baseline of individual motions with a 1D token map, and the two-person interaction with a 2D token map. These findings suggest that preserving spatial information through 2D token maps and focusing on individual motions performs accurate motion reconstruction, and creates a more expressive token map with high-fidelity movement details. This can be further verified by the qualitative comparison provided in Figure 6. As shown, the 1D VQ-VAE struggles to accurately reconstruct the spatial positions and orientations of the joints for both individuals, leading to not only unrealistic interactions but also abnormal individual poses. In contrast, the proposed 2D VQ-VAE provides highly accurate reconstruction at both the individual pose and the collective interaction level.

Inter-M Transformer Table 3 presents an ablation study to evaluate key components of our *Inter-M Transformer*. We compared collaborative modeling against alternative modeling and examined the impact of various attention mechanisms and masking strategies. *Alternative Modeling* (appendix C) models tokens of one person at a time, while conditioning in on the tokens of the other person, leading to an alternative generation during inference. *Collaborative Modeling* on the other hand, models tokens of both individuals simultaneously. Our full InterMask model with collaborataive modeling outperforms all ablated versions, particularly in R-Precision and FID scores. Notably, removing any of the attention mechanisms or employing alternative modeling severely degraded performance, with spatio-temporal attention being the most crucial one. Additionally, step unroll and interaction masking proved to be further fine-tuning the results. These results highlight the synergistic importance of all components in our transformer architecture for effective interaction modeling. The contribution of spatio-temporal attentionis further verified by the qualitative comparison illustrated in Figure 7. While the model without spatio-temporal attention generates somewhat plausible individual poses, such as *punching* and *blocking* in the *boxing* scenario, it fails to fully capture the dynamics of the interaction, leading to incorrect positioning between the two individuals. Similarly, in the *sneaking* scenario, the model without spatio-temporal attention does not capture the gradual change in distance between the two individuals, where one should be moving closer to the other. In contrast, the model with spatio-temporal attention accurately captures these spatial dynamics and interactions.

### 4.3 Application: Reaction Generation

Figure: Figure 8: Reaction generation samples with and without text descriptions, including holistic instruction for both individuals and separate instructions for each.
Refer to caption: x8.png

One of the key strengths of our InterMask model is its seamless support for the task of reaction generation with or without a text condition, where the motion of one person in an interaction is generated based on the provided reference motion of the other person. Unlike previous diffusion-based models, such as InterGen (Liang et al., ), InterMask can perform this task without any task-specific fine-tuning or architectural re-design. This flexibility arises from our use of the masked modeling technique combined with an individual motion tokenizer, which does not require interactions as input. During inference, we simply tokenize the reference motion, leave its tokens unmasked, and progressively generate tokens of the other person. More details are provided in appendix D.

Quantitatively, InterMask achieves an FID of $2.99$ and an R-precision of $0.462$ on InterHuman for reaction generation. In comparison, InterGen suffers from a performance drop with an FID of $52.89$ (vs $5.918$ for interaction generation) and an R-precision of $0.194$ (vs $0.371$) when adopted for reaction generation without fine-tuning, demonstrating its inability to perform reaction generation out-of-the-box. As shown qualitatively in Figure 8, InterMask produces high-quality reaction motions. In cases without text conditioning, when the reference motion involves dancing, the generated motion follows the reference steps and even captures nuanced details like raising the same leg or arm. Similarly, for the physical combat scenario, when the reference motion attacks, the generated motion retreats and vice versa. For text-conditioned samples, InterMask performs equally well, whether the text describes both people’s motions or provides separate descriptions for each. The model accurately identifies the required reaction form text and the reference motion and generates the corresponding motion, maintaining high-quality poses and realistic timings throughout.

## 5 Conclusion

In this paper, we presented InterMask, a novel framework for generating realistic 3D human interactions through generative masked modeling in the discrete space. InterMask first employs a VQ-VAE to transform individual motion sequences into 2D discrete token maps, preserving both spatial and temporal dimensions. Then, a specialized Inter-M transformer collaboratively models the motion tokens of both individuals, capturing intricate spatial and temporal dependencies within and between them. Quantitative evaluations show that InterMask excels in generating realistic and contextually aligned motions, achieving the lowest FID scores and superior results in R-Precision and MMdist on both the InterHuman and InterX datasets compared to existing models. Additionally, user studies comparing InterMask to InterGen reveal a 69% user preference for InterMask, noting higher-quality and more realistic interactions. The flexibility of our approach is also evident by its ability to perform equally well for different body representations. While there are minor trade-offs in diversity (MModality), InterMask consistently maintains high quality and adherence to text while generating a wide range of realistic interactions.

Limitations and Future Work
While InterMask generates high-quality human interactions, there are some limitations. First, when visualized as SMPL Loper et al. () meshes, issues such as body penetration between individuals and jerky movements during rapid actions can occur. Second, our model sometimes interprets motions as dances without explicit prompting, likely due to implicit biases in the training dataset. Lastly, our approach is currently optimized for short sequences up to 10 seconds, reflecting the constraints of our dataset. Visual examples are shown in appendix E. Future research directions include implementing techniques for smoother transitions, addressing dataset biases, and expanding the model’s capacity to handle longer interaction sequences.

## Appendix A Implementation Details

Our models are implemented using PyTorch, with details of the model architecture, training, and inference provided below. Key hyperparameters are summarized in the accompanying tables.

### A.1 Model Architecture

The Motion VQ-VAE employs 2D convolutional residual blocks for both the encoder and decoder. The temporal downsampling factor is $n/N=1/4$ for both datasets, while the spatial downsampling is dataset-specific: $j/J=5/22$ for InterHuman and $j/J=5/56$ for InterX. Strided convolutions are used for downsampling in the encoder, while the decoder restores dimensions via upsampling and convolutional layers. The latent representation in VQ-VAE has a dimension $d^{\prime}=512$, and the codebook size $|\mathcal{C}|=1024$.

For the Inter-M transformer, we use $\mathbf{L}=6$ transformer blocks, each with 6 attention heads. The transformer embedding dimension is $\tilde{d}=384$.

**Table 4: Motion VQ-VAE and Inter-M Transformer Model Parameters**
| Parameter | Value | Description |
| --- | --- | --- |
| $d^{\prime}$ | 512 | Latent space dimension of VQ-VAE |
| $|\mathcal{C}|$ | 1024 | Codebook size (number of entries) |
| $n/N$ | 1/4 | Temporal downsampling factor for both datasets |
| $j/J$ (InterHuman) | 5/22 | Spatial downsampling for InterHuman dataset |
| $j/J$ (InterX) | 5/56 | Spatial downsampling for InterX dataset |
| $\mathbf{L}$ | 6 | Number of transformer blocks |
| Attention heads | 6 | Number of attention heads per block |
| $\tilde{d}$ | 384 | Transformer embedding dimension |
| CLIP version | ViT-L/14@336px | Version of CLIP used for text in transformer |

### A.2 Training Details

The Motion VQ-VAE is trained for 50 epochs with a batch size of 512. The learning rate is initialized at 0.0002 and decays via a multistep learning rate schedule, reducing by a factor of 0.1 after 70% and 85% of the iterations. A linear warm-up is applied for the first quarter of the iterations. The commitment loss factor $\beta$ is 0.02, and the geometric losses for velocity, foot contact, and bone length are weighted differently across the datasets.

The Inter-M transformer is trained for 500 epochs with a batch size of 52, following a similar multistep learning rate decay but with a decay factor of 1/3 after 50%, 70%, and 85% of the iterations. The condition drop probability is 0.1 to allow for flexibility in training with or without text conditioning.

**Table 5: Training Hyperparameters for the Motion VQ-VAE and Inter-M Transformer**
| Parameter | Value | Description |
| --- | --- | --- |
| VQ-VAE batch size | 512 | Number of samples per batch for VQ-VAE |
| Transformer batch size | 52 | Number of samples per batch for transformer |
| Initial learning rate | 0.0002 | Starting learning rate for both models |
| Learning rate decay | 0.1 / 1/3 | Decay factor for VQ-VAE / Transformer learning rate |
| $\beta$ | 0.02 | Commitment loss factor for VQ-VAE |
| $\lambda_{vel},\lambda_{fc},\lambda_{bl}$ (InterHuman) | 100, 500, 5 | Geometric loss weights for InterHuman |
| $\lambda_{vel},\lambda_{fc},\lambda_{bl}$ (InterX) | 100, 100, 5 | Geometric loss weights for InterX dataset |
| Condition drop prob. | 0.1 | Drop probability for text conditioning during transformer training |

### A.3 Inference Details

During inference, the number of iterations $I$ is set to 20 for interaction generation and 12 for reaction generation. A classifier-free guidance (CFG) scale of 2 is applied, and the temperature is set to 1 to balance diversity and coherence in the generated results.

**Table 6: Inference Hyperparameters for Interaction and Reaction Generation**
| Parameter | Interaction Generation | Reaction Generation |
| --- | --- | --- |
| Number of iterations | $I=20$ | $I_{react}=12$ |
| CFG scale | 2 |  |
| Temperature | 1 |  |

## Appendix B Diversity Demonstration

Figure: Figure 9: Diversity Demonstration of our method, where it generates two distinct interaction sequences for the same text prompt.
Refer to caption: x9.png

Our quantitative comparison (section 4.1) shows that InterMask prioritizes adherence to text over extreme diversity, while still being able to generate different distinct interactions for the same text prompt. Here, in fig. 9, we show 2 visual examples to demonstrate this. In both cases, the model remains consistent in generating interactions described in the text prompt while exhibiting distinct features in different samples. For the dancing case, first sample shows waving hands in the beginning followed by a synchronized forward step, while individuals face the same direction throughout. The second sample shows individuals facing each other in the beginning, followed by waving hands and a synchronized spin. For the boxing case, first sample shows one individual punching three times and continuously moving forward, while the second sample shows them punching two times and retreating at the end.

## Appendix C Alternative Modeling

Figure: Figure 10: Overview of the Alternative Modeling approach, where we predict the tokens of one person at a time. (a) During training, only the embeddings of one individual $\mathbf{e}_{a}$ are updated in the transformer blocks, conditioned on the other individual’s embeddings $\mathbf{e}_{b}$. (b) During inference, the process alternates between predicting and remasking the tokens of each individual, starting with both fully masked $\{t_{a}(0),t_{b}(0)\}$. This process continues for $I_{alter}$ iterations for each individual.
Refer to caption: x10.png

In our ablation study (section 4.2), we explore a one-at-a-time modeling framework for interaction generation, referred to as *Alternative Modeling*, which contrasts with the collaborative modeling framework of InterMask. While collaborative modeling predicts the tokens of both individuals simultaneously, alternative modeling follows a sequential process, generating tokens for one individual at a time, conditioned on the thus far predicted tokens of the other.

During training, as shown in Figure 10(a), we randomly mask both individuals’ tokens $\{\tilde{t}_{a},\tilde{t}_{b}\}$ and obtain their embeddings $\{\mathbf{e}_{a},\mathbf{e}_{b}\}$ through the input process. Then, only the tokens of one individual are predicted ($\hat{t}_{a}$) by passing their embeddings through the transformer blocks, conditioned on the embeddings of the other individual using a cross-attention module. During inference (Figure 10(b)), both individuals’ tokens are initially fully masked $\{t_{a}(0),t_{b}(0)\}$. In the first iteration, the tokens of one individual are predicted, and these are remasked based on their confidence scores to obtain $t_{a}(1)$. The second individual’s tokens are then predicted in the next iteration, conditioned on the retained tokens from the first, to obtain $t_{b}(1)$. This alternation continues iteratively, progressively refining the tokens of both individuals. As shown in Table 3, the FID score for alternative modeling is 7.637 (compared to 5.154 for collaborative modeling), and the R-precision is 0.340 (compared to 0.449). These results indicate that while alternative modeling increases diversity, collaborative modeling produces high-quality and more realistic interactions, offering a better balance between diversity and interaction fidelity.

## Appendix D Reaction Generation Inference

Figure: Figure 11: Inference process for the Reaction Generation task. The motion tokens of the reference individual $t_{b}$ are obtained from the encoder and kept unmasked throughout. The second individual’s tokens are initially fully masked $t_{a}(0)$, and are predicted progressively over $I_{\text{react}}$ iterations to obtain $t_{a}$, which is then decoded using the decoder.
Refer to caption: x11.png

InterMask does not require task-specific fine-tuning or architectural re-design for reaction generation, needing only minor adjustments to the inference process, as illustrated in Figure 11. The process begins by encoding the reference individual’s motion $\mathbf{m}_{b}$ into tokens $t_{b}$ using the VQ-VAE encoder. For the other individual, whose reaction is to be generated, we initialize with a fully masked token sequence, $t_{a}(0)$. Over the course of $I_{\text{react}}$ iterations, the transformer progressively predicts and fills in the masked tokens, while the reference tokens remain unmasked throughout. At each iteration $i_{\text{react}}$, the least confident $\gamma\left(\frac{i_{\text{react}}}{I_{\text{react}}}\right)\cdot nj$ tokens are remasked and predicted again, following a cosine scheduling function $\gamma(\cdot)$. Once all tokens are generated, the final token sequence $t_{a}$ is decoded back into motion $\mathbf{m}_{a}$ using the VQ-VAE decoder. Since we drop the conditioning signal during some training passes, reaction generation functions effectively both with and without a text description, enabling the model to generate motions based solely on the reference motion or guided by additional textual instructions.

## Appendix E Failure Cases

Figure: Figure 12: Examples of Limitations of our method. The first row shows body penetration when converted from output skeleton to SMPL mesh. The second row shows implicit bias towards dancing.
Refer to caption: x12.png

In Figure 12, we show visual examples of failure cases emerging from the limitations of our method, as decribed in section 5. In the first row, we show that when converting our output skeletons to SMPL (Loper et al., ) meshes for visualizations, the results can exhibit body penetration among the interacting individuals. One possible future solution to this problem is to include the mesh conversion in the training process and incorporate anti-penetration in the training loss. In the second row, we show that the model suffers from some implicit biases present in the dataset, where it assumes that the individuals are dancing without explicit mention in the text prompt.

## Appendix F User Study

We conducted the user study on the Amazon Mechanical Turk platform, where the interface presented to the users is shown in Figure 13. For each sample, users were provided with clear instructions to rate two animations—one generated by InterMask and the other by InterGen (Liang et al., )—from the same text description, with the order of the animations randomized for each sample to avoid bias. Participants were asked to rate both animations on a scale of 1 to 5 for interaction quality and text adherence. Following the individual ratings, they were asked to select the better animation based on their overall impression. A total of 30 samples were evaluated, with each sample being rated by three users. To ensure high-quality feedback, we filtered users to include only those with Amazon Mechanical Turk *master* status, with a task approval rating of over 97% and more than 1000 previously approved tasks.

Figure: Figure 13: Interface of the User Study on Amazon Mechanical Turk.
Refer to caption: x13.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 2
questions: 1. I noticed that in the second video of the visualization "Everyday Actions," the character on the right makes a sudden arm-raising motion, which seems somewhat disconnected. Additionally, is there a 10-second video available for reference? I saw that the authors mentioned the ability to generate interactive results up to 10 seconds long.

2. In terms of quantitative experiments, it seems there is no comparison with PriorMDM, which also produces results for two-person interactions. Similarly, there doesn't appear to be a comparison with other VQ-VAE methods like T2M-GPT. I'm curious about their results, especially since they perform very well on single-person datasets. If T2M-GPT were given a reference action sequence for one individual as an additional condition, how would it perform in generating interactive individuals, particularly on interactive datasets?

3. Regarding the Inter-M Transformer structure, although there is some evidence in the ablation studies concerning the relative importance of different attention mechanisms, it may not fully clarify their specific roles and trade-offs in different interaction scenarios and data distributions.
soundness: 2
strengths: 1. A Transformer structure combined with VQ-VAE has been proposed for generating 3D human interaction actions. 
2. It utilizes a 2D discrete motion token map to represent motion, distinguishing itself from traditional 1D representation methods. This representation better preserves spatiotemporal details and spatial awareness, which is crucial for accurately capturing complex motion relationships in human interactions. 
3. The framework has achieved state-of-the-art results on the InterHuman and InterX datasets, performing exceptionally well on various metrics. 
4. The average inference time is only 0.77 seconds, significantly lower than InterGen's 1.63 seconds, providing a considerable advantage in practical applications by enabling faster result generation.
summary: This article introduces a new framework called InterMask for generating 3D human interaction actions from text descriptions. The proposed framework appears to be based on the T2M-GPT framework, modeling in discrete space using generative masks, while also incorporating the concept of interhand to collaborate within the mask modeling framework by utilizing tokens from two interacting individuals. During the inference process, it starts with a fully masked sequence and gradually fills in the tokens for both individuals.
weaknesses: 1. In the visualization results, issues such as body penetration between interacting individuals and unnatural sliding movements still occur.
2. The article also mentions the masking strategy, but I believe its effectiveness may be influenced by various factors, such as the setting of the mask ratio and the dynamic adjustment of masks at different stages. The article does not delve deeply into how these factors affect model performance or how to optimally set these parameters.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. Could the authors comment on the suitability of the current framework for multi-person interactions? Given the focus on two-person setups, it would be useful to understand any potential modifications or challenges in adapting this approach for more complex, multi-person interaction scenarios.

2. How does the model handle longer, more complex instructions or instructions that are less structured (in-the-wild)? 

3. Do the authors see potential for combining InterMask with language models to improve interpretability or to handle a wider range of textual prompts?
soundness: 3
strengths: 1. The use of 2D token maps and spatial-temporal attention layers in the Inter-M transformer enables more nuanced modeling of human interactions, demonstrating improved fidelity in motion generation.

2. InterMask achieves state-of-the-art results on key metrics in standard datasets, indicating its effectiveness in generating realistic, text-aligned interactions.

3. The model is relatively lightweight and demonstrates reduced inference time.
summary: The paper presents InterMask, a framework for generating 3D human interactions using masked modeling in discrete space. By encoding motion as 2D token maps and modeling intra- and inter-person dependencies with a custom transformer, InterMask achieves impressive results on the InterHuman and InterX datasets. The model shows strong performance in terms of realism and alignment with text prompts, outperforming existing methods in controlled settings.
weaknesses: 1. The paper’s innovative contribution is not well-defined. Although the authors introduce a transformer module to model intra- and inter-person spatial-temporal dependencies, this approach is widely used in various generative tasks. The ablation studies, which compare only with/without this module, do not sufficiently demonstrate the model’s unique contribution. If this module is intended as a core innovation, the authors should focus on explaining why its internal structure, arrangement, or sequence specifically suits the 3D interaction modeling problem. Merely showing the presence or absence of the module does not justify its relevance, as it is a fairly standard approach.

2. The process of converting each motion sequence into a 2D discrete motion token map lacks detailed description. The main innovation of this paper appears to lie in the combination of the 2D discrete motion token map with MAE; however, there is insufficient explanation of how the 2D token map is constructed. Providing a more thorough description, ideally with a detailed diagram, would enhance clarity and better communicate this aspect of the method.

3. User Study. The user study includes only three participants, which is insufficient to draw robust conclusions on user preference. A larger sample size and detailed descriptions of the study’s methodology would add reliability and provide stronger validation for claims of user preference for InterMask-generated interactions.

4. Upon reviewing the supplementary videos, the generated motions appear to lack the fluidity seen in InterGen’s outputs. A discussion on this aspect would be valuable, particularly to understand any limitations in InterMask’s approach to generating smooth, natural motion.

---

confidence: 5
contribution: 2
initial_rating: 6
presentation: 3
questions: I have no other questions.
soundness: 3
strengths: 1. This paper introduces a novel 2D VQ-VAE that quantizes motion into a 2D discrete map, which can more effectively preserve fine-grained spatio-temporal details than the 1D version.

2. The proposed method achieves SOTA performance on the InterHuman and InterX benchmarks.
summary: This paper addresses the problem of generating human-human interaction from textual descriptions. The authors introduce InterMask, a novel framework that employs 2D VQ-VAE to encode interactions into a discrete space and generative masked modeling to capture complex spatio-temporal interdependencies.
weaknesses: 1. The method does not appear substantially different from prior work. Overall, the paper feels somewhat underwhelming, lacking distinctive insights, though there are no major issues with the experiments or writing. I would rate it as borderline.

2. The proposed approach faces challenges in generating human reactions in an online manner (although this isn’t the primary focus of the current setting). This capability would be more practical for some virtual reality applications.

Meta-review: The paper introduces InterMask, a novel framework for generating 3D human-human interactions from textual descriptions. It employs 2D VQ-VAE to encode motion into a 2D discrete token map and uses masked generative modeling to model spatio-temporal interdependencies. While both the masked modeling approach and transformer-based design are not invented by the authors, the use of them for human interaction modeling is creative. By carefully integrating different components, the proposed method is able to effectively capture 4D structure of humans, achieve state-of-the-art results, and maintain high efficiency. The reviewers are unanimously positive about the paper. While there are initially some concerns about the experimental evaluations and analyses, the authors did a great job during the rebuttal phase and addressed most of them. The ACs urge the authors to incorporate the feedback from the reviewers into the final version of the paper. The ACs recommend acceptance.

Decision: Accept (Poster)