InterMask: 3D Human Interaction Generation via Collaborative Masked Modeling
Authors: Muhammad Gohar Javed
chuan guo
Li cheng
Xingyu Li
Venue: ICLR 2025 Poster (2025)
Keywords: Motion Synthesis
Human Interaction Generation
Masked Generative Transformer
Text-driven Generation
Vector Quantized VAE
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: A novel framework for human interaction generation using collaborative masked modeling in the discrete space, which explicitly models spatio-temporal dependencies within and between the interacting individuals.

Abstract: Generating realistic 3D human-human interactions from textual descriptions remains a challenging task. Existing approaches, typically based on diffusion models, often produce results lacking realism and fidelity. In this work, we introduce *InterMask*, a novel framework for generating human interactions using collaborative masked modeling in discrete space. InterMask first employs a VQ-VAE to transform each motion sequence into a 2D discrete motion token map. Unlike traditional 1D VQ token maps, it better preserves fine-grained spatio-temporal details and promotes *spatial awareness* within each token. Building on this representation, InterMask utilizes a generative masked modeling framework to collaboratively model the tokens of two interacting individuals. This is achieved by employing a transformer architecture specifically designed to capture complex spatio-temporal inter-dependencies. During training, it randomly masks the motion tokens of both individuals and learns to predict them. For inference, starting from fully masked sequences, it progressively fills in the tokens for both individuals. With its enhanced motion representation, dedicated architecture, and effective learning strategy, InterMask achieves state-of-the-art results, producing high-fidelity and diverse human interactions. It outperforms previous methods, achieving an FID of $5.154$ (vs $5.535$ of in2IN) on the InterHuman dataset and $0.399$ (vs $5.207$ of InterGen) on the InterX dataset. Additionally, InterMask seamlessly supports reaction generation without the need for model redesign or fine-tuning.

Published as a conference paper at ICLR 2025
INTERMASK: 3D HUMAN INTERACTION GENERATION VIA COLLABORATIVE MASKED MODELING
Muhammad Gohar Javed', Chuan Guo?, Li Cheng', Xingyu Li'
University of Alberta Snap Inc. {javed4, 1cheng5, xingyu]eua wwerta. ca, cguo2esnapchat.com
ABSTRACT
Generating realistic 3D human-human interactions from textual descriptions re- mains challenging task. Existing approaches, typically based on diffusion mod- els, often produce results lacking realism and fidelity. In this work, we introduce InterMask, a novel framework for generating human interactions using collabo- rative masked modeling in discrete space. InterMask first employs VQ-VAE to transform each motion sequence into a 2D discrete motion token map. Unlike traditional ID VQ token maps, it better preserves tine-graine spatio-temporal details and promotes spatial awareness within each token. Building on this rep- resentation, InterMask utilizes generative masked modeling framework col- laboratively model the tokens of two interacting individuals. This is achieved by employing transformer architecture specifically designed capture complex spatio-temporal inter-dependencces. During training. it randomly masks the mo- tion tokens of both individuals and learns to predict them. For inference, starting from fully masked sequences, it progressively fills in the tokens for both individ- uals. With its enhanced motion representation, dedicated architecture, and effec- tive learning strategy, InterMask achieves state-of-the-ar results, producing high- fidelity and diverse human interactions. It outperforms previous methods, achiev- ing an FID of 5.154 (vs 5.535 of in2IN) on the InterHuman dataset and 0.399 (vs 5.207 of InterGen) on the InterX dataset. Additionally, InterMask seamlessly supports reaction generation without the need for model redesign or fine-tuning
gohar-mal ik. gi thut.hb... io/ intintrrraak
INTRODUCTION
3D human interaction generation fundamental task in computer vision and graphics, with ap- plications in animation, virtual reality, robotics, and sports. Although significant progress has been made generating single-person motion sequences from text descriptions (Tevet et al. 2023 Guo
et al. 2024 2022al Pinyoamuntapong et al. 2024 Zhang et al. 2023 Petrovich al. 2022). gen- crating two-person interactions remains a significant challenge. In two-person interactions, each individual's 's motion depends not only their
state but also the state of their interacting partner. Additionally, precise spatial positioning and orientation becomes crucial, specially in close- contact cases. These factors add to the complexity and nuances of human interactions resulting in higher degree of spatial and temporal interdependence compared to single-person motion.
Existing works on human interaction generation commonly rely on diffusion models. Com- MDM (Shafir et al., (024) bridges two pretrained single-person motion diffusion models using a small neural layer with limited interaction data. contrast, InterGen (Liang al., 20244 proposes a specifically tailored diffusion model for two-person interactions. which uses cooperative trans- formers to model each individual's 's motion conditioned on the latent features of their partner's mo- tion. in2IN (Ruiz-Ponce et al.) 2024) then extends it by conditioning the diffusion model on LLM- generated individual descriptions addition to the overall interaction descriptions, and leveraging motion priors learned from single-person motion datasets like HumanML3D (Guo et al. 2022a)
MoMat-MoGen Cai et al. (2024) retrieves motions from supplementary datasets, based on textual inputs, and refines them for interaction quality. Some very recent efforts (Shan et al. 2024 Fan
ct al. 2024) extend these methods to more than two individuals. Despite these efforts, the generated two-person interactions still fall short of achieving satisfactory realism and fidelity.

---

Published as a conference paper at ICLR 2025
One person takes five staps get the who is chair holding something in
In boxing match, one continuously punching while the other is defending and countarattacking
person sneaks up other from behind
are parforming synchronized dance moves
InterMask generates high fidelity text-conditioned 3D human interactions, with accurate spatial and temporal coordination, including synchronized dance moves, realistic reaction timings in boxing, and correct proximity, while maintaining high-quality poses.
Figure 1:
In this work, we present InterMask, a novel framework based on generative masked modeling in discrete space (Chang al., (022). InterMask consists of two stages. First, a shared VQ- (Q-VAE is trained to transform each individual's 's motion into a 2D discrete token map using learned codebook
(section 3.T). Unlike previous motion VQ-VAFs (Guo et al. 2022b Zhang et al.) 2023 Jiang eet al. 2023 Guo et al. 2024] Pinyoanuntapong et al. 2024), which represent motion
temporal sequence of ID pose vectors and use ID convolutions to capture only temporal context in each token, preserve both temporal and spatial dimensions. Each motion is represented as IR xdxx where N is the number of poses, J is the number of joints, and d the joint feature dimension. Using 2D convolutions, generate 2D token map, which captures body part context short time ranges in each token, allowing for better spatial awareness. This is crucial for interactions, where relative body part positioning (e.g., hands) of two individuals holds significant importance.
In the second stage, a specialized Inter-M Transformer (section 3.2) is trained to collaboratively model the tokens of both individuals using generative masked modeling framework. During train- ing, a random proportion of tokens from either both individuals or only one individual is masked and predicted. The masking proportion follows a scheduling function ensure its tractability during inference. The inference process begins with all tokens masked, which are progressively generated over pre-defined number of iterations. At each iteration, the model predicts all masked tokens, but only the most confident predictions are retained, while the rest are re-masked and re-predicted in subsequent iterations. addition to the standard self attention module, Inter-M Transformer fea- tures shared spatio-temporal attention module to capture fine-grained dependencies within each individual's 's motion, and a cross-attention module to capture dependencies between the two indi- viduals' motion. This design effectively models the dynamics of spatial and temporal relationships among and between the two individuals, allowing enhanced interaction generation capabilities.
The main contributions of this work are follows. First, introduce InterMask, novel masked generative framework for humann-human interaction generation from textual descriptions. Inter- Mask utilizes 2D VQ encoding that transforms motion sequences into discrete 2D token maps and features a dedicated transformer trained with a generative masked modeling framework which effectively captures both intra- and inter-person spatial and temporal dependencies. Second, Inter- Mask achieves ttte-offthhearr results in generating high-fidelity. text-conditioned human interac- tions. Empirically, it achieves an FID of 5,15 (vs. 5.54 of Ruiz-Ponce al. (2024)) on InterHuman and 0.399 (vs. 5.207 of Liang et al. (2024)) on InterX. Third, InterMask seamlessly supports tasks such reaction generation without requiring any task-specific fine-tuning or architectural changes.
RELATED WORK
Quantized Motion Representation Quantized latent representations transform continuous data into discrete tokens, which has been widely explored in human motion generation. Deep Motion Signa- tures (Aristidou et al. 2018) adopted the use of contrastive learning create discrete motif words,
2

---

Published as a conference paper at ICLR 2025
while TM2T (uo et al.
2022b)
one of the first to apply VQ-VAE Van Den Oord et al. 2017)
to human motion data, producing discrete motion tokens. T2M- (Zhag et al. (023) used Ex-
ponential Moving Average (EMA) and code reset techniques to improve the VQ- VAE, later adopted in PoseGPT ( (Lucas et al.
2022) and MotionGPT (Jiang et al. 2023 Zhang et al. 2024b). Mo- Mask (Guo et al. (2024) reduced quantization errors via residual quantization, while MMM (Pinyoa- nuntapong et al. 2024) utilized a large codebook with the factorized codes technique from Yu et al. (2022). Nevertheless, these works often ignore the spatial dimension of motions, where each token encapsulates the whole information of poses in short range. This limits the locality of each token, which is important in applications like interactions and motion editing.
Generative Human Motion Modeling Synthesizing single-person motion has gained interest, driven by the availability of large motion capture datasets and advancements in generative mod-
eling techniques like Diffusions (Tevet et al. 2023 Kim et al. 2023a Zhang et al. 2024a] Tseng
ett al.
2023, Chen et af., 2023) Kong et al. 2023 Lou et al. 2023) and Autoregressive models (Guo
et al. 2022b Zhang et al. 2023 Jiang et al. 2023 Zhang et al. 2024b| Gong al.
2023 Lucas
ctt al.
2022] Gong et al. 2023). While these models produce high-quality motions, they require
numerous sampling steps during inference. Techniques like the DDIM sampler (Song et al. 2021) aim to address this for diffusion models but are still evolving. Additionally, autoregressive models may have limited expressivity since the model only considers past tokens to generate the next one. MoMask (Guo et al., 2024) and MMM (Pinyoanuntapong et al.) 2024) attempted to address both these issues by using masked generative bidirectional transformer to predict all masked tokens simultaneously. These works inspired the base structure of our InterMask.
Generative Humann-Human Interaction Modeling Most research on human interaction modeling has targeted two-person interactions, primarily addressing two major tasks: 1) reaction generation, where the reactor's motion is generated based on the actor's motion, and 2) interaction generation, where both individuals' motions are generated simultancously. While reaction generation has seen increasing interest Chopin et al. 2023 Liu et al., 2023 Xu et al., 2024b; Ghosh et al.] 2024 Ren et al. 2024. Liu et al., 20244, interaction generation is less explored. ComMDM (Shafir et al.) 2024)
trained a small neural network to bridge two single-person motion diffusion model (MDM (Tevet
ctt al. 2)23)) copies on a limited interaction dataset. On the other hand, RIG (Tanaka & Fujiwara 2023 and InterGen (Liang et al. 2024) introduced interaction diffusion models that simultaneously denoise both individuals' motions, conditioned on their partner's latent representation. Subsequent works, like Cai et al. (2024) and Ruiz--PuiP et al.] (2024) extended interaction diffusion models by using additional single-person annotations and supplementary motion datasets, whereas Shan et al. (2024) and Fan et al. (2024) extended them to handle more than two individuals. While these meth- ods achieve impressive results, there remains sigmificant room for improvement in the quality and realism of two-person interactions. In this paper, we address this by explicitly modeling the spatio- temporal dependencies between interacting individuals' using a masked generative framework.
METHODOLOGY
Our objective is to generate two-person interaction (mp)pe (a. given textual description, where mp E Jxd represents the motion sequences of one individual (either b), consisting of N poses, each with . joints and d-dimensional joint features. As shown in Figure P our methodology consists of two stages. First, we learn discrete representation of individual motions using a VQ- VAE (section 3.1. which maps motion sequences to a 2-dimensional token map {tphpefa.t) E (0. 1,.1..... ... |C| - I nxx where and j are down-sampled temporal and spatial dimensions, and [C] the number of codes in the codebook. Then we learn an interaction masked generative transformer (Inter-M Transformer), (section 3.2 to collaboratively model the discrete tokens of both individuals.
3.1 2D DISCRETE MOTION TOKEN MAP
We start by learning discrete latent representation space for individual motions using a VQ-VAE. We tokenize individual motions instead of the entire interaction because the complexity of modeling interactions is significantly higher. This allows both individuals to share the same discrete token space, simplifying the representation and improving expressivity, as empirically shown in Table Additionally, this provides greater flexibility for complementary tasks such as reaction generation.
3

---

Published as a conference paper at ICLR 2025
(a) Motion VQ-VAE
(b) Interaction Masked Generative Tramsformer
XXX6
XX3
persen
strkke
person"
VQ
Input Tokens
Predicted Tokens
Inter-M
& Mask
Transformer
Cadebook indexing
2D Decoder
fo
Person
Person b
Spatio- Tempural Attn
Enput Process
Atti
FFN
Output Process
[SEP] token
Self
FFN O
Concat
MASK] token
Spatio- Temporal
-
Cross Attn
.P
FFN x
AdaLN-
MLP
XL blocks
(c) Transformer Block Design
Figure 2: Overview of InterMask. (a) Individual motions are quantized through vector quantization (VQ) to obtain 2D tokens {la,ts] 1o) for each. (b) Motion tokens from both individuals are flattened, concatenated, masked and predicted collaboratively by the Inter-M Transformer. (c) Each block in Inter-M Transformer consists Self, apatio-Temport and Cross Attention modules to learn complex seatio-temporal dependencies within and between both interacting individuals.
We create 2D token map with both spatial and temporal dimensions by representing motion as mp E xxxd and using 2D convolutions. As illustrated in Figure a). our VQ- VAE downsamples the input sequence m, into a latent representation tp E Rnxjjxd with downsampling ratios n/N and j/J. Each d' d-mmennsional vector is then quantized by replacing with the nearest entry from learnable codebook C = {cx } CI--
k== C Rdd, producing a quantized sequence t, = Q(t,)) E Rexjxa" This is then decoded reconstruct the motion mp D(tp). After training, each vector in t, can be replaced by its respective index A in C to obtain the 2d discrete representation of motion, namely motion tokens ((.)) E 1, ......... .CC 1]nxj, which is modeled in our Inter-M Transformer.
More details are provided in appendix B
3.1.1 TRAINING OBJECTIVE
Our primary VQ-VAE training objective consists of motion reconstruction loss and the commit- ment loss Van Den Oord et al. 2017):
Cog = m mp 111 + BIE sg,
(I)
where sg(.) denotes the stop-gradient operation, and weighting factor. We Exponential Moving Average (EMA) and codebook reset to update C, following Zhang et al. 2023).
We also incorporate geometric losses from Liang al. (2024) impose additional constraints, including the joint velocity loss Loel, the foot contact loss Lyc- and the bone length loss Lsi, which
are explained in appendix The overall loss function Lugeac is weighted sum of these losses:
Luquac Log + Avnt Luel + AfcLfe AmLLai
4

---

Published as a conference paper at ICLR 2025
3.2 INTERACTION MASKED GENERATIVE TRANSFORMER
Next, the motion tokens for both individuals {la,4s) are collaboratively modeled by our Inter-M Transformer, capturing complex spatial and temporal dependencies within and between them. Im- portantly, there is no order to the actions of interacting individuals in this task. Therefore we make sure our data sampling, transformer architecture and embedding design are all permutation invariant.
3.2.1 TOKEN MASKING AND EMBEDDING
As shown in Figure b). we first flatten and concatenate ta and &s, separated by special [SEP] token, to form a consolidated sequence t E 22nj+1, A subset of these tokens is then masked by replacing with learnable special-purpose token [MASK] following a two-stage scheme. In the first stage, we apply random masking with probability pr or interaction masking with 1 - Pr. In random masking, both la and I, are randomly masked with ratio 7(7)), controlled by cosine scheduling function (Chang et al. 2022), as shown in eq. (3). In interaction masking, one of {la; 4 is kept fully unmasked, encouraging the model not to rely on the tokens from the same individual and learn inter-individual dependencies.
((. = cos
cos
TT₂
E [0.1]
2
(3)
T/ 1)0,1)
In the second stage, we employ step unroll masking, adopted from Kim et al. (2023b). where we
remask portion (1( of the predicted tokens with the lowest confidence scores and predict them again. This refines the predictions iteratively, aligning with the inference process, where to- kens are progressively revealed. More details of the masking strategy are provided in appendix D The masked tokens é are then embedded using the input process module, consisting of a learnable embedding layer and a linear transformation. Then, 2D positional encodings, from Wang & Liu (021)) are added to impose spatio-temporal structure on the embeddings. The updated embed- dings, e E [22n5+1)xd where d is the transformer embedding dimension, are then passed through L blocks of the Inter-M transformer.
3.2.2 TRANSFORMER BLOCK DESIGN
As illustrated in co. each Inter-M block starts with a self-attention module (Vaswani et al. 2017) model long-range, global dependencies within and between individuals. Then intro- duce a novel shared spatio-temporal attention layer to model spatial and temporal dependencies within each individual's motion tokens. Finally, we employ a shared cross attention layer to model inter-individual dependencies. This design enables rich and dynamic modeling of interactions, ensuring effective learning of both intra- and inter-individual relationships.
Self Attention Let e E (2n+++)) I)xd represent the input token embeddings to block 1, where e (1,2,.- L). The block architecture begins with self-attention module which computes attention scores for all tokens in the sequence using the standard scaled dot-product attention mechanism:
Attn( Q K, V) = softmax QK /Vāyv
(4)
where the query, key, and value matrices, Q, K, and V, are linear projections of e The self attention module is followed by a feedforward network (FFN).
Spatio-Temporal Attention Next, we split the embeddings of the two individuals and pass them through a shared spatio-temporal attention module. This module consists of two separate attention mechanisms: spatial attention and temporal attention. In the spatial attention module, each token attends only to other spatial tokens within the same temporal instance; and in the temporal attention module, each token attends only to tokens across the temporal dimension at the same spatial location. Let eete e/-1 represent the updated embeddings of individual p E (a, b) in block after the self attention module. We drop the (-1 superscript simplify the notation. Then ep (in.) E Rixd represent the spatial tokens of individual p at temporal instance in. and (i;) E inndd represent the temporal tokens of individual p at spatial location iy. The spatial and temporal attention is defined as:
e Attn(Q Qj, K, Vj) (x)))) Attn( QQ,, K.. Vn)
(5)
5

---

Published as a conference paper at ICLR 2025
f.())
1,(1)
tx
"one person advances to strike the other person"
XTA
Inter-M Transforme
Inter-Mf Transforme
stops
CLEP
Remask
Codebook Lookup
Decoder
(a(
(a(1)
Person tokens
Person 6 tokenss
[SPP] taken
[ASK] token
Figure 3: Inference process. Starting from completely masked token sequences of both individuals (0). (0)), the Inter-M transformer generates all tokens in I iterations. Next, the tokens dequantized and decoded to generate motion sequences {ma, ma} using the VQ-VAE decoder.
where Qj, K, V,, and Q.. K,, Vn the spatial queries, keys and values and temporal queries, keys, and values, obtained from (in)n) and e,, (i,) respectively, After updating the embeddings with spatial and temporal attention, the outputs of these modules are added together for each token:
= ((.) + ( V 0 < n, 0 i
(6)
Cross Attention Next, we apply a shared cross attention module to model dependencies between the two individuals. Each token of individual a attends to all tokens of individual b and vice versa:
e Attn(Qa: K,, V.) e/, Attn(Quu K.,. Va)
where Qa K.. Va. and Qs. K., Vo are the queries, keys and values obtained from embeddings ea eg each individual. Finally, each individual's embeddings are passed through another FFN and concatenated to form the sequence E (R(2nj+1)xd which is passed to the next block.
Text Conditioning We condition transformer blocks the input text replacing the standard layer normalization each attention and FFN module with modulated adaptive layer normalization (AdaI.N-mod) as proposed by Peebles & Xie (023)) In Adal.N-mod the dimension-wise scale and shift parameters are regressed using multi-layer perceptron (MLP), from the conditional vector e, obtained from frozen, pre-trained CLIP network (Radford et al. 202 This modulation allows the model to dynamically adapt its normalization based on the content of c. Additionally, we initialize each residual connection in the transformer block as an identity function by regressing another scale parameter a from which is initialized to output zero vector. This ensures that the network can adapt the modulation during learning without imposing strong biases at initialization.
3.2.3 OUTPUT PROCESS
After the final block, use standard linear projector with indices of codebook C, Cout E 20] where |C] represents the size of C. As defined in eq. (8). the transformer's 's training objective minimizes the negative log- log-likelihood the masked tokens, computed the cross-entropy between the one-hot encoded ground truth and model predictions.
map the output e the
Lmask
Σ
potel
[c=[MASK]
3.3 INFERENCE
As illustrated in Figure the inference process consists of two stages. Starting with fully masked sequence t(),, the Inter-M transformer generates tokens for both individuals over / iterations. At each iteration i, the transformer predicts token probabilities masked locations, then samples to- kens and remasks those with the lowest [> ( 2nj] confidence scores, repeating until reaches I. The final token sequence is decoded back into motion using the VQ-VAE decoder. A cosine schedule y ( controls the number of retained tokens, increasing as progresses. Classifier-free guidance (Ho & Salimans, 2021) is also applied for refinement, following Chang al. (2022).

---

Published as a conference paper at ICLR 2025
Dataset
Method
Precision"
FIN
MM Dist)
Diversity-
MModality
Top 1
Top 2
Top 3
Ground Trath
0.452 Mow
0.610
0. 0.701 m-00
0.2773
3.755 M0s
7.948
T2M (Guo et al. 2022u
0.238 0.238*-002
0.325#-010
0.464#.024
13.76g=.or2
5.731 022
7.046*-022
1.3877t0075
Human
Inter
MDM (Tovel ( et all 2026
ComMDM SSail dlll
InterGen [Lung çi 43144
2024
0.153 012
0.371#-010
0:223
0. 06200
0. 0.515
0. 1.334--00
0.6 0.62
0. 0,466
0.339 012
5.918f..om o09
9.167 609
7.0692-000
7.125 018
5.108 014
6.2124
7,244
7.387#.022 029
7.602 045
2.350 010
1.82244
2.111 9006
MoMalt Motien
2024
0.449
D 1.501 t.00
0.608 o0e
5.674#.083
3,790 oc
8.021
1.295
in21N (Ruiz Prence et al.
0.425 008
0. 0.576 2008
0.002 000
5.535 1200
1.801 o02
7.083
1.215
InterMask
0,449 1926
0.599 t 5.00
0.683 1926
5.154
3.790 1922
7,944
7-944
1.737
Ground Trath
0.422y
0. 0.626*-000
0.736 008
0.002 0002
3.536 918
9.734
T2M (Cino et all 20226
0.1844.000
0. 0.2008 1.22099-a00
0.30g/ 1055
0-488
9.576 9.576- v00
2,771 4.7714-1311
2:761 4-042
MDM (revel er
0.20g uco
0.32g
0.426 oce
23.701 5.007
9.548
5.856=.077
11490
InterX
ComMOM (Sham es
2024
0.024)--002
0. 1.115-4
0.2286
1004
29. 207...6.0777
6.870 037
4.734 4.734440007
0.771 4.053
InterGen Laang 2024
0,207 o04
0. 1.335 80G
0.429 005
5.207 R16
9.580 o1
7.788 m08
3.686 9002
InterMusk
0.403
0.59g no1
0.705
0
3,705 017
9.046 200
2. 2.261 2.261#
Table 1: Quantitative evaluation on the InterHuman and Inter test sets. - indicates a 95% confidence interval and - means the closer ground truth the better. Bold face indicates the best result, while underscore refers to the second best.
4 EXPERIMENTS
Datasets We adopt two datasets to evaluate InterMask for the text-conditioned human interaction generation task: InterHuman (Liang et al., 2024 and InterX (Xu et al. dataset contains 7.779 interaction sequences and InterX contains 11,388, each paired with 3 distinct textual annotations. InterHuman follows the AMASS (Mahmood et al.] tation with 22 joints, including the root joint. Each joint is represented by (Pg. lig, (sd), where Pg E IR' is the global position, ig E RR is the global velocity, and roa E IR is the local 6D rota- tion of each joint, rendering m, E N22222212 InterX follows the SMPL-X (Pavlakos et al. 2019) skeleton representation, comprising 54 body, hands and face joints, accompanied by root orientation and translation. Each joint and root orientation is represented by {rea} and root translation by {py to which we incorporate root velocity rendering root {Py, J and mp E 5 We adhere to the respective body representations of both datasets to demonstrate that our framework is compat- ible with different representation formats and joint counts. Other implementation details including architecture, training and inference hyper-parameters are provided in appendix E
2024a). The InterHuman
2019) skeleton represen-
Evaluation Metries Following Liang et al. 2024). we adopt several feature-space evaluation met- rics assess the performance of our model. These include the Frechet Inception Distance (FID), which measures the fidelity of the generated interactions by calculating similarity between the gen- erated and real interactions feature distributions. Additionally we employ R-precision and MMDist, which evaluate how well the generated interactions align with the corresponding texts. Further- more, we use Diversity to measure the overall variation in generated interactions, and multimodality (MModality) to quantify the ability to generate multiple distinct interactions for the same text.
4.1 COMPARISON WITH STATE-OF-THE-AN APPROACHES
Quantitative Comparison Table I shows quantitative comparison of our InterMask with previous methods. All evaluations are run 20 times (with the exception of MModality, which is run 5 times), and we report the averaged results along with a 95% confidence interval. InterMask achieves state- of-the-art (SOTA) results on both InterHuman and InterX datasets. It records the lowest FID scores (5.154 on InterHuman and 0.399 on InterX), indicating superior realism and quality of generated interactions, and leads in R-Precision and MMdist, showing excellent text alignment. With different body representations in each dataset, these results highlight the robustness of our method, proving is not dependent on a specific body pose structure. Furthermore, the larger performance gap on the larger InterX dataset suggests that InterMask scales effectively with more data. While our MModality is slightly lower than some methods, the high R-Precision and low MMdist emphasize that InterMask prioritizes adherence to text over extreme diversity, yet still achieves high Diversity scores, demonstrating its ability to generate a broad range of interactions. Please refer to appendix for visual demonstration of InterMask's 's ability to generate diverse results from the same text prompt.
7

---

Published as a conference paper at ICLR 2025
InterGen
InterMask
person right leg aggressively
Two people bow eachother
One person sitting and waving their hands at the ather person, while the other drifts away
Figure 4: Qualitative comparison between InterMask and InterGen (Liang et al. 2024 highlighting InterMask's ss superior interaction quality, text adherence and avoidance of implicit biases.
5.0
100
Input Modality
Token Map
Recon FIDJ
MPIPE
User Prede Preeerncc (40
4.00
Individual Motion
1d
3.146
0.354
$a.9
Two-Person Interaction
0.198
30.88
Individual Motion
2d
0.970
0.129
2.5
Table Ablation Study results on InterHuman test set to verify key components of the proposed Motion VQ-VAE. Bold face indicates the best result.
2.0
Tike actan
Adheronce Text
USeT
Quality
Figure 5: User Study comparing our Inter- Mask and InterGen (iang et al. 2024).
Qualitative Comparison In Figure we provide qualitative comparison of interaction sequences generated by our InterMask and InterGen [iang et (2024), trained on the InterHuman dataset, for the same text descriptions. For the prompt "The first person raises the right leg aggressively towards the second", InterGen incorrectly raises both legs, while InterMask accurately raises only unnecessary fighting
the right leg. For "Two people bow to each other", InterGen introduces gesture after the bow, indicating overfitting to implicit biases in the training data, whereas InterMask stays true to the simple bow. Lastly, for "One person is sitting and waving their hands at the other person, while the other drifts away"" InterGen generates a crouching pose instead of sitting and misplaces the waving action, while InterMask faithfully generates the described scene. These examples demonstrate that InterMask produces more higher quality and more reliable interactions compared InterGen. For more qualitative results please refer appendix A
User Study User studies are essential in generative AI, complementing metrics like FID by incor- porating human perception to evaluate the realism and quality of generated content. To compare our InterMask with InterGen (Liang et al. 2024), we conducted a user study with 16 participants, where 30 interaction sequences were generated by both models with identical motion lengths. Each sequence pair was rated by 3 distinct users on interaction quality and adherence to text (scores out of 5), and their preferred interaction sequence among the two. More details are provided in appendix K As shown in Figure 5 InterMask outperforms InterGen in interaction quality (4.089 3.296) and text adherence (3.938 vs. 3.198), with 19.14% of users preferring InterMask over InterGen.
Computation and Time Cost In addition to superior generation fidelity and text adherence, Inter- Mask demonstrates better computational efficiency when compared to InterGen (Liang et al.) 2024). InterMask contains only 74 M inference parameters, compared to 182 M for InterGen, resulting in a considerably smaller model footprint. Furthermore, InterMask achieves an average inference time of 0.77 seconds, less than half of InterGien's 1.63 seconds, highlighting its low latency.
8

---

Published as a conference paper at ICLR 2025
4.2
ABLATION STUDIES
Method
R Procision"
FID,
MM Dist
Diversity->
MModalityy
Top
Top
Top
Ground Truth
0.452 608
0.610 000
0.701
0.277
3.755
7.948 0044
Alternatvve Modeling
0.3:0p4.00
0.425 000
0.51 1o0s
7.637 1o7
3,937 u00
8.424
2.192 fo2tt
Collaborative Modeling
wro Self Attention
0.393 n0s
0.5274 000
0. 0.30640077
5.715
3. 3,827 oca
7.873 034
1.68340076
w/o Cross Attention
0.3226
0.416 ntte
0. 0.508kk0m 2007
8.461 1 ooo
3.972 uoo
7.951
1.9424.076
wlo S Spatio-Torn SptiooTeamprral Atteution
0.296--000
0.391 007
0.470
10. 00.968-400 4-033
4. 4.219
7.659 023
1. 1.607
Step Unroll Interaction Masking
0.4012#.00
0,548 004
0 0.6334000
5.62y 063
3.806 002
7.912 0000
1.741
InterMask
0.449
0,599
0,683
5,154
3.790
7.911 038
1.7375.000
Table 3: Ablation Study results on the InterHuman test set to verify key components of the proposed Inter-M Transformer. Bold face indicates the best result.
Table 2 ablation study to evaluate key components of our proposed Motion VQ-VAE. We investigated two primary factors: the input modality (individual motions vs two- woo-person interactions) and the token map dimensions (Id vs 2d), using Reconstruction FID and MPJPE (Mean Per Joint Position Error). The results demonstrate that using individual motions as input with a 2D token map outperforms both the baseline of individual motions with a ID token map, and the two-person interaction with a 2D token map. These findings suggest that preserving spatial information through 2D token maps and focusing on individual motions performs accurate motion reconstruction, and creates a more expressive token map with high-fidelity movement details.
Table presents an ablation study to evaluate key components of our Inter-M Transformer. We compared collaborative modeling against alternative modeling and examined the impact of various attention mechanisms and masking strategies. Alternative Modeling (appendix [ models tokens of one person at time, while conditioning in on the tokens of the other person, leading to an al- ternative generation during inference. Collaborative Modeling on the other hand, models tokens of both individuals simultancously. Our full InterMask model with collaborataive modeling outper- forms all ablated versions, particularly in R-Precision and FID scores. Notably, removing any of the attention mechanisms or employing alternative modeling severely degraded performance, with temporl attention being the most crucial one. Additionally, step unroll and interaction mask- ing proved to be further fine-tuning the results. These results highlight the synergistic importance of all components in our transformer architecture for effective interaction modeling. Qualitative results for selected ablation studies are shown in appendix H
4.3 APPLICATION: REACTION GENERATION
One of the key strengths of our InterMask model is its seamless support for the task of reaction gen- cration with or without a text condition, where the motion one person in an interaction is generated based the provided reference motion of the other person. Unlike previous diffusion-based mod- els, such as InterGen (Liang et al. 2024), InterMask can perform this task without any task-specific fine-tuning or architectural re-design. This flexibility arises from our use of the masked modeling technique combined with an individual motion tokenizer, which does not require interactions as in- put. During inference, we simply tokenize the reference motion, leave its tokens unmasked, and progressively generate tokens of the other person. More details are provided in appendix !|]
Quantitatively, InterMask achieves an FID of 2.99 and an R-precision of 0.462 on InterHuman for reaction generation. In comparison, InterGen suffers from performance drop with an FID of 52.89 (vs 5.918 for interaction generation) and an R- R-precision of 0.194 (vs 0.371) when adopted for re- action generation without fine-tuning, demonstrating its inability to perform reaction generation out-of-the-box. As shown qualitatively in Figure 6 InterMask produces high-quality reaction mo- tions. In cases without text conditioning, when the reference motion involves dancing, the generated motion follows the reference steps and even captures nuanced details like raising the same leg or arm. Similarly, for the physical combat scenario, when the reference motion attacks, the generated motion retreats and vice versa. For text-conditioned samples, InterMask performs equally well, whether the text describes both people's motions or provides separate descriptions for each. The

---

Published as a conference paper at ICLR 2025
These two take step away from eachother strech their
One Person throws toy to the other person, and the other grasps it
One person approaches the other
Reference Motion
Generated Motion
Figure 6: Reaction generation samples with and without text descriptions, including holistic instruc- tion for both individuals and separate instructions for each.
model accurately identifies the required reaction form text and the reference motion and generates the corresponding motion, maintaining high-quality poses and realistic timings throughout.
5 CONCLUSION
In this paper, we presented InterMask, a novel framework for generating realistic 3D human interac- tions through generative masked modeling in the discrete space. InterMask first employs a VQ-VAE to transform individual motion sequences into 2D discrete token maps, preserving both spatial and temporal dimensions. Then, a specialized Inter-M transformer collaboratively models the motion kens of both individuals, capturing intricate spatial and temporal dependencies within and between them. Quantitative evaluations show that InterMask excels in generating realistic and contextually aligned motions, achieving the lowest FID scores and superior results in R- R-Precision and MMdist on both the InterHuman and InterX datasets compared to existing models. Additionally, user studies comparing InterMask to InterGen reveal 69% user preference for InterMask. noting higher-quality and more realistic interactions. The flexibility of our approach is also evident by its ability to per- form equally well for different body representations. While there minor trade-offs in diversity (MModality), InterMask consistently maintains high quality and adherence to text while generating wide range of realistic interactions.
Limitations and Future Work While InterMask generates high-quality human interactions, there some limitations. First, when visualized as SMPL Loper al. (2015) meshes, issues such as body penetration between individuals and jerky movements during rapid actions can occur. Second, our model sometimes interprets motions as dances without explicit prompting, likely due implicit biases in the training dataset. Lastly, our approach is currently optimized for short sequences up to 10 seconds, reflecting the constraints the dataset. Visual examples are shown in appendix[] Future research directions include implementing techniques for smoother transitions, preventing penetra- tions, addressing dataset biases, and expanding the model's 's capacity to handle longer sequences.
Reproducibility Statement We have made our best effort to ensure reproducibility. including but not limited to: 1) description of our implementation details in appendix B 2) detailed graphic illus- trations of our model architectures, training mechanisms and inference processes in Figures 2 and 13 and 4) public release of code and model checkpoints (appendix
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Andreas Aristidou, Daniel Cohen-Or, Jessica K Hodgins, Yiorgos Chrysanthou, and Ariel Shamir. Deep motif's and motion signatures, ACM Transactions on Graphics (TOG), 37(6):1-13, 2018.
Zhongang Cai, Jianping Jiang, Zhongfei Qing, Xinying Guo, Mingyuan Zhang, Zhengyu Lin, Haiyi Mei, Chen Wei, Ruisi Wang, Wanqi Yin, et al. Digital life project: Autonomous 3d characters with social intelligence. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 582-592, 2024.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freemann. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11315-1122,, 2022.
Xin Chen, Biao Jiang. Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu. Executing your commands via motion diffusion in latent space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18000-18010, 2023.
Baptiste Chopin, Hao Tang, Naima Otberdout, Mohamed Daoudi, and Nicu Sebe. Interaction trans- former for human reaction generation. IEEE Transactions on Multimedia, 25:8842-8854, 2023.
Ke Fan, Junshu Tang, Weijian Cao, Ran Yi, Moran Li, Jingyu Gong, Jiangning Zhang, Yabiao Wang, Chengjie Wang, and Lizhuang Ma. Freemotion: A unified framework for number-free text-to-motion synthesis. arXiv preprint arXiv:2405. 15763, 2024.
Anindita Ghosh, Rishabh Dabral, Vladislav Golyanik, Christian Theobalt, and Philipp Slusallek. Remos: 3d motion-conditioned reaction synthesis for two-person interactions. European Con- ference Computer Vision (ECCV), 2024.
Kehong Gong, Dongze Lian, Heng Chang, Chuan Guo, Zihang Jiang, Xinxin Zuo, Michael Bi Mi, and Xinchao Wang. Tm2d: Bimodality driven 3d dance generation via music-text integration. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 9942-9952, 2023.
Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5152-5161, 2022a.
Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng. Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts. In European Conference on Computer Vision, pp. 580-597. Springer, 2022b.
Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. Momask: Gener- ative masked modeling of 3d human motions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1900-1910, 2024.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrIPS 202/ Workshop Deep Generative Models and Downstream Applications, 2021. URUhtttps://ppeeerii.
n www.fooruiiiioomoo do
Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as
foreign language. Advances in Neural Information Processing Systems, 36:20067-20079, 2023.
Jihoon Kim, Jiscob Kim, and Sungjoon Choi. Flame: Free-form language-based motion synthesis & editing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp- 8255 8263, 2023a.
Sungwoong Kim, Daejin Jo, Donghoon Lee, and Jongmin Kim. Magvlt: Masked generative vision- and-language transformer. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp.
2023b.
Hanyang Kong, Kehong Gong, Dongze Lian, Michael Bi Mi, and Xinchao Wang. Priority-cente centric human motion generation discrete latent space. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 14806-14816, 2023.

---

Published as a conference paper at ICLR 2025
Han Liang, Wenqian Zhang, Wenxuan Li, Jingyi Yu, and Lan Xu. Intergen: iiffusion-based multi- human motion generation under complex interactions. International Journal of Computer Vision, pp. 1-21, 2024.
Yunze Liu, Changxi Chen, and Li Yi. Interactive humanoid: Online full-body motion reaction syn- thesis with social affordance canonicalization and forecasting. arXiv preprint arXhv:2312.08883, 2023.
Yunze Liu, Changxi Chen, Chenjing Ding, and Li Yi. Physreaction: Physically plausible real- time humanoid reaction synthesis via forward dynamics guided 4d imitation. arXiv preprint arXiv:2404.01081, 2024.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned
linear model. ACM transactions graphics (TOG), 34(6):1-16, 2015.
Yunhong Lou, Linchao Zhu, Yaxiong Wang, Xiaohan Wang, and Yi Yang. Diversemotion: Towards diverse human motion generation via discrete diffusion. arXiv preprint arXiv:2309.01372. 2023.
Thomas Lucas, Fabien Baradel, Philippe Weinzaepfel, and Grégory Rogez. Posegpt: Quantization- based 3d human motion generation and forecasting. In European Conference on Computer Vision, pp. 417-435. Springer, 2022.
Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black. Amass: Archive of motion capture as surface shapes. In Proceedings of the IEEE/CVF interna- tional conference computer vision, pp. 5442-5451, 2019.
Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dim- itrios Tzionas, and Michael J. Black. Expressive body capture: 3d hands, face, and body from single image. Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 4195 4205, October 2023.
Mathis Petrovich, Michael J Black, and Gül Varol. Temos: Generating diverse human motions from textual descriptions. In European Conference on Computer Vision, pp. 480-497. Springer, 2022.
Ekkasit Pinyoanuntapong, Pu Wang, Minwoo Lee, and Chen Chen. Mmm: Generative masked motion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1546-1555 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748 -7648. PMLR, 2021.
Jiawei Ren, Mingyuan Zhang, Cunjun Yu, Xiao Ma, Liang Pan, and Ziwei Liu. Insactor: Instruction- driven basse characters. Advances in Neural Information Processing Systems, 36, 2024.
Pablo Ruiz-Ponce, German Barquero, Cristina Palmero, Sergio Escalera, and José Garefa- Rodríguez. in2in: Leveraging individual information to generate human interactions. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1941-1951, 2024.
Yoni Shafir, Guy Tevet, Roy Kapon, and Amit Haim Bermano. Human motion diffusion as a gen- erative prior. In The Twelfth International Conference on Learning Representations, 2024. URL
ew. net forum?id=ddTpb..d
Mengyi Shan, Lu Dong, Yutao Han, Yuan Yao, Tao Liu, Ifeoma Nwogo, Guo-Jun Qi, and Mitch Hill. Towards open domain text-driven synthesis of multi-person motions. In European Conference on Computer Vision (ECCV), 2024.
12

---

Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Zi- wei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(6):4115 4128, 2024a. doi: 10. 10.1000/PAMM.222.335555 Published as a conference paper at ICLR 2025
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2021. URL httpttt://ppeeeeeiiiiiiie net/
Eorum? Corumiid=stllgaarHii
Mikihiro Tanaka and Kent Fujiwara. Role-aware interaction generation from textual description. In ICCV, 2023.
Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffusion model. In The Eleventh International Conference on Learning Repre- sentations, 2023. URL / http://aaaeeeeeeeeeeeeeeoeeoneea foaee//uoommmnnnsssSooo...
net
Jonathan Tseng, Rodrigo Castellon, and Karen Liu. Edge: Editable dance generation from music. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 448 458, 2023.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in Neural Information Processing Systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances Neural Informa- fion Processing Systems, 30, 2017.
Zelun Wang and Jyh-Cham Liu. Translating math formula images to latex sequences using deep neural networks with sequence-level training. International Journal on Document Analysis and Recognition (IJDAR), 24(1):63-75, 2021.
Liang Xu, Xintao Lv, Yichao Yan, Xin Jin, Shuwen Wu, Congsheng Xu, Yifan Liu, Yizhou Zhou, Fengyun Rao, Xingdong Sheng, et al. Inter-x: Towards versatile human human interaction anal- ysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22260-22277 2024a.
Liang Xu, Yizhou Zhou, Yichao Yan, Xin Jin, Wenhan Zhu, Fengyun Rao, Xiaokang Yang, and Wenjun Zeng. Regennet: Towards human action-reaction synthesis. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 1759-1769, 2024b.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong
Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved
VQGAN. In International Conference on Learning Representations, 2022. URL hctps:
//openreview. net. wett/forrmmiddpfffio
Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xi Shen, and Ying Shan. Generating human motion from textual descriptions with discrete representations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14730-14740, 2023.
Yaqi Zhang, Di Huang, Bin Liu, Shixiang Tang, Yan Lu, Lu Chen, Lei Bai, Qi Chu, Nenghai Yu, and Wanli Ouyang. Motiongpt: Finetuned ilms are general-purpose motion generators. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 2024b.
13

---

Published as a conference paper at ICLR 2025
A SUPPLEMENTARY MATERIAL
In addition to the subsequent appendix sections, we provide the following supplementary material for this work:
Demo Videos: provide several demonstration videos showcasing visual interaction results generated using InterMask. These include an animation gallery with generated in- teractions for everyday actions, dance and combat; nuanced description demonstration to highlight the capability of Intermask to follow specific details in text; diversity demon- stration with multiple generated interactions for each text prompt; comparison videos to compare the generated results of InterMask and InterGen (Liang al. [024); comparison videos for the ablation study on Inter-M Transformer; an animation gallery to show reaction generation results; some results on more complex prompts; some results showing longer 10 second interactions; and finally some failure cases and some results addressing the con- cemss on fluidity of the generated motions. The videos are shown in form of webpage, provided as an html file.
Code Implementation: We provide the open-source code implementation of our method ensure reproducibility.
B MOTION VQ- VAE
Our VQ-VAE framework, illustrated in Figure 7 constructs a 2D motion token map to represent individual motion sequences in discrete manner, while retaning both spatial and temporal dimen- sions. The encoder processes motion sequences represented as mp E R xJxd where N is the number of poses, J is the number of joints, and is the joint feature dimension. By employing 2D convolutional layers, the encoder effectively captures spatial and temporal dependencies within the motion data while progressively downsampling both dimensions. The downsampling process achieved through strided 2D convolutions and ResNet blocks, which also use 2D convolutions along with dropout layers. This results in a latent representation of size t, E R=xjixd' where n and j are the downsampled temporal and spatial dimensions, and d' is the latent feature dimension.
The latent representation Ep quantized using learned codebook with C'] entries. Each feature vector & in Ep is replaced by the index of its nearest codebook entry, using the vector quantization process:
q(t,) = arg min l - Cs
CCEO
where e represents the codebook entries.
The resulting quantized representation is a 2D motion token map lp. where each token encodes local spatio-temporal context. This design enables the preservation of both spatial and temporal dimensions in the motion data, enhancing the model's ability to generate realistic and contextually accurate interactions.
C
VQ- VAE GEOMETRIC LOSSES
Equation 10 shows the geometric losses used to train our Motion VQ-VAE to impose physical and geometric constraints on the reconstructed motion, in a data-driven way. The velocity loss Lee! encourages the reconstructed motion sequences to obey the velocity of joints in the ground truth sequences and the bone length loss Ls the distance between adjacent joints. The foot contact loss L. encourages the feet have zero velocity whenever they are in contact with the ground.
14

---

Published as a conference paper at ICLR 2025
Codebook
d'_
mp
2D Encoder
Closest
Codebook Indices
Motion 4p
Tokens
Conv 2d down- sample
Conv 2d down- sample
J,
Conv 2d
(N,J,d'
(2n, 2j,d") d')
Resnet block
(n, J, d')
Resnet block
Conv 2d
(n, d')
mp
tp
Conv 2d
Conv 2d
Dropout
Figure 7: Detailed illustration of the 2d discrete motion token map construction. The 2d encoder, consisting of 2d convolutional layers, downsamples the input motion from (N, d) to (n, j, d'). The downsampled representation is then quantized by replacing each vector with the index of its closest vector in the learned codebook.
Interaction Generation
Reaction Generation
FID 4
R Prec (Topl)
FID +
R Prec (Top1)
0.7
5.214
0.447
2.850
0.476
0.8
5.154
0.449
2.991
0.462
0.9
5.152
0.450
3.368
0.416
Table Interaction Generation and Reaction Generation results of different values of pr. Bold face indicates the best result, while underscore refers the second best.
N N
1
Lvel
(ma - mz,)) - (mi,+1 -m;,)l
Lse
-fi
(10)
N
i,=1
1 Lu N
Σ
B(m...)) B(m.,)ll
Here, m,, represents the ground pose, mi, represents the reconstructed pose at time step i,,, N represents the total time steps in the sequence, fi., {0, 1} represents the binary foot contact label for the heel and toe joints for each pose mi,, and (-)) denotes the bone lengths joining adjacent joints.
D TWO-STAGE TOKEN MASKING
Here we provide more details on the masking strategy (section 3.2.1 used training the Inter- M Transformer. As illustrated in Figure the two-stage masking technique begins with random
15

---

Published as a conference paper at ICLR 2025
Stage
Stage 2
f
fg
fg
Step Masking
Masking
Predict
with prob
86
86
ip
fa
with prob
f.
f
f
Interaction Masking
Predict
Step Masking
Predict
Person tokens
Person tokens
MAAKK tolee
Figure 8: Illustration the two-stage masking technique used during training the Inter-M Trans- former. For stage either apply Random Masking with probability of pr or Interaction Mask- ing with probability 1 - Pr. In stage 2, we apply the Step Unroll Masking on the predicted tokens from stage 1.
masking or interaction masking in the first stage. Random masking teaches the model to predict random tokens from both individuals. Whereas Interaction masking, where only individual's tokens are masked, promotes learning inter-person dependencies critical for coherent interactions and to improve performance in the reaction generation task. The masking strategy alternates between these two methods based on probability parameter pr, with random masking applied of the time and interaction masking (1 P.). To evaluate the effect of Pr, we different values 0.7, 0.8, 0.9 and find that p, 0.8 offers the best balance between interaction and reaction generation, as shown Table In the second stage, step unroll masking applied which retains some of the predicted tokens from stage 1, remasks the remaining tokens and predicts them again. This is employed to incorporate the inference-ting -etencee-tmmeeeee progressive refinement of tokens in the training process.
E IMPLEMENTATION DETAILS
Our models are implemented using PyTorch, with details of the model architecture, training, and inference provided below. Key hyperparameters are summarized in the accompanying tables.
E. MODEL ARCHITECTURE
The Motion VQ-VAE employs 2D convolutional residual blocks for both the encoder and decoder. The temporal downsampling factor is n n/N 1/4 for both datasets, while the spatial downsampling is dataset-specing j/J 5/22 for InterHuman and j/ = /56 for InterX. Strided convolutions are used for downsampling in the encoder. while the decoder restores dimensions via upsampling and convolutional layers. The latent representation in VQ-VAE has dimension d' = 512, and the codebook size [C| 1024.
For the Inter-M M transformer, use L 6 transformer blocks, each with 6 attention heads. The transformer embedding dimension 384.
E.2 TRAINING DETAILS
The Motion VQ- VAE is trained for 50 epochs with a batch size of 512. The learning rate is initialized at 0.0002 and decays via multistep learning rate schedule, reducing by a factor of 0.1 after 70% and 85% of the iterations. A linear warm-up is applied for the first quarter of the iterations. The
16

---

Published as a conference paper at ICLR 2025
Parameter
Value
Description
512
Latent space dimension of VQ-VAE
[C]
1024
Codebook size (number of entries)
n/N N
1/4
Temporal downsampling factor for both datasets
j/J (InterHuman)
5/22
Spatial downsampling for InterHuman dataset
j/J (InterX)
5/56
Spatial downsampling for InterX dataset
L
6
Number of transformer blocks
Attention heads
6
Number of attention heads per block
d
384
Transformer embedding dimension Version of CLIP used for text in transformer
CLIP version
(TT-L/14@336ps
Table 5: Motion VQ-VAE and Inter-M Transformer Model Parameters
commitment loss factor 3 is 0.02, and the geometric losses for velocity, foot contact, and bone length are weighted differently across the datasets.
The Inter-M transformer is trained for 500 epochs with batch size of 52, following similar multi- step learning rate decay but with a decay factor of 1/3 after 50%, 70%, and 85% of the iterations. The condition drop probability is 0.1 to allow for flexibility in training with or without text conditioning.
Parameter
Value
Description
VQ-VAE hatch size
512
Number samples per hatch for VQ-VAB
Transformer batch size
52
Number of samples per batch transformer
Initial learning rafe
0.0002
Starting learning rate both models
Leurning rate decay
1/3
Decay factor for VQ-VAE Transformer learning rale Commitment loss factor for VQ-VAE VAE
0.02
Austs Afe, An (InterHuman)
100, 500,
Geometric loss weights for InterHuman Geometric loss weights for InterX dataset
Avels Affe Ay (InterX)
100, 100, 5
Condition drop prob.
0.1
Drop probahility for text conditioning during transformer training
Pr
0.8
Random Masking prohability for stage masking during training
Table 6: Training typeraaramesers for the Motion VQ-VAE and Inter-M Transformer
E.3 INFERENCE DETAILS
During inference, the number of iterations / is set 20 for interaction generation and 12 for reaction generation. A classifier-free guidance (CFG) scale of is applied, and the temperature is set to 1 to balance diversity and coherence the generated results.
Parameter
Interaction Generation
Reaction Generation
Number of iterations
I -= 20
Preact 12
CFG scale Temperature
1
Table Inference Hyperparameters for Interaction and Reaction Generation
F DIVERSITY DEMONSTRATION
Our quantitative comparison (section 4. shows that InterMask prioritizes adherence to text over extreme diversity, while still being able to generate different distinct interactions for the same text prompt. Here, in Figure we show 2 visual examples to demonstrate this. In both cases, the model remains consistent in generating interactions described in the text prompt while exhibiting distinct features in different samples. For the dancing case, first sample shows waving hands in the beginning followed by synchronized forward step, while individuals face the same dircction throughout. The second sample shows individuals facing each other in the beginning, followed by waving hands and
17

---

Published as a conference paper at ICLR 2025
Two people are waving their hands and performing dance stop together
boxing match, continuously punching defending counterattacking
Figure 9: Diversity Demonstration of our method, where it generates two distinct interaction quences the same text prompt.
a synchronized spin. For the boxing case, first sample shows one individual punching three times and continuously moving forward, while the second sample shows them punching two times and retreating at the end.
G ALTERNATIVE MODELING
In our ablation study (section we explore a one-at-a-time modeling framework for interaction generation, referred Alternative Modeling, which contrasts with the collaborative modeling framework of InterMask. While collaborative modeling predicts the tokens of both individuals si- multaneously, alternative modeling follows sequential process, generating tokens for one individ- ual at a time, conditioned on the thus far predicted tokens of the other.
During training, shown in Figure 10'a), randomly mask both individuals' tokens (ča, & and obtain their embeddings {e., e,} through the input process. Then, only the tokens of one individual are predicted (f,) by passing their embeddings through the transformer blocks, conditioned on the embeddings of the other individual using cross-atiention module. During inference (Figure ()))), both individuals" tokens are initially fully masked {t, (0), & (0)). In the first iteration, the tokens of one individual are predicted, and these are remasked based on their confidence scores to obtain (1). The second individual's tokens are then predicted in the next iteration, conditioned on the retained tokens from the first, to obtain (1). This alternation continues iteratively, progressively refining the tokens of both individuals. As shown in Table the FID score for alternative modeling 7.637 (compared to 5.154 for collaborative modeling). and the R-precision is 0.340 (compared to 0.449). These results indicate that while alternative modeling increases diversity, collaborative modeling produces high-quality and more realistic interactions, offering a better balance between diversity and interaction fidelity.
18

---

Published as a conference paper at ICLR 2025
Person a tokens
Person tokens
[AAK token
Ade-LN
Predicted
fone entoooee strike porsor"
Tokens
7D Foxtioonal
Prending
Enout Process
CLIP
Inter-M Transformer
Enpudt Procett
Self Attn
Temporal
Altin
Allin
FFN
MIL
XL blocks
Alternative Modeling Training
fo(0)
Inter-M Transformer
(1)
(1)
f
X2A
112,
atvances the other person"
Transformel
Laber steps
Codebook
Decocer
Inteer
Remask
(0)
(a
(1)
(b) Altemative Modeling Inference
Figure 10: Overview the Alternative Modeling approach, where we predict the tokens of one person at a time. (a) During training. only the embeddings of one individual are updated in the transformer blocks, conditioned on the other individual's embeddings cb. (b) During inference, the process alternates between predicting and remasking the tokens of each individual, starting with both fully masked (4a (0)). This process continues for Latter iterations for each individual.
Motion Sample
Motion Sample
Ground Truth
20 VQ Tokan
10 VQ Taken Map
Figure 11: Qualitative results for the ablation study Motion VQ-VAE to verify the proposed 2D token map.
H ABLATION STUDY QUALITATIVE RESULTS
In this section, we present qualitative results for our ablation studies to complement the quantitative findings discussed in section Figure shows two ground truth interaction sequences with their reconstructed samples from the 2D token map VQ-VAE and the baseline ID token map VQ-VAE. As shown, the ID VQ-VAE struggles to accurately reconstruct the spatial positions and orientations of the joints for both individuals, leading to incorrect positioning and orientation relative to each other. This results in not only unrealistie interactions but also bizarre individual poses. In contrast, the proposed 2D VQ- VAE provides highly accurate reconstructions at both the individual pose and the collective interaction level.
19

---

Published as a conference paper at ICLR 2025
unt dip doe
Oreas
Fardest
Qualitative results for the ablation study on Inter-M Transformer to verify contributions of the proposed Attention modules.
Figure 12:
Figure illlstrrate the impact of different attention modules our Inter-M transformer by compar- ing outputs by removing the spatio-temporal attention, cross-attention, and self-attention modules independently. We provide results for three distinct interaction scenarios: boxing, sneaking, and synchronized dancing. The spatio-temporal attention module emerges critical component for generating complex poses and ensuring spatial awareness in interactions. Without this module, the boxing scenario exhibits overly simplistic poses, such timid punching and blocking, alongwith the individuals failing to face each other properly. In the sneaking scenario, the absence of spatio- temporal attention eliminates the essential spatial progression, the sneaking individual does not gradually approach the other. Similarly, in the dancing scenario, the generated motions are reduced basic hand raises, and one individual adopts an unnatural crouching pose. By contrast, the inclu- sion of spatio-temporal attention enables accurate spatial positioning and expressive, synchronized interactions. The cross-attention module seems vital for modeling inter-person dependencies, par- ticularly in achieving accurate reaction timing. Without it, the response motions of the interacting individual appear either delayed or prematurely executed across all examples. For instance, in the boxing scenario, the reactive movements fail to synchronize with the initiating individual's 's punches. In the dancing scenario, the lack of cross-attention results in poor synchronization. disrupting the fluidity of the interaction. Lastly, the self-attention module serves hancing the overall quality and coherence of individual motions. Its removal introduces subtle in- consistencies, such jerky transitions less fluid movements, which slightly degrade the interac- tion's 's realism. These observations collectively underscore the importance of each attention module in generating realistic, contextually accurate, and expressive interactions.
refinement mechanism,
I REACTION GENERATION INFERENCE
InterMask does not require task-specific fine-tuning or architectural re-design for reaction gener- ation, needing only minor adjustments the inference process, illustrated in Figure 13 The process begins by encoding the reference individual's 's motion m, into tokens &s using the VQ-V VQ-VAE encoder. For the other individual, whose reaction be generated, we initialize with fully masked token sequence, L, (0). Over the course of Freac iterations, the transformer progressively predicts and fills in the masked tokens, while the reference tokens remain unmasked throughout. At each iteration iram
Freaet the least confident 7
njj tokens are remasked and predicted again, following cosine Traas
scheduling function 7.)) Once all tokens are generated, the final token sequence f, is decoded back into motion m2 using the VQ-VAE decoder. Since we drop the conditioning signal during some training passes, reaction generation functions effectively both with and without text description, enabling the model to generate motions based solely on the reference motion or guided by additional textual instructions.
20

---

Published as a conference paper at ICLR 2025
(1
Inter-M Transforme
stepe
Inter-M Transformer
Codecook
Encader
VQ
Remask
20 Decoder
CLIP
Person tokens
Person tokens
[SHP] token
MASK token
Figure 13: Inference process for the Reaction Generation task. The motion tokens of the reference individual fú are obtained from the encoder and kept unmasked throughout. The second individual's tokens initially fully masked la (0), and are predicted progressively over Irenct iterations obtain la which then decoded using the decoder.
First person is sitting chair, the second takes a step forward with their right foot
first step their foot
Figure 14:
Examples of Limitations of our method. The first row shows body penetration when con- verted from output skeleton SMPL mesh. The second row shows implicit bias towards dancing.
FAILURE CASES
In Figure 14 we show visual examples of failure cases emerging from the limitations of our method, as decribed in section 5 In the first row, we show that when converting our output skeletons to
SMPL (Loper et al. 2015) meshes for visualizations, the results can exhibit body penetration among the interacting individuals. One possible future solution to this problem to include the mesh conversion in the training process and incorporate nntiipenntration in the training loss. In the second row, we show that the model suffers from some implicit biases present in the dataset, where assumes that the individuals are dancing without explicit mention in the text prompt.
21

---

Published as a conference paper at ICLR 2025
K
USER STUDY
We conducted the user study on the Amazon Mechanical Turk platform, where the interface pre- sented to the users is shown in Figure 15 For each sample, users were provided with clear instruc- tions to rate two animations 2024) from the same text description, with the order of the animations randomized for each sam- ple to avoid bias. Participants were asked to rate both animations on scale of to for interaction quality and text adherence. Following the individual ratings, they animation based on their overall impression. each sample being rated by three users. To ensure high-quality feedback, we filtered users to include only those with Amazon Mechanical Turk master status, with task approval rating of over 97% and more than 1000 previously approved tasks.
generated by InterMask and the other by InterGen (Liang et al.)
asked select the better total of 16 users evaluated a total 30 samples, with
dissc iplion fsrevidled below). provide Interaction animation resula, Aniimation video ahome Imo viaiilly
oifferent point
need
their Quality Adherence
Interaction natural rexibett people interaction? Doss and muacion
quallly 1f mis quality
Animation
Animation
Animation
Preferance- Overal witc animation better your preferenos besed above fecions
Animerdor Anmodoo 2
Description:
boxing continuoucly punching other defencing counteratten cousslratbccking
Animation
Front View
View
Animation
Viea
Figure 15; Interface of the User Study on Amazon Mechanical Turk.
22


Reviews:
[{"review_id": "ofRoH1Y3DJ", "reviewer": "Reviewer_jS9r", "summary": "This article introduces a new framework called InterMask for generating 3D human interaction actions from text descriptions. The proposed framework appears to be based on the T2M-GPT framework, modeling in discrete space using generative masks, while also incorporating the concept of interhand to collaborate within the mask modeling framework by utilizing tokens from two interacting individuals. During the inference process, it starts with a fully masked sequence and gradually fills in the tokens for both individuals.", "questions": "1. I noticed that in the second video of the visualization \"Everyday Actions,\" the character on the right makes a sudden arm-raising motion, which seems somewhat disconnected. Additionally, is there a 10-second video available for reference? I saw that the authors mentioned the ability to generate interactive results up to 10 seconds long.\n\n2. In terms of quantitative experiments, it seems there is no comparison with PriorMDM, which also produces results for two-person interactions. Similarly, there doesn't appear to be a comparison with other VQ-VAE methods like T2M-GPT. I'm curious about their results, especially since they perform very well on single-person datasets. If T2M-GPT were given a reference action sequence for one individual as an additional condition, how would it perform in generating interactive individuals, particularly on interactive datasets?\n\n3. Regarding the Inter-M Transformer structure, although there is some evidence in the ablation studies concerning the relative importance of different attention mechanisms, it may not fully clarify their specific roles and trade-offs in different interaction scenarios and data distributions.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. A Transformer structure combined with VQ-VAE has been proposed for generating 3D human interaction actions. \n2. It utilizes a 2D discrete motion token map to represent motion, distinguishing itself from traditional 1D representation methods. This representation better preserves spatiotemporal details and spatial awareness, which is crucial for accurately capturing complex motion relationships in human interactions. \n3. The framework has achieved state-of-the-art results on the InterHuman and InterX datasets, performing exceptionally well on various metrics. \n4. The average inference time is only 0.77 seconds, significantly lower than InterGen's 1.63 seconds, providing a considerable advantage in practical applications by enabling faster result generation.", "weaknesses": "1. In the visualization results, issues such as body penetration between interacting individuals and unnatural sliding movements still occur.\n2. The article also mentions the masking strategy, but I believe its effectiveness may be influenced by various factors, such as the setting of the mask ratio and the dynamic adjustment of masks at different stages. The article does not delve deeply into how these factors affect model performance or how to optimally set these parameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ji0euUneK2", "reviewer": "Reviewer_LgJ2", "summary": "The paper presents InterMask, a framework for generating 3D human interactions using masked modeling in discrete space. By encoding motion as 2D token maps and modeling intra- and inter-person dependencies with a custom transformer, InterMask achieves impressive results on the InterHuman and InterX datasets. The model shows strong performance in terms of realism and alignment with text prompts, outperforming existing methods in controlled settings.", "questions": "1. Could the authors comment on the suitability of the current framework for multi-person interactions? Given the focus on two-person setups, it would be useful to understand any potential modifications or challenges in adapting this approach for more complex, multi-person interaction scenarios.\n\n2. How does the model handle longer, more complex instructions or instructions that are less structured (in-the-wild)? \n\n3. Do the authors see potential for combining InterMask with language models to improve interpretability or to handle a wider range of textual prompts?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The use of 2D token maps and spatial-temporal attention layers in the Inter-M transformer enables more nuanced modeling of human interactions, demonstrating improved fidelity in motion generation.\n\n2. InterMask achieves state-of-the-art results on key metrics in standard datasets, indicating its effectiveness in generating realistic, text-aligned interactions.\n\n3. The model is relatively lightweight and demonstrates reduced inference time.", "weaknesses": "1. The paper’s innovative contribution is not well-defined. Although the authors introduce a transformer module to model intra- and inter-person spatial-temporal dependencies, this approach is widely used in various generative tasks. The ablation studies, which compare only with/without this module, do not sufficiently demonstrate the model’s unique contribution. If this module is intended as a core innovation, the authors should focus on explaining why its internal structure, arrangement, or sequence specifically suits the 3D interaction modeling problem. Merely showing the presence or absence of the module does not justify its relevance, as it is a fairly standard approach.\n\n2. The process of converting each motion sequence into a 2D discrete motion token map lacks detailed description. The main innovation of this paper appears to lie in the combination of the 2D discrete motion token map with MAE; however, there is insufficient explanation of how the 2D token map is constructed. Providing a more thorough description, ideally with a detailed diagram, would enhance clarity and better communicate this aspect of the method.\n\n3. User Study. The user study includes only three participants, which is insufficient to draw robust conclusions on user preference. A larger sample size and detailed descriptions of the study’s methodology would add reliability and provide stronger validation for claims of user preference for InterMask-generated interactions.\n\n4. Upon reviewing the supplementary videos, the generated motions appear to lack the fluidity seen in InterGen’s outputs. A discussion on this aspect would be valuable, particularly to understand any limitations in InterMask’s approach to generating smooth, natural motion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LQliW0NGSG", "reviewer": "Reviewer_YC4H", "summary": "This paper addresses the problem of generating human-human interaction from textual descriptions. The authors introduce InterMask, a novel framework that employs 2D VQ-VAE to encode interactions into a discrete space and generative masked modeling to capture complex spatio-temporal interdependencies.", "questions": "I have no other questions.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper introduces a novel 2D VQ-VAE that quantizes motion into a 2D discrete map, which can more effectively preserve fine-grained spatio-temporal details than the 1D version.\n\n2. The proposed method achieves SOTA performance on the InterHuman and InterX benchmarks.", "weaknesses": "1. The method does not appear substantially different from prior work. Overall, the paper feels somewhat underwhelming, lacking distinctive insights, though there are no major issues with the experiments or writing. I would rate it as borderline.\n\n2. The proposed approach faces challenges in generating human reactions in an online manner (although this isn’t the primary focus of the current setting). This capability would be more practical for some virtual reality applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)