in2IN: Leveraging individual Information to Generate Human INteractions

Authors: chaitanya  ahuja
louis-philippe  morency
  lan
german  barquero
johnny  núnez
sergio  escalera
zhen  xu
wei-wei  tu
isabelle  guyon
cristina  palmero
  didn'
  compar
jane  bromley
yann  lecun
tom  brown
benjamin  mann
nick  ryder
melanie  sub
jared d kaplan
prafulla  dhariwal
arvind  neelakan- 558
pranav  shyam
girish  sastry
amanda  askell
ariel  agarwal
gretchen  herbert-voss
tom  krueger
rewon  henighan
aditya  child
daniel  ramesh
  ziegler
jeffrey  wu
clemens  winter
chris  hesse
mark  chen
mateusz  sigler
scott  litwin
benjamin  gray
  chess
  jack
christopher  clark
sam  berner
alec  mccandlish
  radford
mingyuan  zhang
zhengyu  lin
haiyi  mei
chen  wei
  ruisi
han  liang
wenqian  zhang
wenxuan  li
jingyi  yu
alex  nichol
aditya  ramesh
  pranav
pamela  shyam
bob  mishkin
ilya  mcgrew
mark chen sutskever
alec  radford
jong wook kim
chris  hallacy
gabriel  goh
sandhini  agarwal
pamela  mishkin
jack  clark
Venue: thecvf_com_CVPR_2024_Workshop_HuMoGen
Published: 2024-03-24 00:30:58

Abstract: Generating human-human motion interactions conditioned on textual descriptions is a very useful application in many areas such as robotics, gaming, animation, and the metaverse. Alongside this utility also comes a great difficulty in modeling the highly dimensional inter-personal dynamics. In addition, properly capturing the intra-personal diversity of interactions has a lot of challenges. 
Current methods generate interactions with limited diversity of intra-person dynamics due to the limitations of the available datasets and conditioning strategies. 
For this, we introduce in2IN, a novel diffusion model for human-human motion generation which is conditioned not only on the textual description of the overall interaction but also on the individual descriptions of the actions performed by each person involved in the interaction. To train this model, we use a large language model to extend the InterHuman dataset with individual descriptions. As a result, in2IN achieves state-of-the-art performance in the InterHuman dataset. Furthermore, in order to increase the intra-personal diversity on the existing interaction datasets, we propose DualMDM, a model composition technique that combines the motions generated with in2IN and the motions generated by a single-person motion prior pre-trained on HumanML3D. As a result, DualMDM generates motions with higher individual diversity and improves control over the intra-person dynamics while maintaining inter-personal coherence.

Main paper: 
 introduction: 
 Human Motion Generation refers to creating synthetic human movements that closely mimic those performed by actual individuals. This field has experienced significant advancements alongside the general progress in generative AI over recent years [56] . However, unlike other areas of generative AI, such as image and text generation, annotated motion datasets are scarce due to the need for expensive recording setups and actors. Controlling the generation of a motion based on a given condition is extremely im-Fig. 1 . To do so, we extend the InterHuman dataset [28] with LLM-generated textual descriptions of the individual human motions involved in the interaction. Our approach allows for a more precise interaction generation and achieves state-of-the-art results on the InterHuman dataset.• We introduce a diffusion conditioning technique based on the Classifier Free Guidance (CFG) [22] that allows weighting independently the importance of each condition during the interaction generation. This enables a higher control over the influence of individual and interaction descriptions on the sampling process.• We propose DualMDM, a new motion composition technique to further increase the individual diversity and control. By combining our in2IN interaction model with a single-person (individual) motion prior, we generate interactions with more diverse intra-personal dynamics. 
text-driven human motion generation: 
 A review of recent literature [56] reveals significant progress in this domain over the past two years, with a plethora of methodologies being explored. The first set of methodologies that have been explored is based on aligning the latent spaces of text and motion using the Kullback-Leibler divergence loss [1, 18, 33, 40] . A decoder is trained to convert the text latent representation into the corresponding motion. The main limitation of these approaches is that the scarcity of motion data might lead to latent space misalignments and therefore semantic mismatches between the text and the generated motion.Based on the recent success of auto-regressive approaches in domains like language, with the advent of LLMs [10, 14, 42, 53] powered by Transformers [44] , new approaches have emerged in the motion field [19, 25, 49, 54] . In these, motions are tokenized into discrete codes from a learned codebook, and a Transformer architecture is used to convert text tokens into motion tokens in an autoregressive manner. While these approaches generate more realistic motions, they have some downsides. Firstly, while tokenizing text is a relatively simple task, tokenizing motion is not straightforward because there are no clear individual logic units as can be the words or lemmas in a text. Additionally, due to the nature of auto-regressive models, they cannot model bi-directional dependencies. MMM [34] and MoMask [20] address this limitation using masked attention in BERT [14] style.Diffusion Models [23, 37] have emerged as the best option for many generative tasks [46] , also achieving excellent results in the text-to-motion field. FLAME [26] and MotionDiffusion [51]  
method: 
 In this section, we introduce our main methodological contributions. First, in Sec. 3.1, we describe in2IN, our proposed interaction-aware diffusion model conditioned on both the interaction and the individual textual descriptions.Then, we introduce the multi-weight CFG technique, which increases the user control over the influence that each condition has over the generation process. Finally, in Sec 3.2, we discuss how our second contribution, DualMDM, can increase the control and diversity of the intra-personal dynamics generated by pre-trained interaction models such as in2IN.The architecture of our interaction diffusion model (in2IN) is founded on the principle that interactions between two persons exhibit a commutative property [28], denoted as {x a , x b }, which is considered to be equivalent to {x b , x a }.Building on this concept, we introduce a Transformer-based diffusion model in a Siamese configuration [9] . Two copies of the diffusion model are made, sharing parameters. Each network is responsible for processing its respective noisy motion inputs, x t a and x t b , and aims to produce the denoised versions, x 0 a and x 0 b . We predict the x 0 directly [7, 45] as this allows us to use kinematic losses. Once the losses have been calculated, the motion is noised back to x t-1 to become the input of the next denoising iteration.Similarly to [28, 39] , our diffusion model architecture (Fig. 2 ) has a multi-head self-attention module where it learns the intra-personal dynamics of the motion, and a multi-head cross-attention module that combines the selfattention output with the motion of the other individual in the interaction, thus modeling the inter-personal dynamics.We also condition the generation with adaptative normal- The equation representing our model's sampling function, 257 denoted as G s (x t , t, c), is as follows: EQUATIONIn our second contribution, we propose a motion model composition technique that allows us to combine interactions generated by an interaction model with the motions generated by an individual motion prior trained with a single-person motion dataset. This method uses a singleperson human motion prior to provide the generated humanhuman interactions with a higher diversity of intra-personal dynamics. This model composition technique is built on top of the method proposed in DiffusionBlending [36]:EQUATIONwhere w ∈ R is the blending weight, G a (x t , t, c a ) and G b (x t , t, c b ) are the outputs of the diffusion models a and b, respectively. Since the original proposal was made to combine single-person diffusion models, we adapt the previous formula to our scenario:EQUATIONwhere G I (x t , t, c) is the output of the interaction diffusion model and G i (x t , t, c i ) is the output of the individual motion prior. By choosing w to be constant, authors from [36] assumed that the optimal blending weight is the same 295 along the whole sampling process. However, in line with[24], we argue that the optimal blending weight might vary  
evaluation metrics: 
 We utilize the evaluation metrics proposed in [17] . R- precisionOur in2IN models consist of 8 consecutive multi-head attention layers with a latent dimension of 1024 and 8 heads.We All models have been trained using the AdamW optimizer [29] with betas of (0.9, 0.999), weight decay of 2 × 10 -5 , maximum learning rate of 10 -4 , and a cosine learning rate schedule with an initial 10-epoch linear warmup period. They have been trained using the L2 loss and, thanks to the use of the x 0 parameterization, kinematic losses have also been used. These include the foot contact and the velocity losses from the MDM framework [41] ,and the bone length, the masked joint distance map, and the relative orientation losses suggested in InterGen [28] . Additionally, we have used the kinematic loss scheduler from InterGen. All models have been trained for 2,000 epochs with a batch size of 64 with 16-bit mixed precision. Two Nvidia 3090 GPUs have been required for the span of 5 days.DualMDM schedulers. We test these functions: constant, orEQUATION)where t is the actual denoising step, T is the total number of denoising steps, and λ is the parameter that determines the slope of our scheduler function. We visualize them in Fig. 3 .  
conclusion: 
 We presented in2IN, an interaction diffusion model that  
 
 Appendix: The code, model checkpoints, and data will be publically released on: censored 


Decision: The paper addresses the task of synthesizing human interaction motion from text.

Pros: 
* Multiple novel components
* Quantitative and Qualitative results outperform current work

Cons:
* Some details are missing in the paper
* During training, the decomposition of the textural description into two individual sentences does not necessarily match the ground truth motions. This issue should be specified as a limitation

**Guidance to authors:** 
* Please resolve concerns raised by the authors, particularly ones related to missing details
* Please ensure that the new dataset (i.e., individual text descriptions) is made publicly available

Reviews: review: title: title: The paper presents very interesting concepts but lack explanations.
main_review: main_review: The authors present intriguing concepts and innovative contributions, including:

* The novel diffusion model architecture, which generates two motions symmetrically conditioned on both individual prompts and general interaction prompts, allowing for greater diversity in each character's behavior, following its specific prompt rather than a general one.
* A new technique is demonstrated to guide generation independently towards each condition via multiple CFG.
* The novel motion composition technique - DualMDM, enables the use of a single motion diffusion model's prior.
* The impressive improvment quantitativly on almost all metrices using the new methods mentioned above.
* The qualitative result outperform current state of the art.

yet lacks comprehensive explanations regarding their workings and the impact of their novelty on the achieved results.
* Regarding the architecture of the model, how does information flows between the two models during generation? how is one inference aware of the generated motion from the second inference?
* during inference, what is the latency of generating one motion? Since the duplication of the model times the 4 CFG and the single motion prior model may result in a lot of resources required for a single generation.
* In Table 1, the authors did not quantify the contribution of DualMDM to the obtained results.
* Figure 2 depicts both individual embedding and interaction embedding entering the CLIP and linear layer as two inputs, resulting in an output dependent on both. However, the textual explanation suggests that they enter separately into the CLIP model and linear layer.

If the papers will be accepted, I ask the Authors to explain the list of points lacking of information in the camera ready copy.

After considering these factors, I incline for a weak accpet.
title: title: Enhancing Motion Generation through Data Integration: A Review with Recommendations for Clarity and Methodological Coherence
main_review: main_review: The authors propose an intriguing approach by integrating individual and interaction datasets to enhance motion generation control for each individual and their interactions. This concept is solid and great and represents a significant advancement in the field. However, the articulation of the paper fluctuates between clarity and ambiguity, impacting the delivery of its contributions.

The structure of the paper is generally satisfactory. Nonetheless, certain aspects require elucidation:

1. Methodological Clarity: The introduction of CFG into both interaction and individual diffusion models is innovative. However, the relationship between equation (1) and equation (3) remains ambiguous. It appears as though these equations originate from disparate methodologies, lacking a coherent integration. This aspect suggests a potential oversight in the methodological framework, which warrants clarification for a unified understanding.

2. Experimental Results and Analysis: The experimental section outlines the influence of the scheduler on the outcomes effectively. Yet, the identical figures in the concluding row of Table 1 and the opening row of Table 2, coupled with the \lambda value set to 0.0, introduce confusion. It implies that when the individual has no effect, the model performs better in terms of r-precision and FID. I believe it is important to conduct further experiments to investigate this phenomenon.

3. Data Utilization and Result Interpretation: The presentation of the results does not fully address the anticipated improvements with the incorporation of more comprehensive data sources. The transparency in reporting Table 2 is commendable; however, the expected enhancement in results with additional data  (both two-person and single-person datasets)  was not observed, lacking a satisfactory explanation for this discrepancy.

4. Writing and Logical Flow: The manuscript's logical flow, particularly in the section spanning lines 342 to 369, could benefit from refinement. The primary concept, though discernible, is presented in a manner that obfuscates its significance. Additionally, the use of certain expressions, such as "though" in line 348, introduces unnecessary confusion. Refining this segment could substantially enhance the comprehension of the EID concept, which is pivotal to the paper.

The paper introduces a compelling concept, yet struggles with clarity, experimental validation, and exposition, leaving me feeling complicated about its current form. Addressing these issues could significantly elevate the manuscript, highlighting its innovative contributions more effectively. This work has undeniable potential; with focused improvements, it could truly impact the field.