Two-Person Interaction Augmentation with Skeleton Priors

Authors: l = l b s + l b m + l
  ||q ′ b -q ′ b ||
||  q′ b -q′ b ||
qianhui  men
edmond s l ho
hubert p h shum
] m müller
t  röder
m  clausen
b  eberhardt
b  krüger
springer  international
] xiaogang peng
siyuan  mao
zizhao  wu
  trajectory
yang  jiang
Venue: thecvf_com_CVPR_2024_Workshop_HuMoGen
Published: 2024-03-18 13:34:18

Abstract: Close and continuous interaction with rich contacts is a crucial aspect of human activities (e.g. hugging, dancing) and of interest in many domains like activity recognition, motion prediction, character animation, etc. However, acquiring such skeletal motion is challenging. While direct motion capture is expensive and slow, motion editing/generation is also non-trivial, as complex contact patterns with topological and geometric constraints have to be retained. To this end, we propose a new deep learning method for two-body skeletal interaction motion augmentation, which can generate variations of contact-rich interactions with varying body sizes and proportions while retaining the key geometric/topological relations between two bodies. Our system can learn effectively from a relatively small amount of data and generalize to drastically different skeleton sizes. Through exhaustive evaluation and comparison, we show it can generate high-quality motions, has strong generalizability and outperforms traditional optimization-based methods and alternative deep learning solutions.

Main paper: 
 motion retargeting: 
 Motion retargeting adapts a character's motion to another of a different size while maintaining the motion semantics. Early research employs space-time optimization based on contact [9] , purposefully-designed inverse kinematics solver for different morphologies [13] , data-driven reconstruction of poses based on end-effectors [4, 40] , or physical filters [43] and physical-based solvers [3] considering dynamics constraints. Recently, deep learning has achieved great success, e.g. recurrent neural networks with contact modeling [49] , skeleton-aware operators without explicitly pairing the source and target motions [2] , and variational autoencoders for motion features preservation during retargeting [48] . Beyond skeletal motions, the skeleton structure is also effective in video based retargeting [60] . Fast deep learning methods are pursued for real-time robotic control [63] . Unlike previous research, we propose a novel deep learning architecture for motion retargeting/generation of two-character interactions, which are intrinsically more complex than single-character retargeting.Interaction retargeting involving more than one person is more challenging than single-body retargeting, due to their complex motion constraints [22] such as topological constraints [14] , but these constraints involve heavy manual designs. As a more general solution, InteractionMesh [16] uses dense mesh structures to represent the spatial re- Recently, there is a surge of deep learning methods on interactions, including human-object interaction [21, 36, 58], motion generation as reaction [6] , from texts [44] and by reinforcement learning [65] . Interaction has also been investigated in motion forecasting [33, 46, 59] . Among these papers, the closest work is interaction motion generation but existing work either cannot deal with skeletons of different template motion, restricting p(q ′ A , q ′ B , B s , qA , qB ) to only model the distribution of offsets from {q A , qB }.There are many ways to factorize p(q ′ A , q ′ B , B s , qA , qB ) theoretically. Our new factorization follows:EQUATIONwhere (i) gives the conditional probability of p(q ′ A |q ′ B , B s , qA , qB ), and its prior p(q ′ B , B s , qA , qB ).Further, p(q ′ B , B s , qA , qB ) can be factorized into p(q ′ B |B s , 
network architecture: 
 In Fig. Instead of directly learning the distribution of q ′ B , learning the distribution of the differences △q B = q ′ B -qB is easier [45, 52] : p(q ′ B |B s , qB ) = p(△q B |B s ), which is easier as it becomes learning the distribution of offsets from the template motion qB and a skeleton variation B s . We encode △q B into a latent space then decode it back to the data space by:EQUATIONwhere in both the encoding and decoding processes, we also incorporate the first and last frame of the template motion q0 B , qT B because they help stabilize the dynamics based on our results. After decoding, we add the predicted △q ′ B back to the template motion to get the new motionq ′ B = q B + △q ′ B .Next, given a motion q ′ B , character A needs to adjust its motions to keep the interaction, leading to a distribution of possible q ′ A . Similarly, we focus on learning △q A = q ′ A -qA by an autoencoder: EQUATIONwhere after decoding we compute the new motion q ′ A = 316 qA + △q ′ A .EQUATION)346where z is the latent variable.ω 4 = 1 -ω 1 -ω 2 -ω 3 , M 347is the total number of motions. q′ A and q ′ A are the predicted 348 and the ground-truth B motion.ω 4 = 1 -ω 1 -ω 2 -ω 3 ,and ω 1 = 0.75, ω 2 = 0.1 and ω 3 = 0.05. BL(q ′ A , q ′ A ) is 350 the same bone length loss as in Eq. 7.To our best knowledge, there are few public datasets fo-cusing on close and continuous interactions except [12] .To construct our dataset, we first obtain base motions and 355 augment them. The base motion details are shown in 356 the SM. We obtain "Judo". From CMU [1] , we choose"Face-to-back", "Turn-around" and "Hold-body". FromExPI [12] , we choose "Around-the-back", "Back-flip","Big-ben", "Noser" and "Chandelle". These interactions  
retargeting and generation: 
 We present the main results here and refer the readers to the SM for more results and details.We first show quantitative evaluation in Tab. 2. Across the two tasks, generation is harder than retargeting, as the bone scales are not given in generation. Naturally, the bone length error E b is almost always slightly worse than Retargeting and so is JPD. But even the worst case is 330% in E b and 206.89% worse in JPD which suggests the model generalizability on unseen scales and interactions in general is strong. We show visual results in Fig. 4 and the video. Together with the scaled skeleton, the poses are automatically adapted on both characters to keep the geometric relations of the interaction.E r E b JPD F ID E b JPD M1 Random 1.In terms of generation settings, the overall difficulty We predict larger scales. The scales are beyond our dataset(including the testing data). We show one example of Turn- To our best knowledge, it is new for deep learning to be employed for interaction augmentation with varying body Here we show two downstream tasks: motion prediction 525 and activity recognition. In motion prediction we train two 526 models [56] and [12] on the ExPI dataset [12] with/without 527 our data augmentation, following their settings. The testing 528 protocols and evaluation metrics follow [12] . The resultsare shown in Tab. classifiers, which shows that our generated data has more similar features to the ground-truth. Given close interaction data is new [12] and its limited variety and amounts, our method provide an efficient way of augmenting such data for activity recognition.Our model combines existing network components in a novel way for interaction augmentation, so a natural question is if there are other better alternative architectures. We test several alternative network architectures inspired by existing research. The selection criteria is they need to be data efficient for learning, so we exclude some data-demanding architectures such as Transformers or Diffusion models.The details and results are shown in the SM, but overall our model outperforms the alternative architectures. Given the small number of motions needed by our method, this is still a fast pipeline to acquire a large number of interactions with varying body sizes. rics) in Fig.5in comparison with ground-truth.474 
: 
 tons [13] . With a single skeleton, the problem can be formulated as optimizations with respect to keeping key geometric and dynamic constraints [3, 43] . However, this process quickly becomes intractable with the increase of constraints such as foot contact and hand-environment contact, let alone retargeting two people with close and continuous interactions like wrestling and dancing, where intercharacter geometric/topological constraints need to be retained [14, 30] . Consequently, multiple runs of complex optimization with careful hand-tuning of objective function weights are needed [15, 16] for a single motion, which is prohibitively slow and therefore can only be used to generate small amounts of data.Meanwhile, data-driven approaches for single body retargeting [4] , despite being successful, cannot be directly extended to two-character interaction. Methodologically, these methods do not model inter-character geometric constraints, which is key to the semantics of interactions [16] .From the data point of view, these approaches, especially those using deep learning [2, 48] , require a large amount of data, which is largely absent for two-character interaction. Existing two-character interaction datasets are for action recognition [7, 37] and low-quality, or only consist of a small amount of data with limited variations in body sizes [12] , hardly covering the distribution of possible body variations. Considering the high cost of obtaining interaction data, a method that can learn effectively from limited data and generate interactions with diversified body variations is highly desirable.We propose a novel lightweight framework for twocharacter skeletal interaction augmentation, easing the need to capture a large amount of data. Our key insight is the joint relations evolving in time (e.g. relative positions, velocities, etc.) can fully describe an interaction, e.g. hugging always involves wrapping one's arms around the other's body. These relations change when the body size changes, but the distribution of them should stay similar in the sense that one's arms should still wrap around the other, such that the hand-to-body distance is always smaller than e.g. the foot-to-body distance. Meanwhile, this distribution should be very different from other types of interactions 123 ture the joint relations [27] . Generative flows combine 124 the style and content in the latent space [57] . Transform-125 ers co-embed human motion and body parameters into a la-126 tent representation [35] . Diffusion models provide a larger 127 capacity and are less prone to mode collapse in genera-128 tion [47, 64] . But all the above research is on a single 129 body. While there is some research in modeling humanenvironment interactions [20, 61] , two-body interactions are more complex. Very recent research shows successful synthesis of two interacting characters, but their focus is either on single character control [24, 41], or fix one while generating the other [10, 28] . None of them models interactions, especially under varying body sizes and proportions. To our best knowledge, there is no deep-learning method for complex two-character interactions. 
 
 Appendix:  


Decision: The paper received two weak accepts. The paper contributes augmentations of the ExPI dataset, the algorithm design is reasonably novel to clear the bar of a workshop paper. The results presented in the supplementary video also present a convincing picture. Therefore, the AC recommends to accept this paper. There were concerns regarding model overfitting as test results look similar to the ground-truth. The authors are encouraged to address this concern in their revised camera ready.

Reviews: review: title: title: The authors present a deep learning based approach for two person skeletal interaction motion augmentation. The method is technically sound and has potential to benefit motion generation with multiple skeleton sizes without capturing such data..
main_review: main_review: The paper proposes a learning based method that can be used to generate interaction between different skeleton sizes without the need to capture data with such skeletons. The method is able to learn the joint relation distribution between two characters based on a given skeleton and retarget the relation to another skeleton of different size. 
Strengths:
1.  The idea is novel and interesting to explore since it benefits from not having to capture more motions with different sizes of skeletons. The method is technically sound.
2. The paper is clearly written and easy to understand.
3. The paper provides adequate baselines for the tasks of motion prediction and activity recognition. However, [12] could also be used as another baseline for the motion prediction since it involves two person interaction.

Weakness/Questions:
1. The difference between the ground-truth and the generated motions are not identifiable.
2. Even though the two person interaction is being modeled with STGCN, how does the method make sure that the contact points are intact? E.g. wrapping one's arms around the other's body, how is it ensured that the contact points between the arms are correct when the skeleton sizes are changed.   
3. The related work section can benefit from adding some recent deep learning based two-person interaction generation methods such as  InterGen[*1], ReMoS[*2], ReGenNet[*3]. The first paper also propose two-person datasets, which might be worth looking into.

[*1] Liang, Han, et al. "Intergen: Diffusion-based multi-human motion generation under complex interactions." International Journal of Computer Vision (2024): 1-21.
[*2] Ghosh, Anindita, et al. "ReMoS: Reactive 3D Motion Synthesis for Two-Person Interactions." arXiv preprint arXiv:2311.17057 (2023).
[*3] Xu, Liang, et al. "ReGenNet: Towards Human Action-Reaction Synthesis." arXiv preprint arXiv:2403.11882 (2024).
title: title: Review
main_review: main_review: # Summary
This paper focuses on close and continuous human interaction with rich contacts. A method is proposed to learn two-body skeletal interaction to generate contact-rich interactions with varying body sizes and proportions: predicting the joint relation distributions based on a given skeleton (specifically, bone length vector). Furthermore, the method allows for learning from a relatively small amount of data and generalize to drastically different skeleton sizes.  To train the model, the authors also introduce a new dataset augmented from single interaction examples with different body sizes and proportions using the existing method InteractionMesh.

# Exposition
“To our best knowledge, there is no deep-learning method for complex two-character interactions. ”-> This sentence is too big; it is better to revise it.

The resolution of Fig1 should be enhanced.

L256 - 257 “shown in Fig. 1 Compared xxx” -> shown in Fig. 1. Compared 

“SM” is used for the short of Supplement Material… Consider using “The Supplementary” or “Appendix”...

# References

Generally OK.

Some recent related references are missing:
- For single-person human motion generation:

Generating diverse and natural 3d human motions from text. CVPR22.

T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations, CVPR 2023.

…

- For two-person interactions:

Strategies for Physically Simulated Characters Performing Two-player Competitive Sports, ACM Transactions on Graphics (SIGGRAPH 2021).
- For skeleton-agnostic retargeting:

SAME: Skeleton-Agnostic Motion Embedding, SIGGRAPH Asia 2023.

# Technical Correctness 

Yes. The whole pipeline makes sense to me.
The encoder part uses Spatio-Temporal Graph Convolution Networks to capture spatial and temporal information about the skeleton, while the decoder uses a gated Recurrent Unit to reconstruct the body motion. The dataset is constructed from a single-person motion dataset by employing the off-the-shelf model InteractionMesh.  The innovative aspect of the proposed model is somewhat constrained. but it can be taken as a step for generating two-person motions considering varying body sizes (skeletal structure). The evaluation validates the method to some extent. 
I highly suggest the authors release their codes for reproduction purposes.


# Overall Recommendation

The whole pipeline is reasonable, though the novelty is still a little limited. The tested dataset scale is not that large.
But generally, this is a well-made paper for publishing at the workshop.