InterControl: Zero-shot Human Interaction Generation by Controlling Every Joint
Authors: Zhenzhi Wang
Jingbo Wang
Yixuan Li
Dahua Lin
Bo Dai
Venue: NeurIPS 2024 poster (2024)
Keywords: human motion generation
human interaction generation
diffusion model
controllable generation
Primary area: diffusion_based_models
TL;DR: the first method to generate human interactions of arbitrary number of humans in a zero-shot manner with only single-person training data

Abstract: Text-conditioned motion synthesis has made remarkable progress with the emergence of diffusion models. However, the majority of these motion diffusion models are primarily designed for a single character and overlook multi-human interactions. In our approach, we strive to explore this problem by synthesizing human motion with interactions for a group of characters of any size in a zero-shot manner. The key aspect of our approach is the adaptation of human-wise interactions as pairs of human joints that can be either in contact or separated by a desired distance. In contrast to existing methods that necessitate training motion generation models on multi-human motion datasets with a fixed number of characters, our approach inherently possesses the flexibility to model human interactions involving an arbitrary number of individuals, thereby transcending the limitations imposed by the training data. We introduce a novel controllable motion generation method, InterControl, to encourage the synthesized motions maintaining the desired distance between joint pairs. It consists of a motion controller and an inverse kinematics guidance module that realistically and accurately aligns the joints of synthesized characters to the desired location.  Furthermore, we demonstrate that the distance between joint pairs for human-wise interactions can be generated using an off-the-shelf Large Language Model (LLM). Experimental results highlight the capability of our framework to generate interactions with multiple human characters and its potential to work with off-the-shelf physics-based character simulators. Code is available at https://github.com/zhenzhiwang/intercontrol.

InterControl: Zero-shot Human Interaction Generation by Controlling Every Joint
Zhenzhi Wang', Jingbo Wang', Yixuan Li', Dahua Lin Bo Da
'he Chinese University of Hong Kong, *Shanghai Artificial Intelligence Laboratory, Thhe University of Hong Kong
(wz1 12, ly122, dhlin) @ie.e.. cuhk. edu. hk, wangjingbo@pji lab. org. cn bdai@hku.ht .k
Abstract
Text-conditioned motion synthesis has made remarkable progress with the emer- gence diffusion models. However, the majority these motion diffusion models are primarily designed for a single character and overlook multi-human interactions. our approach, we strive to explore this problem by synthesizing human motion with interactions for group of characters of any size in a zero-shot manner. The key aspect our approach is the adaptation of human-wise interactions as pairs human joints that can be either in contact separated by desired distance. In contrast to existing methods that necessitate training motion generation models on multi- sulti--hmmnn motion datasets with fixed number of characters, our approach inher- ently possesses the flexibility to model human interactions involving an arbitrary number of individuals, thereby transcending the limitations imposed by the training data. We introduce a novel controllable motion generation method, InterControl, encourage the synthesized motions maintaining the desired distance between joint pairs. It consists of motion controller and an inverse kinematics guidance module that realistically and accurately aligns the joints of synthesized characters to the desired location. Furthermore, we demonstrate that the distance between joint pairs for human-wise interactions can be generated using an off-the-shelf Large Language Model (LLM). Experimental results highlight the capability of our framework to generate interactions with multiple human characters and its potential to work with off- the-shelf physics-basee character simulators. Code is available https: //gittttbb con/ zhenzhi www.g/////////mm... intercontrol.
Introduction
Generating realistic and diverse human motions is vital task in computer vision, as it has diverse applications VR/AR, games, and films. In recent years, great progress has been achieved in human motion generation by introducing VAE [31]. Diffusson Model [23, 53] and large language models [5]. These methods commonly investigated single-person motion generation given texts or action classes [15, 14, 46, 71, 55, 6, 13, 45], part of motion [10, 19, 55], or other related modalities [35, 34, 56, 3, 18], yet overlooked multi perton interactions. By naively putting their generated single- person motions in shared global space, such motions could easily penetrate each other. They cannot even perform simple interactions like handshaking due to lack of the ability to control two people's hands to reach the same location at the same time. Many multi-person datasets [1, 16, 42, 59] lacks text annotations and focus on motion completion given prefix motions. Recently, InterGien [36] collected a two-person interaction generation dataset, and let model to learn two-person motions from data. is limited by the fixed number of characters and cannot generalize to arbitrary numbers. Previous methods commonly ignore good design for general interaction modeling.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a) Daily life Three-legged race
both hands
Hold hands in dancing
Hold hands in groups
(b) Fighting
Kick foat
Kick head foot
Hold hands
head fighting
(c) Physics animation on our interaction
Figure I: InterControl able to generate interactions a group of people given joint-joint contact separation pairs spatial condition, and it is only trained on single-person data. Our generated interactions are realistic and similar to real interactions in internet images in (a) daily life and (b) fighting. (c) shows our generated group motions (red dots) could serve reference motions for physics animation.
This paper investigates special yet widely used form of human interactions: interactions that could be quantitatively described by spatial relations of human joints, such as distances or orientations, as shown in Fig. (a) and (b). Such interactions from spatial relations. Thus, they do not require additional interaction data. It only needs pretrained models from single-person data and could be generalized an arbitrary number of humans. We define human interactions as steps of joint-joint contact pairs and devise a single-person motion generation model take such contact pairs control signals. Besides, orientations could also be used in control, such as making two people face each other. In this way, interaction generation is transformed controllable motion generation. Inspired by [64], we adapt descriptions of interactions as joint contact pairs by leveraging Large Language Models (LLMs). Thus, human interactions are annotation- free, and interactions could also involve multiple human joints.
conceptually simple, their semantics are almost
As interactions are adapted to our defined joint contact pairs, the key challenge to generate interactions the precise spatial control satisfy the constraint of spatial controls. This difficulty lies two parts: (I) the discrepancy between control signals in global space and relative motion representation in mainstream pretrained models [14, 55]: semanties of motions are independent to global locations, previous works [14, 55] commonly utilize the relative motions, where global locations could only be inferred by aggregating velocities. It poses challenges control local human poses with global conditions. Previous attempts [55, 51] exploit the inpainting ability of a pretrained model, yet they are unable control joints. GMD [27] proposes two-stage model of separated root trajectory generation and local pose generation. Although it manages control root positions, controlling every joint any time is still infeasible. (2) the sparse control signals in the motion sequence: Control signals could be sparse in both temporal and joint dimension, model needs to adaptively adjust trajectories in uncontrolled frames to satisfy the intermittent constraints.
In this paper, we propose InterControl, a novel human interaction generation method that is able to precisely control the position of any joint at any time for any person, and is only trained on single- person motion data. By adding spatial controls to MDM [55], InterControl is unified framework of two types of spatial control modules: (1) Motion ControlNet inspired by ControlNet [70]: It is initialized from a pretrained MDM [55] and takes global spatial locations as input for joint control the global space. is able to generate coherent and high-fidelity motions yet joint positions in global space are not perfect. (2) Inverse Kinematics (IK) Guidance for joint locations: To further align generated motions and spatial conditions precisely, we use inverse kinematics (IK) [44] to
guide the denoising steps towards desired positions. It could be regarded classifier guidance [9].

---

yet it has no extra classifiers. We utilize L-BFGS [37] as the optimizer to directly align the global conditions in the local space. With two proposed modules, InterControl is able control multiple joints of any person at any time. Furthermore, InterControl is able to jointly optimize multiple types of spatial controls, such as orientation alignment, collision avoidance, and joint contacts, as long as the distance measures in IK guidance are differentiable. By exploiting its joint control ability, our model is able to generate multi- interactions with rich contacts, where no multi-person interaction datasets are needed. Our generated interactions could further serve the reference motion to generate physical animation with meaningful human-wise reactions simulators. As shown in Fig. (c), one character could actually hit down the other with his fists by taking generated fighting motions as input. Extensive experiments in HumanML3D [14] and KIT-ML [47] datasets quantitatively validates our joint control ability, and the user study on generated interactions shows clear preference over previous methods.
To summarize, our contributions are twofold: (1) We are the first to generate multi- pellii-person interactions
with single-person motion generation model zero-shot manner. (2) We are the first to perform precise spatial control of every joint in every person at any time for interaction generation.
2
Related Work
2.1 Human Motion Generation
Synthesizing human motions is a long-standing topic. Previous efforts integrate extensive multimodal data as condition to facilitate conditional human motion generation, including text [15, 14, 46, 71, 55, 6, 30], action label [13, 45], part of motion [10, 19, 55J, music [35, 34, 56], speech [3, 18] and trajectory [49, 27, 28]. As texts are free-form information that convey rich semanties, recent progress in motion generation are mainly based on text conditions. For example, FLAME [30] introduces transformer [58] to process variable-length motion data and language description. MDM [55] intro- duces the diffusion model and uses classifier-free guidance for text-conditioned motion generation. MLD [6] further incorporates VAE [31] to encode motions into vectors and makes the diffusion process in the latent space. Physdiff [68] integrates physical simulators as constraints in the diffusion process to make the generated motion physically plausible and reduce artifacts. PriorMDM [51] treats pretrained MDM [55] as generative prior and controls MDM by motion inpainting. Our InterControl also use pretrained MDM. yet we further train Motion ControlNet instead of using inpainting. A concurrent work OmniControl [65] also incorporate classifier guidance [9] and controlnet [70] modules to control all joints in MDM, yet it focuses on single-person motion generation and does not investigate human interaction generation.
2.2 Human-related Interaction Generation.
As human motions could be affected or interacted by surrounding humans [72, 29, 57], objects [66,
54, 12, 33, 26] and scenes [62-64, 73, 20, 61], generating interactions is also an important topic. Previous methods are mainly about semman-scene/obeer interaction. For example, Interdiff [66] uses the contact point of human joints and objects as the root to generate object motions. UniHSI [64] exploits LLM to generate contact steps between human joints and scene parts as an action plan and control the agent perform the plan via reinforcement learning. As previous human-human interactions datasets [42, 59] only contains very few multi-person sequences, previous numan-human interaction methods [60, 67] are mainly limited unsupervised motion completion without texts. Recently, interaction generation, yet it only consider the two-person situation and is not able model more people's 's interaction. To the best of our knowledge, we are the first to enable a single-person text-conditioned motion generation model perform interactions between a group of people by controlling diverse joints of each person.
InterHuman dataset [36] is proposed for text-conditioned
2.3 Controllable Diffusson Models
Diffusion-basse generative models have achieved great progress in generating various modalities,
such as image [50, 22, 9, 53), video [11, 17, 24] and audio [32]. Conditions and controlling ability in diffusion models are also well studied: (1) inpainting-based methods [8, 7] predict part of the data with the observed parts as condition and rely on diffusion model to generate consistent output, which
is used in PriorMDM [51]. (2) Classifier-guidann [9] trains separate classifier and exploits the
3

---

gradient of classifier to guide the diffusion process. Our InterControl inherits the spirit of classifier- guidance, yet our guidance is provided by Inverse Kinematics (IK) and no classifier is needed. (3) Classifier-free guidance [22] trains conditional and an unconditional diffusion model simultaneously and trade-off quality and diversity by setting weights. (4) ControlNet [70] introduces trainable copy of pretrained diffusion model to process the condition and freezes the original model to avoid degeneration of generation ability. It enables diverse types of dense control signals for various purpose with minimal finetuning effort. Our InterControl also incorporate the idea of ControlNet [70] to finetune the pretrained MDM [55] to process spatial control signals and improve the quality of generated motions after joint control.
3 Inter
InterControl aims generate interactions with only single-person motion data by precisely controlling every joint of every person at any time, conditioned on text prompts and joint relations. We first formulate interaction generation in Sec. 3.1, and then introduce control modules for single-person motion diffusion model in Sec. 3.3 and Sec. 3.4. Finally we show details to generate interactions from our model in Sec. 3.5.
3.1 Formulation of Interaction Generation
Inspired by human-scene interaction [64], we define human interactions as joint contact pairs C C= (S1, S2....), where S; is the ht contact step. Taking two-person interaction as an example, each step S has several contact pairs S = { { S. (i, (1, (1,1), (22.12, Lg 1g. 02, c2,d2] ...... where jk is the joint of person I. x is the joint of person 2, t and E means the start and end frame of the interaction, O₁ means contact type from [contact, avoid) to pull or push the joint pairs, dy is the desired distance in the interaction. By converting the contact pairs S to the mask m and distance d, and taking others' joint positions as condition, we could guide the multi-person motion generation process to interact between joints in the form of spatial distance. In this way, interaction generation is transformed to be controllable single-person motion generation taking text prompt B and spatial whose joints
control signal E R xx3 as input. Its goal is to predict motion sequence x E R N x in the global space is aligned with spatial control c, where N is number of frames, J is number of joints (e.g., 24 in SMPL (38]), and D is the dimension of relative joint representations (e.g., 263 in ML.3 [14]). Incorporating spatial control in motion generation presents challenges due to the discrepancy between relative motion representation æ and global
3.2 Human Motion Diffusion Model (MDM)
Relative Motion Representation. HumanML3D [14] dataset proposes widely-used [55, 68, 51, 6] relative motion representation, and is proved to be easier to learn realistic motions, as the semantics of human motion is independent of global positions. It consists of root joint velocity, other joints' positions, velocities and rotations in the root space, and foot contact labels. To convert it to the global space, root velocities are aggregated, then other joints will be computed based on root. Please refer to Sec. 5 of HumanMI.3D [14] for details. Due to such discrepancy, previous inpainting-base methods [55, 51] not able control MDM in global space. GMD [27] decouples motion generation to two separated generation process of root trajectory and pose relative to root, yet it can only control root joint. Directly adopting global joint positions to generate motions yields unnatural human poses, such as unrealistic limb lengths.
Diffusion Process in MDM. Motivated by the success of image diffusion models [22, 50, 70, 53], Motion Diffusion Model (MDM) [55] is proposed to synthesize sequence-level human motions conditioned on texts p via classifier-free guidance [22]. The diffusion process is modeled
noising Markov process q (xx, 1z-1) = N' (V (1 - o..) I). where O4 E (0, 1) are small constant paree--arameees, thus ~ (o, I) if ar is small enough. Here 2; E R is the entire motion sequence at denoising time-step t, and there are T stee-steps in total. Thus, 2o is the clean motion sequence, and xr is random noise to be sampled. The denoising Markov process is defined as po (#t-1 æt; p) N (1t (2t, L, p), (1 ax)) I), where Mh (we, L, p) is the estimated posterior mean for the 1. - 1 step from neural network based on the input n, and 0 is its parameters. Following MDM, we predict the clean motion (2, I, p; 0) instead of the noise via transformer [58], and the posterior mean Me ( t, p) is

---

Interaction description: Two people are fighting with each other.
Motion ControlNet
Spatial
IK Guidance
C
Control
LLM Planner
head Gine hand
Control
Trainable Copy of MDM
L-BFGS(L(M,,c)
Optimize k times
How
people fighting with each
Motion Diffusion Model
(1,, E.]
Diffusion Stops
other?
Transfo rmnrr
Transformer
Transi pelllorma mer Laye
Text for single person: A person righting with his fists and foots like martial arts. Contact Joint pairs: using hands to
Gaussian Noise
Layer
Layes
P
other's head many timesteps.
prompt
dean motion at Freme 80
Frame
Frame
Frame 20 contact
Freme 30
Frame 40
Freme 60
COMA
Figure 2: Overview. Our model could precisely control human joints in the global space via the Motion ControlNet and IK guidance module. By leveraging LLM to adapt interaction descriptions to joint contact pairs, it could generate multi-pert person interactions via Lingle-person motion generation model zero-shot manner.
No (1 t, = V 11 xo 2o(2,,t,,:00 + Vot (1 ōt-1)
21. ā
āi
where Bt = a₂ and 0₂
II ag.. MDM's parameter is trained by minimizing the lz-loss |00 (2 t, 0) zē 2 where wi is the ground-truth motion and (o t, t; is MDM's prediction of æo at denoising timestep
3.3 Motion ControlNet for MDM
As MDM is initially conditioned on texts p. requires fine-tuning to accommodate spatial conditions
C. This is challenging due to the potential sparsity of e across temporal and joint dimensions: (1) Control may be required for only few joints, necessitating adaptive adjustment of the remaining joints to preserve realistic motion. (2) Control may be desired for only select few frames, thus the model must interpolate natural human motions for the rest of the sequence.
Inspired by ControlNet [70], we introduce Motion ControlNet to generate realistic and high-fidelity motions guided by condition c. is a trainable copy of MDM, while MDM is frozen our training process. Each transformer encoder layer in ControlNet is connected to its MDM counterpart via vero-initialized linear layer. This allows InterControl to commence training from a state equivalent to pretrained MDM, acquiring residual feature for c in each layer through pack-propagation. To process c, the uncontrolled joints, frames, and XYZ-dim are masked as We find that the vanilla E Nx3J is effective enough to control the pelvis (root) joint, yet it is still sub- sub-optimal for other joints. Thus, we design relative condition indicating the distance from the current positions of each joint to global space R(x) R N.NxJa x33 the relative condition is c' = c R(x). To provide additional clues, we also use c" = c Rrar)root to represent the distance from the current root to the desired position. We also use the normal of triangles (pelvis, left/rght shoulder) n² and (pelvis, left/right hip) n to represent the current orientation of human. The final condition passed ControlNet is I|n²",, where || is concatenation. Please refer to Appendix A.2 for more details.
Suppose R(-) is forward kinematics (FK) to convert relative motion E R
If final =
Network Training. Motion ControlNet is the only part that needs finetuning in our framework, while IK guidance an optimization method the test time and the LLM in our framework is an off- -he-thh--helf GPT-4 [43]. We adopt the standard ControlNet [70] training strategy, and the only difference the data format: we first convert the relative motion to be global locations by FK, and then use random masks that keeps part of global joints to be non-zero spatial control signals. The training objective identical MDM. The spatial conditions are randomly sampled in the temporal or joint dimension. The training data is single-person data only, e.g., HumanML3D [14].
5

---

3.4 Inverse Kinematies (IK) Guidance
While Motion ControlNet can adapt joint positions according to sparse conditions, the alignment between predicted poses and global spatial conditions often lacks precision. As Inverse Kinematics (IK) a classic method for optimizing joint rotations to achieve specific global positions, we employ it to guide the diffusion process towards spatial conditions at test time in a classifier guidance [9] manner, named IK guidance.
IK Guidance on general form of losses. Inspired by classifier guidance [9] and loss-guided dif- fusion [52], we employ losses in the global space to steer the denoising process. IK guidance accommodates various forms of distance measurements, enabling both minimization and maximiza- tion for flexible control over joint interactions, such as attraction or repulsion. Given the global position c E R RNxJx3, the distance between a joint and condition is dn, = I|eny R(pg) p4 )nz llg. where H4 is short for (z p) mentioned in Sec. 3.2, and R(-) forward kinematics (FK). To allow the interaction of joints with some given distances d' E IR NxJx3 loss of one joint is Lnj = ReI.U (dnj d'n)) to make the joint and condition be contacted within distance d'ys and it Inj = ReLU (d'a dnÿ) to make the joint and condition be far away, where ReLU is a function to keep values > 0 and set values I 0 to O. Finally, with binary mask m {0, Nxx the total
loss for all joints and frames is
T(4, c) En Mos Lnj
mnj
As f2-loss and FK are highly differentiable, we optimize c( in Equ. 2 w.r.t je, using the second- order optimizer L-BFGS [37], which is commonly used in Inverse Kinematics, rather than first-order gradient methods. Classifier guidance [9] utilizes pre-trained image classifier direct the diffusion towards a target image class by the gradient Vx, log fo (y æz),, where fo is the classifier, y is image class. Unlike this method, we do not rely on a large neural network classifier. L-BFGS has been demonstrated to better align global positions and offer quicker convergence than first-order methods. We update the posterior mean 14 using L-BFGS for iterations at each denoising step, where k is a This optimization facilitates both pull and push types of IK guidance, corresponding to two contact types in our interaction model. To maintain consistency in data distribution between training and inference, we also apply IK guidance when training ControlNet. Additionally, employing IK guidance on To eliminates the need for training Motion ControlNet, thus enhancing training efficiency. In practice, using L-BFGS on both To and μ4, can yield satisfactory joint and spatial condition alignment. Detailed algorithm for interaction generation is presented in Appendix A.
As the root position at frame n is derived from cumulative root velocities up to frame in FK, a single condition at frame n can influence all preceding root positions. This effect also extends to non-root, joints, as their global positions are calculated from the root. Consequently, IK guidance can adaptively modify velocities from the start to frame n to meet the condition frame n. Moreover, IK guidance can control any combination of human joints, frames or XYZ-dims, such as controlling the left hand and right foot at a specific frame n.
3.5 Interaction Generation
Inverse Kinematics (IK) guidance can optimize various distance measures to facilitate interactions such avoiding obstacles, preventing collisions, facilitating face-to-face engagements, or enabling joint contacts between individuals. This method allows for intricate interactions among any human joints for an indefinite number of people, despite being trained exclusively on single-person data. As delineated in Section 3.1, we characterize interactions as pairs of contacting joints. A notable feature of our IK guidance in generating interactions is that both terms of the IK guidance loss function are predicted, allowing for simultancous optimization within single process. Specifically, the single- person loss Li (1t,, c) transforms into Limutts ( 1²) for interactions, where a and b represent two individuals. The L-BFGS optimizer concurrently optimizes both participants by minimizing Lymutte (1&. pa)), with 2e and pa being the respective joints engaged in interaction. Beyond distance measures, our IK guidance can optimize orientation measures as well. For example, one can calculate a person's orientation through the spatial relationship of their joints, like the cross-product of vectors from the left shoulder to the right and from the pelvis to the head. By setting two individuals' unit orientation vectors to O, they can face each other or turn away. To ensure they face each other, we can further adjust the relation between one person's 's orientation vector and the vector from their head to

---

the other's. s. Such orientation relationships are vital for producing realistic interactions when we only exploit single-person motion generation ability and can be easily expanded to include larger groups. Another useful strategy in IK guidance is to prevent collision through joint separation pairs, ensuring that the torso joints of two people (such as pelvis, hips, and spines) maintain certain distance, thereby reducing the likelihood of collisions when other joints are in contact. Besides, we can also regulate the motion region by confining the root joints within the XZ-plane using IK guidance. For the PyTorch-like code illustrating loss functions that enforce joint contacts, separations, or orientation alignment, please refer to Appendix A. for details.
In our framework, interaction generation is realized by using joint-joint contact pairs as control signals. These pairs can be manually crafted by users to create desired interactions, akin utilizing ControlNet [70] in image generation. However, manually constructing joint contact pairs can be tedious, so we employ an automatic off-the-shelf GPT-4 [43] as a planner. GPT-4 infers text prompts that describe the actions of multiple people, p prowdei and converts them into single-pers peggl-eepsoonnn prompts, p, and contact plans, C, through prompt engineering. The inputs for the LLM Planner include the multi-person sentences p pmuuits, background scenario details B, human joint data J, and predefined instructions, rules, and examples. Specifically, encompasses the number of individuals, total motion sequence frames, and video playback speed; J contains names of all joints (for example, the 22 joint names in HumanML3D [14]); and the rules outline the joint contact pair format and guide the LLM to generate feasible contacts and timesteps. Our method leverages the pre-traine pre-trained capabilities of GPT-4 to comprehend human joint relationships from interaction descriptions via prompt engineering without any fine-tuning. Thus, the inference process of our model is not related to LLMs, making our comparison with other methods be fair. Please refer to Appendix A.3 for details of prompts and contact plans.
Experiments
Datasets. We conduct experiments HumanML3D [14] and KIT-ML [47] following MDM [55]. HumanML3D contains 14,646 high-quality human motion sequences from AMASS [41] and Hu- manAct12 [13], while KIT-ML contains 3 motion sequences with more noises.
Evaluation Protocol. We adopt metrics suggested by Guo et. al. [14] to evaluate the quality of alignment between text and motion, which are Frechet Inception Distance (FID), R-Precision, and Diversity. We also report metrics related to spatial controls following GMD [27] on HumanMLL3D dataset, which are Foot skating ratio, Trajectory error, Location error and Average error. Please refer to Appendix B.5 or papers [14, 27] for more details.
Due to the page limit, we put the implementation details and text-to-motion generation in the Appendix B.1 and B.2.
4.1 Single-Person Controllable Motion Generation
In Tab. 1, we compare InterControl with other spatially controllable methods [51, 27, 65]. We also include results of MDM [55] to show the controlling metrics [27] without spatial MDDt's trajectory can significantly deviate from the intended path in the absence of control signals, with an average error often exceeding 1m. In contrast, inpainting-basee information, results in considerable divergence, seen with PriorMDM [S]]. GMD [27] decouples this problem and generates root trajectories in the global space, so it achieves better performance in spatial control metrics. However, its limitation only the root joint constrains its spatial control and interaction capabilities. Our InterControl could achieve very small errors in spatial control metrics for all-joint control thanks to the power of Inverse Kinematics and L-BFGS optimizer. Meanwhile, Motion ControlNet could ensure the motion data is still in the same distribution with the training set by adapting the posterior mean updated by IK guidance in its training stage, leading even better FID than previous methods. It is worth noting that we only use single model to learn the control strategy for all joints, while previous method [51] needs to train separate models and blend them for multiple joints. Our method achieves similar performance with controlling one joint when extending to control multiple joints (last two rows in Tab. 1). Compared to the recent concurrent work [65], we achieve significantly better FID and Traj./Loc. errors than it in both root joint control or random joint control. It [65] also shows a notable gap between two form of joint controls (0.310 vs. 0.218), while our method is more robust to joint variants (0.178 vs. 0.159) thanks to special designs
control, unaware of global spatial
7

---

Table 1: Spatial control results on HumanML3D [14]. means closer to real data is better. Random One/Iwo/Three reports the average performance over 1/2/3 randomly selected joints in evaluation. means our evaluation on their model.
Method
Joint
FID 4
R R-precision (Top-3)
Diversity +
Fuul skating
Traj. etr. 4
Loc. err. 4
Avg. err.
ratio J.
cm)
(50
(m)
Real data
0.002
0.797
9.503
0.0000
0.0000
0.0000
0.0000
MDM [55]
No Control
0.544
0.611
9.446
0.0943
0. 0.8909
0.6015
1.1843
PriorMDM [51]*
0.498
0.586
9.167
0.0924
0.3726
0.2210
0.4552
GMD [27]*
Root
0.276
0.655
9.245
0.1108
0.0987
0.0356
0.1457
OmniCoutrol [65]
0.218
0.687
9,422
0.0547
0.0387
0.0096
0,0338
Ours
0.159
0.671
9.482
0.0729
0,0132
0,0004
0.0496
OnniControl [65]
Random one
0.310
0.693
9,502
0.0608
0.0617
0.0107
0.0404
Ours
0.178
0.669
9.498
0.0968
0.0403
0.0031
0.0741
Ours
Random two
0.184
0.670
9.410
0.0948
0.0475
0.0030
0.0911
Ours
Random three 0.199
0.673
9.352
0.0930
0.0487
0.0026
0.0969
Table 2: Evaluation on (left) spatial errors and (right) user preference in interactions.
Spatial Errors
Traj. err. (20 cm)
Loc. err. (20 cm)
Avg. em. (m) 3
User-study
Preference
PriorMDM [51]
0.6931
0,3487
0.6723
PriorMDM [51]
18.8%
Ours
0,0082
0.0005
0.0084
Ours
81.2%
of more inputs in Motion ControlNet. Its R- R-precision and foot-skating ratio are slightly better than ours, we believe the reason is that their 1-st order optimization tolerates more errors when the joint aligmment is hard. It is also supported by their worse Traj./Loe. yet better Avg. err., i.e., their method shows more outliers with large errors. However, their design need much more times of optimization compared to ours (e.g., 100 vs. 5) and leads to longer inference time than ours (120s vs. 80s).
4.2 Zero-Shot Multi- Interaction Generation
To validate our model's interaction generation ability, we analyze the spatial control results in interac- tion scenarios and perform an user study qualitatively compare our model with PriorMDM [51]. We also introduce an potential application of our interaction generation method for physics animation.
Spatial Control. In Tab. 2 (left), compare spatial-related metrics with PriorMDM in zero- shot human interaction generation. Specifically, we collect 100 descriptions two-person actions from InterHuman Dataset [36] and let an off-the-shelf GPT-4 [43] to adapt them single-person motion descriptions and joint-joint contact pairs via prompt engineering (see Tab. 7 in Appendix). Then, we utilize an InterControl model pretrained on the HumanML3D dataset generate human interactions conditioned on text prompts and joint contact pairs. The spatial-related metrics are reported over controlled joints and frames. InterControl achieves good performance of spatial errors in interaction scenarios, indicating its robustness in precise spatial control for multiple humans. In contrast, PriorMDM [51] could only take interaction descriptions as input and unable to perform spatial control, leading to much larger spatial errors.
User Study. We conduct user study to qualitatively compare method with PriorMDM on the text-conditioned two-person interaction generation. 134 unique users were participating in the user study, where each user will answer 19 single choice questions to compare our results with PriorMDM [51]. Results in Tab. 2 (right) shows that our generated interactions are clearly preferred over PriorMDM by a percent of 81.2% We also shows an example sequence of qualitative comparison with PriorMDM [51] in the user study in Fig. 3. PriorMDM [51] shows severe torso collision between two human skeletons and the generated two-people motion is not aligned with the interaction description, while our model has no torso collision thanks to the collision avoidance loss in our IK guidance. Besides, our method also produces reasonable kicking actions between two people according to the semantics of interaction description. Please refer to Appendix B.4 for details. Qualitative results: Although our model is only trained on pengle-person data, is still possible to generate interactions between an arbitrary number of people via our designed format of interaction. In Fig. 4, we show two representative results of zero-shot interaction generation.
(1) Two-person dancing: In addition to the single person dancing from the pretrained ability of single-person model, we further let them hold hands from time to time and prevent them from collision between their torsos. To further make their dance natural, we also employ a loss to promote their orientations to
8

---

Description: Ore parson buga mucult towards the other, other jumpate wipordiwth
hie XDM
Durs
Jubb
Figure 3: Comparison with PriorMDM [51] in user-study of zero-shot human interaction generation.
Interaction
Description
Two and dteiing with wach
other and they
Twon people fihting with the their hands loots.
Figure 4: Qualitative results of zero-shot human interaction generation.
be face-to-face. (2) Three- person fighting: In addition to single person performing punching and kicking, we further let them punch or kick others' head and torso, and also prevent their torsos from collision. Compared to existing interaction generation method [36]. method is able to generate interaction between any number of people, while InterGen [36] is only able to generate two-person interaction. Besides, our method is the first method leverage single-person motion generation model to generate human interactions in a zero-shot manner.
Application: Our method is able to seamlessly integrate with off-the- character simulation approaches, allowing us synthesize physically plausible human reactions. As shown in Fig. (c), method synthesizes the motions, where the orange character fighting with other two characters, as the reference the SoTA physics-aware motion imitator [40]. The interactions our motions are designed to hit heads of other characters with fists. Leveraging the precise spatial control provided by our approach, the animated characters in the simulator can accurately respond to these impacts, resulting in realistic reactions such as being knocked down. This capability to generate spatially coherent multi-human interactions enables our method to improve the plausibility and responsiveness of synthesized reactions within physics- based character animations.
4.3 Ablation Studies
To further investigate the effectiveness of InterControl, we ablate our method in Tab, 3 and reveal some key information in controlling the motion generation model in the global space. Then also analyze the computational costs of our method to ensure our control is efficient. We will refer to the variants of InterControl by row numbers in Tab. 3. All experiments are trained all joints and evaluated with randomly selected joints to report average performance.
Motion ControlNet. By dropping ControlNet, we find that IK guidance could still follow spatial controls with very low errors, yet motion quality (e-g.. FID) significantly damaged (row vs. row 2). Our ControlNet could adapt the posterior distribution updated by IK guidance, and produce high-quality motion data. We also find that our final provides key information in controlling all joints: For root control only, the FID of final and shows small difference. However, the FID of root control is always slightly better than all-joint control (~ 0.07) when we use c, indicating insufficient information in all-joint control. We alleviate this by introducing extra information in finat for Motion ControlNet and improve the FID of all-joint control from 0.227 (row 3) to 0.178 (row 1).
IK guidanee. By dropping IK guidance, Motion ControlNet can produce good semantic-level metrics (e.g., FID) compared with MDM by using extra spatial cues (row 4). However, this variant will lead to more spatial errors and cannot strietly follow spatial controls in global space. precise joint alignment is vital for interactions, IK guidance is important for InterControl. Another variant is

---

Table 3: Ablation studies the HumanMM3D [14] dataset.
Item
Method
FID 4
R-precision (Top-3)
Diversity
Foot skating
Traj, err.
Loc. err. 4
Avg. em..
ratio
(50 cm)
(50 cm)
(m)
(1)
Ours (random joint)
0.178
0.669
9,498
0.0968
0.0403
0.0031
0.0741
(2)
w/o NontrolNet
0.965
0.621
9.216
0.1624
0.0879
0.0059
0.1013
w/ original
0.227
0.656
9,544
0.1004
0.0697
0.0042
0.0785
(4)
w/o IK guidance
0.187
0.664
9.598
0.0704
0.8569
0.4553
0.6557
(5)
IK guidance
0.211
0.668
9,394
0.1164
0.0907
0.0088
0.0981
(6)
w/ 1-st order grad
0.198
0.668
9.472
0.0987
0.0879
0.0096
0.0877
(7)
sparsity 0.25
0.248
0.671
9,442 9.520
0.0801
0.0106
0.0007
0.0546
(8)
sparsity = 0.025
0.255
0.663
0.0705
0.0015
0.0001
0.0067
Table 4: Inference time analysis on a NVIDIA A100 GPU.
Sub-Modules
MDM
Control Module
GGuidance [10, 999]
Guidance [0.9]
Time (s)
39.1
57.3
76.5
80.1
updating IK guidance on ControlNet's prediction Zo (row 5), instead of the posterior mean f4,, Its advantage faster training speed because IK guidance is longer needed in training ControlNet (similar to classifier guidance [9]) yet it leads to slightly worse FID than using p4, We believe the reason that IK guidance still changes the data distribution in denoising steps even if it is updated on Zp. Finally, we also report the result of 1-st order gradient in classifier guidance [9] (row 6) instead of L-BFGS. We find it takes more computations to achieve similar performance with L-BFGS, which is analyzed below.
Inference time analysis. In practice, find that IK guidance in last few denoising steps (e.g.. t E [0, 9) is vital for precise joint control, while most denoising steps E [10, 999] are less important yet take most of computations. IK guidance on zo with only once L-BFGS in t E [10, 999] and 10 times in & E [0, 9] could leads FID 0.234 in controlling all joints, yet leads to minimal extra computations. We report its total inference time of 1000 denoising steps by adding sub-modules step-by-step in Tab, 4. GMD [27] needs 110s to run two-stage diffusion models, while we only needs 80s. Gradient-bart based optimization in the recent work [65] needs 120s achieve similar control quality. Leveraging GPU parallel computing capabilities. InterControl can efficiently generate motions for a batch of 32 people in 91 seconds, enabling efficient group motion generation.
Sparse control signals in temporal. As a key challenge of spatial control is the sparsity, we also report results with sparsely selected frames control (sparsity = 0.25 and 0.025) in Tab. 3 (row 7 and 8). Our model demonstrates consistent performance in both spatial error and semantic-level metrics when using sparse signals, e.g., FID 0. 0.255 and avg. err. 0.0467 with sparsity 0.025, while GMD [27] achieves FID 0.523 and avg. err. 0. 139 with the same sparsity.
5 Conclusion and Limitations
We presented InterControl, a multi-person interaction generation method that is only trained on single-person motion data. It could generate interactive human motions of an arbitrary number of people. We achieve this by enabling a text-conditioned motion generation model with the ability to control every joint of every person at any time. We propose two complementary modules, named Motion ControlNet and IK guidance, to improve both the spatial alignment between joints and desired positions, and the overall quality of whole motions. Extensive experiments are conducted on HumanML3D and KIT-ML benchmarks to validate the effectiveness and efficiency of our proposed modules. We enable InterControl the ability of text-conditioned interaction generation by leveraging the knowledge of LLMs. Qualitative results and user study validate that InterControl could generate high-quality interactions by precise spatial joint control.
Limitations. As InterControl is not trained on multi-person data, its definition of interaction is based on distances (being contacted or separated) or orientations. Its motion quality is from motion generation model trained on single-person motion data, and the plausibility of interactions is from the knowledge of L.MMs, i.e., to what extent the joint contact pairs are consistent to the semantics of interaction descriptions. Yet, InterControl could generate interactions of an arbitrary number of people, while all existing interaction generation methods cannot.
10

---

Acknowledgment. This project is funded in part by Shanghai Artificial Intelligence Laboratory, CUHK Interdisciplinary AI Research Institute, and the Centre for Perceptual and Interactive Intelli- gence (CPII) Ltd under the Innovation and Technology Commission (ITC)'s InnoHK, We would like to thank Tianfan Xue for his insightful discussion.
References
[1] Cmu graphics motion capture database.
[2] Chaitanya Ahuja and Louis-Philipee Morency. Language2pose: Natural language grounded pose forecast- ing. 3DV. IEEF, 2019.
[3] Tenglong Ao, Qingzhe Gao, Yuke Lou, Baoquan Chen, and Libin Liu. Rhythmic gesticulator: Rhythm- aware co-speech gesture synthesis with hicrarchical neural embeddings. ACM Trans. Graph. 2022.
[4] Uttaran Bhattacharya, Nicholas Rewkowski, Abhishek Banerjee, Pooja Guhan, Aniket Bera, and Dinesh Manocha. Text2gestures: transformer-based network for generating emotive body gestures for virtual agents, In VR. IEEE, 2021.
[5] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell. Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Fric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Nlya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurlPS, 2020.
[6] Xin Chen, Biao Jiang. Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu. Executing your commands via motion diffusion in latent space. In CVPR, 2023.
[7] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. ILVR: conditioning method for denoising diffusion probabilistic models. In ICCV, 2021.
[8] Hyungjin Chung. Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. In NeurlPS, 2022.
[9] Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models heat gans on image synthesis. In NewrlPS, 2021.
[10] Yinglin Duan, Tianyang Shi, Zhengxia Zou, Yenan Lin, Zhehui Qian, Bohan Zhang, and Yi Yuan. Single- shot motion completion with transformer. arXiv preprint arXiv:2103.0077 2021.
[11] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. In ICCV, 2023.
[12] Anindita Ghosh, Rishabh Dabral, Vladislav Golyanik, Christian Theobalt, and Philipp Slusallek. Imos: Intent-driven full-body motion synthesis for human-object interactions. Comput. Graph. Forum, 2023.
[13] Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. Action2motion: Conditioned generation of 3d human motions. In ACM MM, 2020.
[14] Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In CVPR, 2022.
[15] Chuan Guo. Xinxin Zuo, Sen Wang, and Li Cheng. Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts. In ECCV, 2022.
[16] Wen Guo, Xiaoyu Bie, Xavier Alameda-Pineda, and Francese Moreno-Noguer. Multi-person extreme motion prediction. In CVPR, 2022.
[17] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animated- iff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint amXiv:2307.04723 2023.
[18] Ikhsanul Habibic, Mohamed Elgharih, Kripasindhu Sarkar, Ahsan Abdullah, Simbarashe Nyatsanga, Michael Neff, and Christian Theobalt. A motion matching-basse based framework for controllable gesture synthesis from speech. In SIGGRAPH (Conference Paper Track), 2022.
[19] Félix G Harvey, Mike Yurick, Derek Nowrouzezahrai, and Christopher Pal. Robust motion in betbeewwwen..... ACM Transactions Graphics (TOG), 2020.

---

[20] Mohamed Hassan, Duygu Ceylan, Ruben Villegas, Jun Saito, Jimei Yang, Yi Zhou, and Michael J. Black. Stochastic scene-aware motion prediction. In ICCV, 2021.
[21] Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[22] Jonathan Ho and Tim Salimans. Classifier- free diffusion guidance. arXiv preprint arXiv:2207.12544 2022.
[23] Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
[24] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang. Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint 221iv:22000023000 2022.
[25] Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as a foreign language. arXiv preprint arXiv:2306. 14795, 2023.
[26] Nan Jiang, Tengyu Liu, Zhexuan Cao, Jieming Cui, Yixin Chen, He Wang, Yixin Zhu, and Siyuan Huang. Chairs: Towards full-body articulated buman-object interaction. arXiv preprint 2:X.::2212.10622 2022.
[27] Korrawe Karunratanakul, Konpat Preechakul, Supasorn Suwajanakorn, and Siyu Tang. Guided motion diffusion for controllable human motion synthesis. In CVPR, 2023.
[28] Manuel Kaufmann, Emre Aksan, Jie Song, Fabrizio Pece, Remo Ziegler, and Otmar Hilliges. Convolutional autoencoders for human motion infilling. In 3DV, 2020.
[29] Jongmin Kim, Yeongho Seol, and Taesoo Kwon. Interactive multi-character motion retargeting. Comput. Animat. Virtual Worlds, 2021.
[30] Jihoon Kim, Jiscob Kim, and Sungjoon Choi. FLAME: free-form language- based motion synthesis editing. In AAAI, 2023.
[31] Diederik P. Kingma and Max Welling. Auto-cncoding variational bayes. In ICLR, 2014.
[32] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint N00::2009009766, 2020.
[33] Nilesh Kulkamni, Davis Rempe, Kyle Genova, Abhijit Kundu, Justin Johnson, David Fouhey, and Leonidas Guibas. Nifty: Neural object interaction fields for guided human motion synthesis. arXiv preprint arXiv:2307.05511, 2023.
[34] Buyu Li, Yongchi Zhao, Shi Zhelun, and Sheng. Danceformer: Music conditioned 3d dance generation with parametric motion transformer. In AAAI, 2022.
[35] Ruilong Shan Yang. David A Ross, and Angjoo Kanazawa. choreographer: Music conditioned 3d dance generation with aist++. In ICCV, 2021.
[36] Han Liang, Wenqian Zhang, Wenxuan Li, Jingyi Yu, and Lan Xu. Intergen: cestusion-based multi-human motion generation under complex interactions. ar preprint arXiv:2304.0568) 2023.
[37] Dong C. Liu and Jorge Nocedal. On the limited memory BFGS method for large scale optimization. Math. Program., 1989.
[38] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: skinned multi-person linear model, ACM Trans. Graph., 2015.
[39] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR (Poster), 2019.
[40] Zhengyi Luo, Jinkun Cao, Alexander Winkler, Kris Kitani, and Weipeng Xu. Perpetual humanoid control for real-time simulated avatars. In ICCV, 2023.
[41] Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black. Amass: Archive of motion capture surface shapes. In ICCV, 2019.
[42] Dushyant Mehta, Oleksandr Sotnychenko, Franziska Mueller, Weipeng Xu, Srinath Sridhar, Gerard Pons- Moll, and Christian Theobalt. Single-shot multi-person 3d pose estimation from monocular RGB. In 3DV, 2018.
[43] OpenAL. GPT-4 technical report. arXiv preprint arXiv:2303.08774 2023.
12

---

[44] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani. Timo Bolkart, Ahmed A. A. Osman. Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3d hands, face, and body from single image. In CVPR. 2019.
[45] Mathis Petrovich. Michael Black, and Gül Varol. Action-condtioneed 3d human motion synthesis with transformer vae. In ICCV, 2021.
[46] Mathis Petrovich, Michael J Black, and Gül Varol. Temos: Generating diverse human motions from textual descriptions. In ECCV, 2022.
[47] Matthias Plappert, Christian Mandery, and Tamim Asfour. The KIT motion-language dataset. Big Data, 2016.
[48] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. /CML, 2021.
[49] Davis Rempe, Zhengyi Luo, Xue Bin Peng, Yuan, Kris Kitani, Karsten Kreis, Sanja Fidler, and Or Litany. Trace and pace: Controllable pedestrian animation via guided trajectory diffusion. In CVPR, 2023.
[50] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[51] Yonatan Shafir, Guy Tevet, Ray Kapon, and Amit H Bermano. Human motion diffusion as a generative prior. arXiv preprint arX1v:2333.01-118 2023.
[52] Jiaming Song. Qinsheng Zhang, Hongxu Yin. Morteza Mardani, Ming- Yu Liu, Jan Kautz, Yongxin Chen, and Arash Vahdat. Loss-guided diffusion models for plug-and-play controllable generation. In ICML, 2023.
[53] Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
[54] Sebastian Starke, He Zhang, Taku Komura, and Jun Saito. Neural state machine for character-scene interactions. ACM Trans. Graph., 2019.
[55] Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit Haim Bermano. Human motion diffusion model. In ICIRR 2023.
[56] Jonathan Tseng, Rodrigo Castellon, and C. Karen Liu. EDGE; editable dance generation from music. In CVPR, 2023.
[57] Joris Vaillant, Karim Bouyarmane, and Abderrahmane Kheddar. Multi-character physical and behavioral interactions controller. IEEE Trans. Vis. Comput. Graph., 2017.
[58] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
[59] Timo von Marcard, Roberto Henschel, Michael J. Black, Bodo Rosenhahn, and Gerard Pons-Moll. Recovering accurate 3d human pose in the wild using imus and moving camera. In ECCV, 2018.
[60] Jiashun Wang, Huazhe Xu, Medhini Narasimhan, and Xiaolong Wang. Multi-person 3d motion prediction with multi-range transformers. In NewrIPS, 2021.
[61] Jiashun Wang, Huazhe Xu, Jingwei Xu, Sifei Liu, and Xiaolong Wang. Synthesizing long-term 3d human motion and interaction in 3d scenes. In CVPR, 2021.
[62] Jingbo Wang. Sijie Yan, Bo Dai, and Dahua Lin. Scene-aware generative network for human motion synthesis. CVPR, 2021.
[63] Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, Wei Liang, and Siyuan Huang. HUMANISE: language- conditioned human motion generation in 3d scenes. In NewrlPS, 2022.
[64] Zeqi Xiao, Tai Wang, Jingbo Wang. Jinkun Cao. Wenwei Zhang, Bo Dai, Dahua Lin, and Jiangmiao Pang. Unified human-scene interaction via prompted Mhain-of-contacee arXiv preprint MXXi:2309.079/6 2023.
[65] Yiming Xie, Varun Jampani, Lei Zhong, Deqing Sun, and Huaizu Jiang. Omnicontrol: Control any joint at any time for human motion generation. arXiv preprint a000022310 2023.
13

---

[66] Sirui Xu, Zhengyuan Li, Yu-Xiong Wang, and Liang-Yoe Yan Gui. Interdiff: Generating 3d human-object interactions with physics-informed diffusion. In ICCV, 2023.
[67] Sirui Xu, Yu-Xiong Wang, and Liangyan Gui. Stochastic multi-person 3d motion forecasting. In ICLR, 2023.
[68] Ye Yuan, Jiaming Song, Umar Iqbal, Arash Vahdat, and Jan Kautz. Physdiff: Physics-guided human motion diffusion model. In ICCV, 2023.
[69] Jianrong Zhang, Yangsong Zhang. Xiandong Cun, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xi Shen, and Shan Ying. Generating human motion from textual descriptions with discrete representations. In CVPR. 2023.
[70] Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In ICCV, 2023.
[71] Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Mo- tiondiffuse: Text-driven human motion generation with diffusiin model. arXiv preprint arXiv:2208. 15001, 2022.
[72] Yunbo Zhang, Deepak Gopinath, Yuting Ye. Jessica Hodgins. Greg Turk, Jungdam Won. Simulation and retargeting of complex multi-character interactions. In SIGGRAPH (Conference Paper Truck), 2023.
[73] Kaifeng Zhao, Yan Zhang, Shaofei Wang. Thabo Beeler, and Siyu Tang. Synthesizing diverse human motions in 3d indoor scenes. arXiv preprint a3Xiv:2305. 2202222351111,,, 2023.
14

---

Appendix
A More Details about InterControl
Algorithm 1
Twoopeople interaction model inference
Require: Motion Diffusion Model M with parameter 0, Motion ControlNet with parameter di interaction
prompts p pmutts number of L-BFGS K, Forward Kinematics operation FK, masked selection operation
1: æt, 2T ~ (0,
2: for from T to do #LL.M-Planner
3:
p¹
mask +- mask ILM(p matts)
5: # Copy Spatial Condition from Each Other
6: c² +- S(FK(a))) mask)
7:
e² +- S(FK(xI) ), mask)
8:
Motion ControlNet
9:
[f] (wt, t,p", :);
10:
[f] + C (xl,t, ".";))
11:
# Mootio Diffusion Model
12:
T < M (x (
())
13:
vu f- M (xi,t,p
())
14:
Hº E. + L xf), E
# Posterior
15:
M < y (a (a). a!) E,
Posterior
16:
for & from to K do
17:
# IK guidance
18:
M pb; L-BFOS(L(M) p!)) end for
19:
20: xi-1 NW, 2x)
21: D;-1 N(p Ex)
22:
end for
23:
return mg.
A.1 Pseudo-code of IK guidance
Here we elaborate the details of IK guidance's algorithm. As we mentioned in the main paper, IK guidance could be performed predicted clean motion (i.e., 25g) posterior mean in denoising step t (i.e., (4.) In practice, we find that 2o works well in root control, and it does not require IK guidance in training Motion ControlNet, leading to faster training speed. Besides, it also requires less times of L-BFGS [37], which means faster inference speed. μa, leads to better FID in controlling all joints, yet requires more times of L-BFGS [37] and also need IK guidance training Motion ControlNet. We show the pseudo-code of InterControl in interaction generation in Algorithm 1.
A.2 Details of Motion ControlNet
In this subsection, elaborate the details of Motion ControlNet's architecture. Motion ControlNet is designed to adaptively generate realistic and high-fidelity motion sequences based on condition c. trainable copy of MDM, and each transformer encoder layer of ControlNet and the original MDM is connected by a zero-initialized linear layer, as shown in Fig. 5. The parameters in the original MDM pretrained and frozen in the entire training process. Thus, framework in the finetuning process starts from the weights that is equivalent to a pretrained MDM due to the zero-initialized linear layers. ControlNet will leam residual feature for spatial control signals in each transformer layer by the back propagated gradients. Thus, our model is able to implicitly adjust model weights for all joints and frames based on a sparse spatial condition e by learning the spatial- conditional distribution in addition to the semantic- distribution.
To process condition the uncontrolled joints, frames and XYZ-dim masked 0. Then use linear layer to project the condition e E IR NxaJ to the hidden dimension transformer layers as e Nx and feed e' to transformer encoder layers in ControlNet. We use a zero-initialized linear layer link the output each layer in ControlNet to the transformer encoder layer of pretrained and frozen MDM via residual connection [21]. We use extra information as condition for Motion
15

---

Motion Diffusion Model
Noisy E Motion
Transformer Layer
Transformer Layer
Tramsformer Layer
T0
Text P
prompt
Motion ControlNet
Zero-init
Zero-init
Zero init
Linear
Linear
Linear
Spatial
Linear Layer
Transformer
Transformer
Transformer
Control
Layers
Layer
Layer
Mask
Figure 5: Architecture of Motion ControlNet.
ControlNet final cat(c''c', nn,n"). main paper.
The details of ef has been explained in Sec 3.3 in our n²).
A.3 LLM-Planner
In this section, we further elaborate the details LLM Planner. Specifically, we collect 100 sentences describing human interactions with joint contacts from the description of InterHuman Dataset [36]. Then, we use GPT-4 [43] with the prompt in Tab. 7 to let GPT-4 to produce joint-joint contact plans for us. For each collected sentence, generate 10 task plans for us, as shown in Tab. 8. We manually correct typos of task plans generated by LLM, such as typos of joint name, invalid joint name, or invalid start frame or end frame. It leads to 989 valid task plans. Finally, we write Python scripts to transform the natural language tasks plans Python Json format, as shown in Tab. We take single-person language prompts in task plans as texts for motion diffusion model, and transform information in 'steps' to joint contact masks in the spatial condition. Specifically, we update the other person's joint positions as the current person's spatial condition in each denoising step, and use the spatial condition to guide Motion ControlNet and IK guidance in the same way with single-person scenarios. We evaluate the quality of interactions by using metrics like trajectory error and average error proposed by GMD [27] in the same way with single-person scenarios. We only evaluate on joints and frames in the joint-joint contact pairs. The result on our collected 989 task plans is shown in Tab. 5 in the main paper.
replace as the instruction in the prompt, and LLM will
B
Additional Experiments
B.1 Implementation Details.
We initialize parameters of both original MDM and Motion ControlNet from pretrained MDM [55] weight and freeze the parameters of original MDM during training. Following MDM [55], we use CLIP [48] model to encode text prompts. We run L-BFGS [37] in IK guidance times for the first 990 denoising steps and 10 times for last 10 denoising steps on the posterior mean μly and once for the first 990 steps and 10 times for the last 10 steps clean motion Zg. We use IK guidance in training ControlNet when using it on μt, We set two types of mask m E {0, 1 x (1) Only keeps pelvis (root) joint for root control fairly compare with previous methods: (2) Randomly keep one joint in each iteration to learn to control all joints for interaction generation. Each type of mask will be used in both training and inference for consistency. Thus, we get two model weights, where (1) could be fairly compared with previous methods and we use (2) for interaction generation. We AdamW [39] optimizer and set the learning rate le-5.
B.2 Text-to-Motion Generation Results
To generally compare our InterControl with previous text-conditioned motion generation methods, report the alignment quality of text and generated motions suggested by Guo et. al. [14] in Tab. 5.
16

---

Table 5: Text-to-motion evaluation on the (left) HumanML33 [14] and (right) KIT-MI [47] datasets. The right arrow - means closer to real data better. Methods in the upper part are unable perform spatial control. means our implementation.
HumanML3D
FID
R-precision (Top-3)
Diversity
KIT-ML
FID 4
R-prevision (Top-3)
Divensity >>
0.002
0.797
9.503
JL2P
11.02
0.486
7.676
0.031
0.779
11.08
Text2Gesture [4]
7,664
0,345
6.409
T2M [14]
3.022
0.681
10.72
T2M [14]
0.740
9.188
MotionDiffuse [71]
1.954
0.739
11.10
MotionDiffuse [71]
0.630
0.782
9,410
MLD [6]
0.404
0.734
10.80
ML.D [6]
0.473
0.772
9.724
T2M-GPT [69]
0.514
0.745
10.92
PhysDiff [68]
0.433
0.631
MotionGPT [25]
0.510
0,680
10.35
T2M-GPT [69]
0.116
0.775
9.761
MotionGPT [25]
0.232
0.778
9,528
MDM [55]
0.497
0.396
10.84
MDM [55]
0.544
0,611
9.446
PriurMDM [51]
0.830
0.397
10.54
PriorMDM [51]
0,540
0.640
9.160
GMD [27]
1.537
0,385
9.78
GMD
0.212
0.670
9,440
OmniControl [65]
0.702
0.397
10.93
OmniControl [65]
0.218
0.687
9.422
Our InterControl
0,580
0,397
10.88
InterControl
0.671
9,482
R-precision f Diversity
Foot skating
Traj.
Loc.
Avg. err. (m)
Method
Joint
FID 4
(Top-3)
ratio |
(50
(50 cm)
Ours (all)
Root
0.184
0.672
9.315
0.1044
0.0317
0.0018
0.0693
Ours (all)
Left foot
0.242
0.664
9.184
0.1005
0.0696
0.0024
0.0671
Ours (all)
Right foot
0.236
0.669
9.201
0.0983
0.0798
0.0029
0.0680
Ours (all)
Head
0.172
0.678
9.359
0.0958
0,0523
0.0044
0,0846
Ours (all)
Left wrist
0.260
0,660
8.965
0.0915
0,0375
0.0012
0.0874
Ours (all)
Right wrist
0.284
0.655
9.003
0.0920
0.0364
0.0010
0.0872
Table 6: Spatial control results on the HumanML3D [14] dataset. Ours (all) means the model is trained on one randomly selected joint among all joints each iteration.
Note that methods in the upper part both tables are unable to perform spatial control, thus they are incapable of generating controllable motions and interactions even if they have lower FID or higher R-precision. For instance, T2M-GPT [69] and MotionGPT [25] tokenize human poses to discrete tokens and is unable to incorporate any spatial information. MLD [6] uses latent diffusson to accelerate denoising steps, yet performing spatial control needs to convert each step latent feature back to motion representations. It leads to much more computation than MDM [55] and is opposite to MLD's s motivation of latent diffusion. Among methods that are suitable for spatial control [51, 27] in Tab. 5, InterControl achieves the best performance in most of semantic-leve metrics, and is better than the recent work OmniControl [65] that focuses on single-person motion yet shares similar design of spatial controlling with us.
B.3 More Single-joint Control Results
In Tab. of our main paper, we have shown the spatial control results with root joint and randomly selected one/two/three joints. Following the recent work [65], we also show the spatial control performance on specific joints in Tab. 6. We find that feet and hands are more difficult to control due their flexibility, while root (pelvis) and head are more easier follow, leading better FID and R-precision.
B.4 Details of User Study
In the user study, our method generates 50 samples from the contact plans collected from LLM- planner. We also use the original interaction description to generate two-person interactions from ComMDM PriorMDM [51]. Fig. 6, we show our designed questionnaire's evaluation instructions and the first question as an example. Each questionnaire has 19 single choice questions randomly sampled from all samples. In the folder named sssr--sudy-videos' we provide 25 videos sampled from our Intercontrol and PriorMDM for reference.
17

---

Evaluation of two-person motion interactions
This is set of ainllecchoiee questions that will take approximately 5-6 minutes to complete. The questionnaire measures which set of interactions in paired motions better matches the language description and is more natural. The videos show skeleton representations similar to stick figures, with purple and yellow representing two individuals, and each video is about 5 seconds long. The selection priority is as follows: first, choose the interactive motion more closely matches the languege description; if they are equally matched, then choose the motion that more natural. Naturalness includes whether the movement speed of the person's limbs is reasonable, whether the feet are suspended the air, whether there sliding between the feet and the ground, etc. Unreasonable parts are considered unnatural. The descriptions may include words such as swords, shields, mobile phones, and other items, but all the videos only show human skeletons, and the items are not displayed. actions matches the description and is natural if the person is assumed to be holding these items their hands.
can consider whether the set
*1. Which of the following sets of interactive motions between the stick figures appears more consistent with the description and more natural? Description: One person extends the right hand toward the other.
11
O
Figure 6: Example of the questionnaire of user-study.
B.5 Details of Evaluation Metries
Here we select some descriptions for metrics used to evaluate controllable motion generation methods from HumanMI.3D [14] and GMD [27] to save reader's time.
Semantic-level Evaluation Metries from HumanMLLD [14]: Frechet Inception Distance (FID), diversity and multi-modality. For quantitative evaluation, motion feature extractor and text feature extractor is trained under contrastive loss to produce geometrically close feature vectors for matched text-motion pairs, and vice versa. Further explanations of aforementioned metrics as well as the specific textual and motion feature extractor are relegated to the supplementary file due to space limit. In addition, the R-precision and MultiModal distance are proposed this work as complementary metrics, follows. Consider R-precision: for each generated motion, its ground-truth text description and randomly selected mismatched descriptions from the test set form a description pool. This is followed by calculating and ranking the Euclidean distances between the motion feature and the text feature of each description in the pool. We then count the average accuracy at top-1, top-2 and top-3 places. The ground truth entry falling into the top-k candidates is treated successful retrieval, otherwise it fails. Meanwhile, MultiModal distance is computed as the average Euclidean distance between the motion feature of each generated motion and the text feature of its corresponding description in test set.
Spatial-level Evaluation Metrics from GMD [27]: We use Trajectory diversity, Trajectory error, Location error, and Average error of keyframe locations. Trajectory diversity measures the root mean square distance of each location of each motion step from the average location of that motion step across multiple samples with the same settings. Trajectory error is the ratio of unsuccessful trajectories, defined as those with any keyframe location error exceeding threshold. Location error
18

---

is the ratio of keyframe locations that are not reached within threshold distance. Average error measures the mean distance between the generated motion locations and the keyframe locations measured at the keyframe motion steps.
19

---

Table 7: Detailed prompting example of the LLM Planner.
Input
Instruction: two people greet each other with a handshake, while holding their cards in the left hand. Given the instruction, generate 10 task plans according to the following background information, rules, and examples. Each task plan should completely reflect an entire process of actions described in the instruction. [start of background Information
Human has JOINTS: ['pelvis'. 'left hip'. 'right hip'. "left_nne"" knee'. 'right knee'. 'left ankle", 'right_ankle", "left foot", 'right_fool "neck'. 'leftt collar'. 'right collar 'head', reit_shoulder". 'right shoulder' elboow, hight_elbow", "left_wrist" 'right_wrist' [.
The total number of TIME-STEPS of human motion is 99, the frame- sene--er-sccond of motion is 20. The provided text instruction is describing two people performing some actions containing human joint contacts. The height of all people is 1.8 meters, the arm length is 0.6 meters, and the leg length is 0.9 meters. Two people are meters away at the beginning (i.e.. TIME-STEPS=O). lend of background Information]
istart of rules]
I. Each task plan should be composite into detailed steps.
2. Each step should contain meaningful joint-joint pairs.
3. Each joint-joint pair should be formatted into (JOINT, JOINT, TIME-STEP, TIME-STEP, CONTACT TYPE, DISTANCE). JOINT should be replaced
JOINT in the background information. IMPORTANT: The first JOINT belongs to person 1, and the second JOINT helongs to person 2. Each joint-joint pair represents a contact of a joint of person and a joint of person 2. The first TIME-STEP is the start frame number of contact, and the second TIME-STEE MEMETTPPEEEE is the end frame number of contact. CONTACT TYPE should be selected from (contact, avoid]. DISTANCE should be float number representing how many meters should be the distance of two joints in the joint-joint pair. For CONTAATTTPPEE contact), the distance of two joints should be SMALLER than the DISTANCE; for [CONTACT TYPE: avoid), the distance of two joints should be LARGER than the DISTANCE. IMPORTANT: Consider the transition of contact types, leave time-steps more than 20 frames without any joint-joint pair between different contact types. Use small DISTANCE variance between different contact types: for the joint-joint pairs that are with [CONTACT TYPE: contact), do NOT use DISTANCE larger than 0.5m in the following [CONTACT TYPE: avoid]; for the joint-joint pairs that are with [CONTACT TYPE: contact), use [CONTACT TYPE; avoid] after 20 frames; for the joint-joint pairs that are with [CONTACT TYPE: avoid', use NO joint pairs for 20 frames if the following CONTACT TYPE is contact. Try to not uver-use [CONTACT TYPE: avoid]: if there is explicit semanties of being far away. just do not use joint-joint pair in that frames; if there explicit semantics of being far away, then use joint-joint pair with [CONTACT TYPE: avoid].
4. Consider which JOINT will he interacted when two people perform the action described in the text instruction. Translate the text instruction to be steps of joint-joint pairs. Do not include extra joint-joint pairs that is unrelated the text instruction. IMPORTANT: make joint-joint pairs in different task plans diverse in TIME-STEPS and JOINTs. Each joint-joint contact pairs should be lasting from 3 10 frames.
5. Be plausible. Do not generate uncommon interactions. Generate plausible interaction time-steps, and consider the velocity of human motions.
6. Usc one sentence to deseribe what action should person 1 do and one sentence to deseribe what action should person do according to the text instruction at the beginning the task plan. IMPORTANT: the sentence starts from "text 1:' describing the action of person from the perspective person and the sentence starts from 'text 2:' describing the action of person 2 from the perspective of person 2. Sentences should NOT contain words like 'person I' or person 2', use 'a person' to refer to himself in the sentence and 'others' to refer others.
7. The steps in the task plan are for both two people. Use one set of steps to describe both two people. The first
JOINT belongs to person and the second JOINT belongs to person 2.
8. IMPORTANT: Do NOT add explanations for the steps in task plans. Each step only have one joint-joint pairs.
(end of rules)
(start of example]
Instruction: two people greet cach other with a handshake, while holding their cards in the left hand.
|Start of Plan 1]
Text 1: a person make a handshake with others using his right wrist, while holding his cards in the left wrist.
Text 2: person make handshake with others using his right wrist, while holding his cards in the left wrist.
Step 1: (right wrist, right wrist, 0, 10, avoid, 0.3)
Step (right wrist, right wrist, 50, 60, contact, 0.05)
Step 3: (right wrist, right wrist, 90, 100, avoid, 0.3)
[End of Plan 1]
(end of an example]
20

---

Table 8: Example of the LLM generated task plans.
LLM Generated Task Plans (only show due to page limit)
Instructions: The first feneer lunges at the second, who parries the attack and counters with a successful strike to the head.
[Start of Plan 1]
Text 1: A person lunges towards another with his right foot.
Text 2: A person parries the lunged attack while preparing to counter.
Step 1: tright_foot, left_ fneeknee,,5 5, 10, contact, 0.3)
Step 2: (right wrist, left_ ceft_collar, 20, 30, avoid, 0.3)
Step 3: [left_elboww head, 70, 80, contact, 0.05)
[End of Plan 1]
[Start of Plan 2]
Text 1: A person lunges at the other person with his right foot.
Text 2: person blocks the lunged attack.
Step foot, left ankle, 3, 10, contact, 0.2)
Step 2: (right_wre wrist, right_collar, collar, 20, 30, avoid, 0.25)
Step 3: [left_wrist, head, 70, 79, contact, 0.02)
[End of Plan 2]
[Start of Plan
Text person takes a lunge step towards another.
Text 2: A person parries the attack and counters.
Step 1: (right_foo foot, right_knee, knee, 7, 14, contact, 0.3) Step (left_wrist, right_collaa 22, 30, avoid, 0.25)
Step 3: (right_ wrist, head, 69, 77, contact, 0.03)
[End of Plan 3]
[Start of Plan
Text 1: A person lunges northerly towards another with his left foot.
Text 2: A person parries the attack and prepares countcrattack.
Step 1: (left_foot, right_ankle, 6, 10, contact, 0.35) Step (left_wrist,
collar, 22, 30, avoid, 0.28)
Step 3: (right_elhow, head, 71, 80, contact, 0.05) [End of Plan 4]
[Strrt of Plan
Text person lunges at another using his right foot.
Text person deflects the approaching lunge and immediately counters.
Step 1: (right foot, left knee, 4. 12, contact, 0.31]
Step wristt right shoulder, 20, 30, avoid, 0.3)
Step 3: (right wrist, head, 73, 81, contact, 0.05)
[End of Plan 5]
21

---

Table 9: Example of processed json file from task plans generated by LLM.
Processed Json format, (only show due to page limit)
Format of 'steps': [index of contact joint of person index of contact joint of person 2,
start frame, end frame,, contact type (contact = 1, avoidance = 0), desired distance (unit as meter) ). .......
"text personl": "A person lunges towards another with his right foot.", "text person2": "A person parries the lunged attack while preparing to counter.", steps":
11,, 4,5, 5, 10, 0.3). 21, 13, 30, 40, 0, 0.3]. [1, 15, 70, 80, 1,
"text pexsone" "A person lunges at the other person with his right foot.",
"text person2": "A person blocks the lunged attack.", "steps":
11, 7, 3, 10, 1, 0.2), 21. 14. 30, 40, 0, 0.25]. [20, 15. 70, 79, 1, 0.02
"text_person!" I "A person takes lunge step towards
another."
person2":
"A person parries the attack and
ters.".
"steps":
11. 5, 14, 1, 1.3) 20, 14. 34, 42, 0, 0.25). 15, 69, 1, 0.03
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We are the first method to perform zero-shot human interaction generation by leveraging only ainleepprsoon motion generation model, which could be supported by abstract, introduction and method.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We have discussed our limitation in Sec. Conclusion and Limitations. The main limitation is that we only investigated certain form of interactions which could be quantitatively described by spatial relations.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
23

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
We do not have theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have included all the information to reproduce the main experimental results, and we also provide the code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
24

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification:
We have provided the code in the supplemental material and the website.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nip) cc/
publ c / gu des CoodeSuom i ss onPol cy) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips. cc hts/publiic /ugu/u/uu/a ////uudeee iede//// CCodeeodeuuua Submi as sswsssolssssooooooooooooooooooom milycy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We include all training and test details in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: It will be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
25

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We include the information of computer resources in appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /neurips.....// httpp//cc/ttttttttiiiieeeeeeeeee
Question:
Answer: [Yes]
Justification: Our paper follows the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We have carefully considered potential societal impacts and determined that technical contribution generating 3D skeletal animations poses minimal risks. Our method is designed for generating multi-people 3D skeletons, and these skeletal representations do not pose negative societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
26

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not pose risks requiring such safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited the used code, data and models.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use package should be provided. For popular datasets, papersw rswi thoode. com / da sets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
27

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We have included the documentation of our code the supplementary materi-
als.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
28


Reviews:
[{"review_id": "di19jVjOiH", "reviewer": "Reviewer_VAgJ", "summary": "### Summary\n\nThe paper \"InterControl: Generating Human Motion Interactions by Controlling Every Joint\" aims to generate interactions between multiple people based on text descriptions, with precise joint control. It leverages a pre-trained single-person motion diffusion model and extends it to multi-person scenarios using a large language model (LLM) to guide joint interactions.\n\nThe methodology involves:\n- Using an LLM planner to map out interactions and distances between joints.\n- Combining priorMDM with ControlNet, conditioned on LLM-extracted distances.\n- Optimizing interactions with LGBFS to enhance performance.\n\nEvaluations are conducted using Text2Motion datasets and user studies, focusing on joint control and positional accuracy. The approach also assesses 100 interactions from the InterHuman dataset.\n\n### Contributions\n\n1. **Dynamic LLM Guidance**: Introduces LLMs to guide joint interactions dynamically, making the process scalable and suitable for large-scale data generation.\n2. **Precise Joint Control**: Allows precise control over any joint, improving flexibility compared to methods requiring predefined control signals.\n3. **Multi-Person Interaction**: Extends a single-person model to handle multi-person interactions, demonstrating scalability.\n4. **Enhanced Visualization**: Suggests using human body models like SMPL or GHUM for better visualization and interpretability of interactions.\n5. **Comprehensive Evaluation**: Provides thorough evaluation and comparison with previous models, addressing issues like interpenetration and alignment between control and text prompts.\n\nWhile the paper presents promising results and tackles the challenging task of generating realistic multi-person interactions, it also identifies areas for further improvement, such as better handling interpenetration and improving alignment between spatial control and text conditions.", "questions": "### Questions and Suggestions for Improvement\n(importance sorted)\n\n**Evaluation and Results**:\n   - Expand evaluations to include more comprehensive datasets, especially those involving multi-person interactions, and provide a detailed performance analysis.\n   - Provide an analysis of the LLM's error rates in generating plausible contact maps and discuss how this impacts the model's performance.\n   - Evaluate the model on available multi-person datasets and collect text descriptions for these datasets to provide a thorough assessment.\n\n**Methodological Improvements**:\n   - Address the issue of interpenetration in generated motions by refining IK guidance and adding constraints to improve realism.\n   - Elaborate on the process of fine-tuning ControlNet on the HML3D dataset and extracting necessary control features.\n   - Implement and compare a simpler baseline that generates each person’s motion using existing methods and optimizes interperson distances using LGBFS.\n\n**Clarification of Contributions**: Clearly distinguish the paper's contributions from existing works like OmniControl [49] and GMD [26]. Highlight the novel aspects and how this work advances the field beyond these prior studies.\n\n**Visualization and Presentation**:\n   - Use human body models like SMPL for better visualization of motions, facilitating better perception of contact and interaction.\n   - Include mesh-based visualizations and use original AMASS skeletons for training to enhance interpretability and clarity of results.\n   - Improve the clarity of the presentation, particularly in explaining complex steps and figures, such as the Gaussian noise issue and Figure 2.\n\nBy addressing these points, the paper can provide clearer contributions, ultimately strengthening the overall quality and impact of the work.", "limitations": "The Limitations are discussed.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The use of large language models (LLMs) to dynamically guide joint interactions introduces a scalable and innovative approach to multi-person motion generation.\n- The method allows for precise control over any joint at any time, enhancing flexibility and improving upon previous methods that relied on predefined control signals.\n- The approach successfully extends a single-person motion model to generate realistic interactions among multiple people, demonstrating its scalability and applicability to complex scenarios.\n- The paper is well-written and easy to follow, with a clear presentation of previous works and a strong introduction that provides good context.\n- The simplicity of the IK guidance and LLM-generated contact points helps automate the generation process, which is beneficial for large-scale data generation pipelines.\n- The paper includes comprehensive evaluations and comparisons with existing models, showing that the results for single-joint control and multi-person interactions are indeed better in several aspects.\n- The integration of ControlNet and LGBFS optimization with LLM planning is a novel contribution, pushing the boundaries of 3D human motion generation.", "weaknesses": "- The contributions of the paper seem limited, with similarities to existing works like OmniControl and GMD.\n- The generated motions often suffer from interpenetration, which reduces the realism of the interactions.\n- There are discrepancies between the spatial control and text conditions, suggesting that the alignment between them needs improvement.\n- It is unclear how ControlNet is finetuned, especially regarding the use of the HML3D dataset and the extraction of necessary control features.\n- A simpler baseline for generating each person’s motion and optimizing interperson distances is missing, which could highlight the necessity of additional modules.\n- The qualitative results lack sufficient visualization, making it difficult to assess the plausibility of interactions from the provided videos.\n- The method focuses on multi-person motion generation but only evaluates single-person text-to-motion datasets, missing evaluations on available multi-person datasets.\n- The use of 3D joint locations without a human body surface makes it hard to judge motion plausibility and perceive contacts between people.\n- The presentation lacks clarity in some sections, particularly in explaining complex steps and figures, such as the Gaussian noise issue and Figure 2.\n- The approach relies on LLMs for joint distances, which may hallucinate content and lead to errors in generating plausible contact maps.\n- The work does not include suitable multi-person motion capture data during training, limiting the robustness of the interactions modeled.\n- The focus on joint-to-joint contacts overlooks other types of human contact, such as grabbing an arm, which are not modeled effectively.\n\n___\n\nSmall notes\n- The teaser is misleading as it suggests conditioning on images for interaction generation.\n- The claim that TEMOS performs worse than MDM is not substantiated; TEMOS actually performs better in some cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the detailed review. Please refer to general response for more common questions.\n\nQ1: Similarity to OmniControl and GMD.\n\nA1: (1) Our method could control all joints while GMD only controls root joint. (2) We focus on zero-shot interaction generation and use controllable single-person motion generation as the approach to achieve this goal, while OmniControl only consider single-person motion. Our method shows better joint control ability in non-root joint control (refer to Table 1 of our paper, and qualitative results). Besides, our work is concurrent to OmniControl and we design InterControl independently.\n\nQ2:  Discrepancies between the spatial control and text conditions.\n\nA2: It is unclear the meaning of discrepancies. We will appreciate it if the reviewer could elaborate it. \n\nQ3: How Motion ControlNet is finetuned.\n\nA3: We have included the training details of Motion ControlNet in Line 193-199. The control features are global positions converted from HumanML3D data format, by using forward kinematics. \n\nQ4: A simpler baseline for optimizing inter-person distances on single-person motion should be compared.\n\nA4: It is unclear that how the 'inter-person distances' in the review is defined. If it is defined on the distance of any joints from different people, our method itself is similar to such 'simple baseline'. If is is only defined on root joint, GMD is such baseline for spatial control, which has been included in the spatial control comparison quantitatively. \n\nQ5: Qualitative results lack sufficient visualization.\n\nA5: We have provided 3D-skeleton visualization on user study cases and in-the-wild cases with both two-people and three-people scenarios. Furthermore, we provided physical animation results from our generated kinematics-based motions for better illustration of the execution of a hard interaction case: fighting. The effectiveness of our visualization is agreed by reviewer#HwRy. \n\nQ6: Missing training and evaluations on multi-person datasets.\n\nA6: As our method is a zero-shot interaction generation method, we only train our method on single-person motion datasets and it does not learn data distribution on multi-person datasets. Thus, it is not suitable to evaluate FID or Top-3 precision on these benchmarks. The usage of multi-person motion capture data during training is beyond this paper's scope and we leave this to our future work. \n\nQ7: 3D joint locations without a body surface makes it hard to judge motion plausibility and perceive contacts.\t\n\nA7: As the mainstream motion datasets adopt 3D joint locations as motion data, it is more direct to visualize motions with 3D skeleton. SMPL visualization used by previous methods need an additional step that utilize SMPLify method to convert 3D joint locations to SMPL meshes. However, such conversion will introduce errors and the converted mesh will not be faithful to the original motion output. Furthermore, the mainstream motion datasets such as HumanML3D commonly remove the hand joints from SMPL, leading to a 22-joint data format instead of 24-joint data format in original SMPL. Therefore, many interactions could only be achieved by using wrists instead of hands in current interaction results. The effectiveness of our 3D joint location visualization is agreed by three other reviewers. Finally, we have shown that physical animation could be useful to improve the plausibility of both motion and contact. Our method has the potential to further be improved by leveraging surface optimization methods.\n\nQ8: More explanations on Gaussian noise and Figure 2.\t\n\nA8: We will carefully revise our paper in the final version. \n\nQ9: The LLM may hallucinate content and make errors in generating contact maps\t\n\nA9: Thanks for this suggestion. We agree that LLM could hallucinate contents in our framework. As our main motivation and contribution is to design a zero-shot interaction generation framework, we leverage LLM to provide necessary information for this goal. From our empirical results, we think LLM (e.g., GPT-4 in our experiments) could provide a reasonable contact plan in most cases, which is agreed by many previous works in robotics, such as [64] and [a].\n\nFurthermore, we believe the contact plan generated by LLM will be more reliable with the new improvements of LLMs. Whether or not LLMs could guarantee to generate reliable contact plan without hallucination is beyond our paper's scope. \n\nQ10: The focus on joint-to-joint contacts overlooks other types of human contact, such as grabbing an arm.\n\nA10: Thanks for this insightful suggestion. We agree that our joint-to-joint contacts could not handle all kinds of human contact, such as grabbing or hugging. We have clearly discussed such limitation of interaction definition in Line 35-43. Furthermore, our method could be seamlessly extended to joint-to-bone contacts by sampling keypoints in the bone. It could be easily achieved by interpolation between two adjacent joints.\n\nQ11: Teaser could be misleading.\n\nA11: Thanks for the suggestion. We want to illustrate that our definition of interactions could result in meaningful human interactions in our daily life. As we clearly mentioned zero-shot motion interaction generation in the title and paper, we believe the teaser will not be misleading or hard to understand, which is agreed by other reviewers. \n\nQ12:TEMOS performs worse than MDM is not substantiated.\n\nA12: We does not claim that TEMOS performs worse than MDM in our paper. We will appreciate that if reviewers could check our paper again. \n\nQ13: Analysis of the LLM's error rates.\n\nA13: As LLM is not our focus, we empirically show that LLM could work well in the interaction generation. It is worth noting that we use an off-the-shelf LLM without any finetuning. The effectiveness of LLM has been agreed in many previous works, such as [64] and [a].\n\n[a] Song, Chan Hee, et al. \"Llm-planner: Few-shot grounded planning for embodied agents with large language models.\"  in CVPR, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XUpJ4qlj1S", "reviewer": "Reviewer_HwRy", "summary": "The paper introduces InterControl, a method designed to address the task of controllable human motion generation and zero-shot human interaction generation. By leveraging the prior knowledge of LLM, InterControl can generate human interactions involving any number of people in a zero-shot manner. Specifically, the authors utilize LLMs to convert textual descriptions of interactions into contact plans, transforming the task of multi-person interaction generation into single-person motion generation. The InterControl model, which is based on Motion ControlNet and IK Guidance, is then used to achieve controllable single-person motion generation. Experimental results demonstrate the superiority of InterControl over previous controllable motion generation methods and its ability to produce realistic human interactions.", "questions": "- What is the corresponding text for the supplementary material?\n- Can the LLM really provide a reliable contact plan?", "limitations": "Although the authors included a \"Limitations'' subsection in Sec.5, from my perspective, they did not clearly claim the limitations of the method. The authors may want to point out that the realism of the interactions generated by InterControl heavily depends on the LLM's correct interpretation of the text, which cannot be guaranteed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed task(zero-shot human interaction generation) is interesting, important, challenging, and meaningful.\n- The related work section is great, providing an excellent summary of relevant research.\n- The proposed method makes sense and is logically sound.\n- The results are reasonable. I particularly appreciate the application section, where the character interactions in the simulation are executed very well.", "weaknesses": "- The qualitative results are not really good. a) The supplementary video shows some cases where some joints have abrupt velocity changes, which conflicts with L232: \"IK guidance can adaptively modify velocities from the start to frame n.\" b) There are some cases of body penetration and unrealistic interactions in the three-people-fighting/two-people-fighting scenarios.\n- It might be beneficial to incorporate some visualization of the contact plan. According to Sec 3.1, contact should include both contact and avoid forms, described by distance d. However, in the supplementary demo, I only observe d=0 contact. I would like to see some examples of avoidance and how non-zero distance contact plans guide motion generation.\n- Maybe more discussion is needed on the differences between omnicontrol and intercontrol, as they are similar methods (both of InterControl/OmniControl use ControlNet-like framework and classifier guidance). The author may want to discuss if using OmniControl as the control yields better results in the task of zero-shot human interaction generation?\n- The paper didn’t compare InterControl with data-based methods such as InterGen. The author may want to demonstrate the superiority of InterControl as a zero-shot method, possibly by highlighting interactions that InterGen cannot generate (2 people interaction).\n- It might be beneficial to include some qualitative comparisons between InterControl and state-of-the-art methods(e.g. OmniControl) in the task of Single-Person Controllable Motion Generation to better demonstrate the superiority of InterControl.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the insightful review. We will revise our paper according to the constructive suggestions in our final version. Please refer to General Response for comparison with OmniControl and InterGen, and the explanation of penetration and unsmooth motion issues.\n\n**Q1: Examples of avoidance.**\n\n**A1**: Thanks for the kind suggestion. We do have examples of avoidance in our qualitative results, yet we did not annotate them out. For example, the 11-th second of video 'two-people-dancing.mp4' (or the Figure 1 of the attached PDF in the author rebuttal) shows an example of contact and avoidance at the same time: when two characters are holding hands, their other hands are away from each other by at least 2.4 meters. Such avoidance leads to stretching dance motions, which could not achieved by joint contact alone. \n\n**Q2: Discussion of the differences between omnicontrol and intercontrol.**\n\n**A2**: First of all, we want to clarify that our work is an concurrent work with Omnicontrol, and we are not aware of this paper when we did our work. We have discussed the difference of technical contribution in Sec.A.4. Besides, our work shows notable improvement of non-root joint control over OmniControl in HumanML3D dataset. We have included the zero-shot interaction generation results of OmniControl in the Table of A2 in general response.\n\n**Q3: Qualitative comparisons between InterControl and OmniControl.**\n\n**A3**: We have include detailed quantitative comparsion with OmniControl in Table.1. Besides, our main contribution is the ability of zero-shot interaction generation. The proposed single-person spatial control is an approach to achieve our main contribution. In Figure 2 of the attached PDF in the author rebuttal, we show a qualitative comparison between InterControl and OmniControl, where our method shows better hand joint control in hand-shaking.\n\n**Q4: Corresponding text for supp. mat.**\n\n**A4**: For user study videos, the texts are in 'two_person_text_prompts.txt'. For visualization videos, such texts are writted by ourselves to show in-the-wild results. Here are specific texts: 'two-people-dancing.mp4': Two people are dancing, sometimes they stand in an open dance position: one hand joined together, while their other hands are extended outward. 'two-people-winning-gesture.mp4': Two people walk slowly, their arms raised high while holding each other's hands, displaying a triumphant gesture after a fighting contest. 'three-people-holding-hands.mp4': One person held the hands of two others, each with one hand, forming an arc with all three facing the same direction. 'three-people-fighting.mp4': Two people are fighting against a third person, using their wrists and feet to attack. The third person is also counterattacking.  'two-people-fighting.mp4': One person is fighting against another person, using his wrists and feet to attack. The second person is also counterattacking. \n\n**Q5: Can the LLM really provide a reliable contact plan?**\n\n**A5**: Thanks for this insightful review. We agree that LLM is not guaranteed to provide a reliable contact plan in our framework. As our main motivation and contribution is to design a zero-shot interaction generation framework, we leverage LLM to provide necessary information for this goal. From our empirical results, we think LLM (e.g., GPT-4 in our experiments) could provide a reasonable contact plan in most cases, which is agreed by many previous works in robotics, such as [64] and [a]. Whether or not LLMs could guarantee to generate reliable contact plan is beyond our paper's scope. \n\n**Q6: Limitations.**\n\n**A6**: We agree that the LLM's correct interpretation of interaction descriptions cannot be guaranteed. As our paper's focus is not LLM, our framework provide a pioneer exploration to perform zero-shot interaction generation, and we empirically find that LLM works well in most cases. With the improvement of LLM's knowledge and robustness in the future, we believe the contact plan generated by newer LLMs will be more reliable.\n\n[a] Song, Chan Hee, et al. \"Llm-planner: Few-shot grounded planning for embodied agents with large language models.\"  *in CVPR,* 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6s2L3jlcde", "reviewer": "Reviewer_7GL6", "summary": "This paper tackles the challenge of generating human interaction motions involving a flexible number of characters. To simplify the representation of these interactions, the authors propose a joint-pair contact/avoid representation. Given an interaction description, a large language model (LLM) generates motion instructions and identifies contact joint pairs. These outputs serve as inputs for the subsequent motion diffusion model. The Motion ControlNet is trained to produce motion sequences based on the motion instructions and to optimize the joints to align with the specified joint contact pairs.", "questions": "This paper is addressing an important task - human interaction generation, by only leveraging single-person dataset. In addition to some of my questions presented in the weakness section, I have a few general questions here, and I am glad to hear some feedback or insights from authors:\n1. If we want to scale up to more involved human characters\n* How about the efficiency at inference time of the proposed pipeline?\n* Would LLM planning still be effective and fine-grained enough for multiple-person interactions?\n2. It would also be insightful to see the comparison with Intergen or fine-tuning on InterHuman dataset as well in the two-person interaction scenario.", "limitations": "Yes, the authors have addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is addressing an important human interactions learning task that is not well studied due to the lack of multi-person interactions datasets, and this paper suggests a framework that only leverages the single-person dataset to produce reasonable multi-person interactions.\n2. The joint-pair distance representation of human interactions are simple and the IK-guidance shows to be effective.\n3.  The manuscript is well-written and easy to follow.", "weaknesses": "1. It is not clear how does the contact joint pairs for each frame are generated by LLM. A joint jump usually presents when the contact between two humans happens, and what can be this unsmooth motions artifacts result from?\n2. I still observe that there are still severe human collision happens in some generated human interactions. The joint pair contact type produced from LLM might not be fine-grained enough to produce natural and plausible interactions. The followed physical simulator tracking step might mitigate the collision artifacts a bit, but still it needs better spatial control over body roots to avoid severe collisions during motion.\n3. The iterative IK guidance based optimization happens at each diffusion step, and this might result in less efficient framework. Might directly optimise the noise like done in DNO [1] or a single-step post-optimisation also work?\n\n[1] Optimizing Diffusion Noise Can Serve As Universal Motion Priors, CVPR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the insightful review. We will revise our paper according to the constructive suggestions in our final version. Please refer to General Response for comparison with OmniControl and InterGen, and the explanation of penetration and unsmooth motion issues.\n\n**Q1: How does the contact joint pairs for each frame are generated by LLM?** \t\n\n**A1:** The format of contact joint pairs (contact plans) is in Table 9 of our Appendix. The prompt for generating contact plans by LLM is shown in Table 7 of our Appendix, where the raw output of LLM is in Table 8. \n\n**Q2: Comparison with DNO, and efficiency issues.**\n\n**A2:** Thanks for the insightful suggestion. Firstly, the concurrent work DNO is a good exploration of motion editing which shares similar idea with our IK guidance. However, it also require 300 or 500 steps for motion editing or refinement, which takes more than 3 minutes according to their paper (Sec.6 in page 6). On the contrary, our method need 80s for inference, while previous methods takes more seconds to control the motions following spatial conditions (GMD needs 110s and OmniControl needs 120s). Secondly, if the inference speed is really important, we could use other speed-up techniques in diffusion models to improve the inference speed, such as DDIM and Consistency Model. For example, our framework could utilizes the recent MotionLCM (https://dai-wenxun.github.io/MotionLCM-page/) to speed up the inference to 30ms per denoising step and get realistic motions within 4 steps. Yet, our main contribution of zero-shot interaction generation ability will not be influenced. \n\n**Q3: Scale up to more characters.**\n\n**A3:** Our framework is able to perform motion generation in a batch to speed up multi-human motion generation, where all characters' motion are generated together. Thanks to the batch computation ability in GPUs, the inference time is almost the same with single-person motion generation. The IK guidance optimization process will add little extra burden with the increase of number of characters. In practice, 2-character inference takes about 80s and 3-character inference takes about 90s with a standard 1000-step DDPM inference. As we mentioned above, we could use MotionLCM to further speed up the inference process of multi-character interaction generation.\n\n**Q4: LLM planning for more characters**\t\n\n**A4:** We do not collect contact plans of more characters from LLM for quantitative experiments. Yet, as we show in the supp. mat., LLM works well in the three people cases of fighting and holding hands by providing meaningful prompts. Furthermore, we believe the ability of LLM will be better in the future to handle more complex interactions. Our current experiments illustrate that GPT-4 works well in the two-people interaction contact plan generation. And it also works in some three-people interaction cases. Finally, LLM is a tool to scale up the contact plans in a batch. Yet, our framework does not necessarily need a LLM to produce reliable contact plans: contact plans could also be provided by professionals such as artists. Instead of manually designing keyframes, writing contact plans could greatly alleviate their efforts to generate interacted motions in their work flow.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J3NSKEBYCL", "reviewer": "Reviewer_BPk3", "summary": "This paper focuses on text-to-motion synthesis, specifically, to generate motion of multiple interacting people. The method can potentially work with an arbitrary number of people and models inter-person interactions as pairs of joints that can be either in contact or not, separated by a certain control distance. The method is composed of two main stages: a LLM planer and a diffusion model based on Human Motion Diffusion Model (MDM) that incorporates a motion controller, called Motion ControlNet, and an inverse kinematics (IK) module to have finer control over the joints of generated motion sequences and, thus, generate interactions between synthesized characters. The authors show that the distance between pairs of joints for interactions can be generated with off-the-shelf LLMs. A key feature of this work is that it is zero-shot, thus, the models used do not need to be trained with multi-person motion datasets. However, as it uses a motion controller (based on the ControlNet paradigm) which is fine-tuned to generate global pose sequences (global motion) as opposed to relative motion typically generated by SoTA methods such as MDM. This would be the first method to enable a single-person text-conditioned motion generation for generating interactions for multiple people. \n\nThe paper includes comparisons with the SoTA using standard metrics (FID, R-precision, Diversity, Foot Skating, Trajectory error, Location error and Average error). Authors compare with single-person motion generation methods (MDM, GMD, OmniControl). For the two-person setting the paper includes a comparison of Spatial Errors and a user study comparing with PriorMDM.", "questions": "Aside from the questions posted in the Weakness section, I have the following:\n\n### **Need for clarifications**\n- I don't fully understand how the model generates the motion for all people. Is it done first individually and then merged somehow? Does the method need to run one MDM+Motion ControlNet instance per person?\n- From the qualitative results, it seems to me that I see the effect of an optimization post-processing step after the generation? Do the authors use a post-processing step with the IK or is this the effect of the guidance by the IK module?\n- Is the IK module guidance applied both at the end of each denoising step and also after $x_0$ is obtained from the diffusion model? \n- To apply the IK module guidance, does the method use the gradient information from the optimization steps to influence or does it apply directly on the estimated poses in a similar fashion as in PhysDiff?\n\n\n### **Additional comments**\n\nI would suggest including a discussion on the Inter-X dataset at the first paragraph of the introduction. Even though this work is concurrent and this is not strictly necessary, this can make the paper stronger and more up-to-date. \n\n- L34: Could the authors expand a bit about which methods ignore a good design?\n- L38: Could the authors include a counter-example for such interactions that do not require 'additional interaction data'. Are these interactions closer interactions such as hugging or, the opposite, more subtle interactions such as talking or holding a meeting?\n- L108: I would suggest adding a comment about the method proposed by InterHuman which is also based on the MDM architecture.\n\n### Minor (typos and grammar)\n- L155: \"is not able\" should be “are” instead of “is”.\n- L160: is proposed--> proposes\n- L166: is “almost” a random noise to be sampled. Almost? Why almost? $x_T$ should be an isotropic Gaussian at T. \n- L187: desperately needs a comma, otherwise the sentence completely changes meaning!: \"(FK) to convert\" -->\"(FK), to convert\"", "limitations": "Yes, the authors include limitations.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* In part, the paper is clear and well-written.\n* The proposed InterControl framework is meaningful.\n* Although comparisons with multi-person motion generation models is not comprehensive mostly due to differences in training datasets used (which is understandable), the paper clearly shows an improvement compared to the priorMDM model, both quantitatively and qualitatively.", "weaknesses": "### **Method section needs a bit of re-writing**\nThe method section is somewhat confusing especially in the beginning. There are some definitions that the reader needs to make assumptions about in order to continue reading. For example:\n- There is no overview of the method in this section and even if Fig. 2 gives an overview, there is no link to it in the text. I would advise the authors to add this after the formulation and preliminaries so that the reader can easily understand the overall architecture. \n- L154: It is not so obvious to me what this discrepancy between relative and global motion is precisely and why it makes priorMDM less controllable in global space. I understand that generating global motion poses different challenges than relative motion, but could the authors explain this further? Doesn't it boil down to generating the global trajectory and then coupling/refining the global motion so that it is coherent with this trajectory as they do in [1]?\n- L157: While it may be considered as concurrent work, I suggest the authors discuss a bit [1] which also addresses this issue.\n- L174: At this point it is not clear to me what exactly are the spatial conditions. Authors could clarify with a simple example.\n- L175+: At this point, it is not clear why the control signal c is potentially sparse? Is it because it comes from text? This is hard to understand as there is no overview pointing out that the method uses a “Contact Plan” based on a LLM. \n- L177: I don't understand why the control will be desired for only a select few frames. Don't we want this control all the time? But again, authors may need to first clarify what exactly is this control signal and where does it come from.\n\n### **Additional baselines**\nI believe that authors did not include other multi-person motion generation baselines because those explicitly learn from multi-person data. However, I am curious to know how different the results from the InterHuman method differ from this approach? \n\nReferences:\n[1] Zhang, Siwei, et. al, RoHM: Robust Human Motion Reconstruction via Diffusion (CVPR 2024)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive review. We will revise our paper according to the insightful suggestions in our final version. We are happy that our paper is regarded as ‘clear’ and ‘well-written’ and our framework is ‘meaningful’. We have tried our best to clarify questions and we hope that our response could address your concerns.\n\n**Q1: Paper Writing.**\t\n\n**A1:** Thanks for your suggestions, we will add the reference to Fig.2 in the beginning of Sec.3 and give more detailed overview of our method. \n\n**Q2: Discrepancy between relative and global motion.**\n\n**A2:** Yes, the discrepancy between relative and global motion is similar to RoHM, where the root joint is represented in global space, while other joints' position are represented in a relative position by setting the origin to the root joint, i.e., relative (local) space. Such data format is adopted in HumanML3D and it is proved to be useful in motion generation by MDM, PriorMDM, etc. However, the global positions of joint in such data format is not available except for the root joint, which makes PriorMDM unable to control joints in the global space. We achieve this by converting such data format to global positions for the input of Motion ControlNet and IK guidance. In RoHM, they also use diffusion models to model root trajectories in global space and other joints' positions in the body-root space, which shares similar idea with motion generation methods in HumanML3D. \n\n**Q3: A concurrent work RoHM for pose estimation.**\n\n**A3:** Thanks for your kind suggestion. We will include this paper in the discussion in the final version. \n\n**Q4: Clarification of spatial condition.**\n\n**A4:** Spatial conditions are coordinate points in the global space, such as the origin (0,0,0). Our framework could accept multiple points together as the spatial condition, e.g., let the left wrist to (1,0,0) and right wrist to (2,0,0) in the same time. \n\n**Q5: Sparsity of spatial condition.**\n\n**A5:** A motion of single person with N frames will have (N, 22, 3) in global space in HumanML3D, where 22 is number of joints and 3 is xyz space. However, for example, we may only set 2 points as spatial condition according to the contact plan among total 22*N joint positions. We want to generate coherent motions with dim (N, 22, 3) by using spatial conditions such as 2 global points. In such cases, spatial conditions are sparse compared to the entire motion sequence. \n\n**Q6: The reason of control being desired for only a select few frames.**\n\n**A6:** For example, when we generate hand-shaking interactions, the contact plan outputted by LLM could only have contact pairs of one's right wrist and the other's right wrist for only 1s (e.g., 30 frames), and does not impose constraints over other frames. In other frames without control, we want such motions to be coherent to the frames being controlled. \n\n**Q7: Qualitative comparison with InterGen.**\n\n**A7:** As our method does not explicitly learn interaction data distributions, it is not suitable to evaluate metrics like FID and Top-3 precision. We shows some qualitative comparison with InterGen in Q1 of general response and Figure 1 of the attached PDF: Our method performs better in fine-grained distances in interactions, e.g.,  it is hard for InterGen to actually let two people holding hands (e.g., distance = 0). The hand-holding interaction in InterGen always leaves some small distances between hands.\n\n**Q8: How the model generates the motion for all people in the inference process.**\n\n**A8:** We generate all people in a batch, e.g., use a tensor of (B, N, D) to represent all people to be interacted, where B is the batch size (number of people), N is frame number, and D is the motion data dimension. The IK guidance is conducted between motions within a batch, leading to IK optimization back-propagated to each individual motion at the same time. Thus, our network is utilized once per denoising step for interaction generation.\n\n**Q9: Post-processing step**\t\n\n**A9:** We does not use post-processing step. Our IK module is applied at each denoising step in the diffusion process, not at the end of inference process. \n\n**Q10: Details of IK guidance.**\n\n**A10:** No. IK guidance is only applied at the end of each denosing step, like a classifier-guidance operation. \n\n**Q11: Optimization steps in IK guidance.**\n\n**A11:** We use L-BFGS in IK guidance to optimize global joint positions. Like PhysDiff, we don't use this gradient to train network parameters. Instead, the updated motion data feeds into the next denoising step. IK guidance addresses the discrepancy between relative joint data and the need for global position optimization in each step.\n\n**Q12: Inter-X dataset.**\n\n**A12:** Thanks for your suggestion. We will include this paper in our final version. \n\n**Q13: More clarifications.**\n\n**A13:** \n\nL34:\t\n\nPrevious methods could only generate interactions of fixed number of people. Thus, we think they are not 'good' general interaction modeling methods, as they are not flexible to generate group motion with arbitrary number of people. \n\nL38:\t\n\nThanks for this question. We think the counter-example is more like 'hugging', where close interactions are difficult to be described only by distance between joint pairs: it also require the mesh penetration control to avoid unnatural interactions with penetrations. On the contrary, for talking and holding a meeting, we could use a constant distance to constrain two people's root joints (e.g., 2 meters) and also constrain their head orientation to be face-to-face to make such interaction looks realistic. \n\nL108:\t\n\nThanks for the suggestion. We will include the discussion of InterGen in our final version. \n\nTypos:\n\nThanks for pointing out the typos. We will carefully revise our paper.  \n\nL166: Yes, x_T is a random gaussian noise; \n\nL187: Yes, we should add a comma after (FK).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank reviewers’ effort for our paper and the insightful review for us to improve our paper. We carefully read all reviews and address common concerns point by point here.

**Q1@VAgJ, HwRy and 7GL6**: Comparison with InterGen. 

**A1**: Firstly, our method is fundamentally different from InterGen. As a zero-shot multi-person motion generation method, InterControl does not learn data distributions from multi-person interaction datasets. This approach offers a significant advantage: the ability to generate interactions for groups of three or more people, as demonstrated in our qualitative results. In contrast, InterGen is inherently limited to generating interactions between only two people. Furthermore, InterControl exhibits greater sensitivity to specific distances, accurately representing scenarios such as physical contact (distance = 0) or maintaining a particular separation (e.g., distance ≥2m). InterGen, however, often struggles to ensure desired distances, particularly in close-contact scenarios like hand-holding. A qualitative comparison between InterControl and InterGen can be found in Figure 1 of the accompanying PDF.

**Q2@VAgJ and HwRy**: Comparison with OmniControl.

**A2**: Firstly, OmniControl does not propose the framework of zero-shot interaction generation, thus our focus is fundamentally different from it. We were unaware of OmniControl when we chose controllable single-person motion generation to achieve the goal of zero-shot interaction generation. The detailed comparison with OmniControl can be found in Sec. A.4 in the Appendix. Our major advantage over OmniControl in single-person motion control is the superior performance on non-root joint control. Secondly, we have included a quantitative comparison with OmniControl on single-person motion evaluation in Table 1 of our paper. To address the concern of HwRy, we also include a quantitative comparison with OmniControl on zero-shot interaction generation using our collected contact plans in the following Table. OmniControl exhibits poorer joint control performance in the contact joint pairs. Thirdly, qualitative comparison with OmniControl can be found in Figure 2 of the PDF. It also demonstrates cases where the joint alignment in contact joints (e.g., distance = 0) is not as precise as ours.

| Method | Traj. err. (20 cm) ↓ | Loc. err. (20 cm) ↓ | Avg. err. (m) ↓ |
| --- | --- | --- | --- |
| PriorMDM [51] | 0.6931 | 0.3487 | 0.6723 |
| OmniControl [65] | 0.0322 | 0.0029 | 0.0194 |
| Ours | 0.0082 | 0.0005 | 0.0084 |


**Q3@VAgJ, HwRy and 7GL6**: Interpenetration (body penetration, human collision) issues.

**A3**: As our method is a kinematics-based motion generation approach, it primarily focuses on semantic alignment with language rather than local penetration or artifacts, which is consistent with MDM and other kinematics-based methods. The main evaluation metrics for single-person motion generation are FID or Top-3 classification precision, which measure the ability to learn data distribution. Consequently, local artifacts are not the primary concern of kinematics-based motion generation methods, and eliminating such artifacts is challenging for these approaches. For instance, InterGen also exhibits body penetrations (e.g., "Two people embrace each other" in https://tr3e.github.io/intergen-page/). In a fair comparison with other zero-shot methods, e.g., PriorMDM, we demonstrate superior results in body penetration avoidance. Our IK guidance ensures that any two characters' torso joints maintain a minimum distance of one meter. PriorMDM displays significant torso collision in the qualitative results (refer to Figure 3 in our paper ), while our method only shows minor collisions of limbs.

Moreover, if we need to eliminate penetration issues in specific applications, physical animation in a simulator presented in our application section (Lines 334-342 and Figure 1(c)) will be an effective solution. This is effective because the simulator inherently prevents collisions. Our qualitative results on physical animation demonstrate that training-free imitation learning methods [40] execute our generated interactions effectively, a point acknowledged by Reviewer#HwRy.

**Q4@HwRy and 7GL6**: Unsmooth motion (abrupt velocity changes) in qualitative results.

**A4**: Thank you for the detailed review. We set the FPS to 15 in the visualization video for qualitative results, while the common practice is 30. It may misleadingly suggest that our generated motions are not smooth enough. Additionally, some cases in the visualization videos use more than 5 contact pairs and short intervals between contact and separation, potentially leading to higher velocities in these intervals. As explained in our paper, IK guidance can adaptively modify velocities from the start to frame n. Generated motion will be smoother if the frame number n is larger. However, some cases in the visualization use a very small n, which could be easily misinterpreted. We sincerely appreciate the insightful review and will carefully revise it in our final version.

Furthermore, we have observed that many motion generation methods (e.g., InterGen) use a 1D Gaussian filter to smooth the generated motion without affecting the semantics. In the following Table, we present quantitative results of acceleration in generated motions where both methods adopt the same post-processing. Our acceleration is similar to InterGen's when utilizing the standard 1D Gaussian filter as post-processing, demonstrating that our velocity changes are comparable to existing methods'. Additionally, we provide a sequence of acceleration in two people fighting to qualitatively compare the acceleration between our generated interaction and InterGen's. This figure can be found in Figure 3 of the attached PDF. It further illustrates that our acceleration is comparable to InterGen's.

| Method | Acceleration (m/frame^2) |
| --- | --- |
| InterGen [36] | 0.0046 |
| Ours | 0.0042 |

Meta-review: Post-rebuttal, reviews were split across borderline, with one in favor of acceptance, two borderline accept, and one borderline reject. In post-rebuttal discussions, the borderline reject reviewer commented that they would be in favor of acceptance if the other reviewers were too (which they are), leading to all accept recommendations. The AC examined the paper, the reviews, the rebuttal and the discussion and is in favor of accepting the paper: this work will be of interest to the community. While there are some lingering concerns that might be fixed in to-be-seen revision, none are show-stoppers

Since the rebuttal and ensuing discussion was particularly important for the acceptance of the paper, AC strongly urges the authors to incorporate the rebuttal and feedback from the discussion into the final version of the paper. In particular, the teaser should be changed to remove images (which may give a misleading impression), and the final version of the paper should use the extra page to also have a discussion about mesh-based representations and body surfaces. While there is no mechanism to review the paper, the AC would remind the authors that other readers may have similar reactions, and so incorporating the results will also maximize the impact of the paper.
Decision: Accept (poster)