Ready-to-React: Online Reaction Policy for Two-Character Interaction Generation
Authors: Zhi Cen
Huaijin Pi
Sida Peng
Qing Shuai
Yujun Shen
Hujun Bao
Xiaowei Zhou
Ruizhen Hu
Venue: ICLR 2025 Poster (2025)
Keywords: Human Interaction Generation
Reactive Motion Generation
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: This paper addresses the task of generating two-character online interactions. Previously, two main settings existed for two-character interaction generation: (1) generating one's motions based on the counterpart's complete motion sequence, and (2) jointly generating two-character motions based on specific conditions. We argue that these settings fail to model the process of real-life two-character interactions, where humans will react to their counterparts in real time and act as independent individuals. In contrast, we propose an online reaction policy, called Ready-to-React, to generate the next character pose based on past observed motions. Each character has its own reaction policy as its ``brain'', enabling them to interact like real humans in a streaming manner. Our policy is implemented by incorporating a diffusion head into an auto-regressive model, which can dynamically respond to the counterpart's motions while effectively mitigating the error accumulation throughout the generation process. We conduct comprehensive experiments using the challenging boxing task. Experimental results demonstrate that our method outperforms existing baselines and can generate extended motion sequences. Additionally, we show that our approach can be controlled by sparse signals, making it well-suited for VR and other online interactive environments. Code and data will be made publicly available.

Published as a conference paper at ICLR 2025
READY-TO-REACT: ONLINE REACTION POLICY FOR TWO-CHARACTER INTERACTION GENERATION
Zhi Cen', Huaijin Sida Peng', Qing Shuai', Yujun Shen", Hujun Bao', Xiaowei Zhou's
Ruizhen Hu
State Key Lab of CAD&CG, Zhejiang University, "The University of Hong Kong, 3 Ant Group,
Shennzenn University
zhicenezji edu. en, suizhen. wwwgmail.cc..m
Ours
per-trans
per-clip FID: 25.283
Time
1800 Frames Minute)
GPT-based method per-trans per-olip FID: 122.463
Error Accumulation
Frames
Wrong orientation
Out
Get
online reaction policy for two-character interac- tion generation on the challenging task of boxing. Ready-to-Resd predicts the next pose of an agent by considering its own and the counterpart's 's historical motions. Our method can successfully gen- erate 1800 frames of motion, whereas the GPT-based approach struggles after about 200 frames, displaying issues such as incorrect orientation, leaving the ring boundary, or freezing in place due to the accumulation of errors over time.
Figure 1: Demonstration of Ready-to-React
ABSTRACT
This paper addresses the task of generating two-character online interactions. Pre- viously, two main settings existed for two-character interaction generation: (I) generating one's motions based on the counterpart's complete motion sequence, and (2) jointly generating two-character motions based on specific conditions. We argue that these settings fail to model the process of real-life two-character interactions, where humans will react to their counterparts in real time and act independent individuals. In contrast,
propose online reaction policy, called Ready-to-React, to generate the next character pose based on past ob- served motions. Each character has its reaction policy its "brain", abling them to interact like real humans in a streaming manner. Our policy is implemented by incorporating diffusion head into auto-regressive model, which can dynamically respond to the counterpart's motions while effectively mitigating the error accumulation throughout the generation process. We con- duct comprehensive experiments using the challenging boxing task. Experimen- results demonstrate that our method outperforms existing baselines and can generate extended motion sequences. Additionally, we show that our approach can be controlled by sparse signals, making well-suited for VR and other on- line interactive environments. Code and data will be made publicly available at https: / / ju3vv. github.
INTRODUCTION
This paper aims to learn reaction policy from data that can generate two-character interactions a streaming manner. Such a policy is essential for applications robotics, gaming, and virtual
"oorreponniigg author.

---

Published as a conference paper at ICLR 2025
reality, where the character needs to interact with other entities in real-time. Generating reasonable online reactions is quite challenging, considering two key perspectives. Firstly, the policy should dynamically adjust its own actions based on the counterpart's responses at each time step, which is essential for enabling online applications. Secondly, downstream applications require the policy to generate natural and physically plausible motions while maintaining consistency and diversity throughout these sequences.
Previously, there were two main settings to generate two-character interaction: (1) generating one's motions based on complete sequence of the counterpart's motion (Siyao et al., 2024; Ghosh et al., 2024: Xu al., 2024b), and (2) jointly generating two-character motion sequences based on specific conditions, such as textual input (Tanaka & Fujiwara, 2023; Gu et al., 2024; Ruiz-Ponee et al., 2024). However, we argue that both settings do not model the process of real-life two-person interactions. As humans, we dynamically produce the reaction based on the counterpart's actions at each time step, engaging in independent mind rather than sharing a collective consciousness. To achieve two- character online interaction, it is crucial to develop a reaction policy that can be separately applied to two characters, allowing them to react to each other like real humans.
In this paper, we propose novel reaction policy, called Ready-to-React, for generating two- character interactions, as illustrated in Figure 2. Our core innovation lies in incorporating a diffusion head into auto-regressive model, which respond to the counterpart's motions in a streaming manner while ensuring the naturalness and diversity of the motions. Specifically, we first encode the observed motions into latent vectors using motion encoder. Given the history motion latent of the character and the counterpart's motion, our reaction policy uses an auto-regressive model to pre- dict a conditioning feature vector. This vector guides diffusion model generate the next motion latent, which is then decoded into the next character pose. As illustrated in Figure I, the proposed reaction policy can generate long and natural boxing sequences compared the baseline method.
We chose boxing as the task for online two-character interaction generation considering its fast pace and frequent shifts between offense and defense. These dynamic interactions make an ideal scenario for testing and validating our approach. We conducted experiments to validate the effec- tiveness of our approach on our self-collected boxing dataset DuoBox. Our method was evaluated under three settings: (I) reactive motion generation, (2) two-character interaction generation, and (3) long-term two-character interaction generation, where it outperforms the baselines. We show that method can generate very long motion sequences (~ minute), just relying the initial poses of the two characters. Additionally, we carried out experiments on sparse control motion generation, demonstrating that method is well-suited for VR online interactive settings.
RELATED WORK
Single-character motion generation. In recent years, deep learning methods for motion synthe- sis have gained increasing attention (Fragkiadaki et al., 2015; Holden et al., 2016; Martinez et al., 2017: Dou al., 2023: Xie al., 2023; Pi et al., 2023; Cen al., 2024). Various techniques, in- cluding multi-layer perceptron (MLP) (Holden et al., 2017), mixture of experts (MoE) (Zhang et al., 2018), and recurrent neural networks (RNN) (Harvey al., 2020) are employed to tackle this task. Additionally, to generate diverse and natural results, generative models such as conditional varia- tional auto-encoders (cVAE) (Ling et al., 2020), generative adversarial networks (GAN) (Li et al., 2022), and normalizing flows (Henter et al., 2020) have shown promise in addressing this challenge. Moreover, the success of generative pre- trained transformers (GPT) (Zhang et al., 2023) and dif- fusion models (Tevet et al., 2023) further underscores their potential in this area. Recently, Chen et al. (2024); Shi et al. (2024) adopt auto-regressive diffusion models to generate single-character motions. Although our reaction policy generates character's motion based the other's, ss tech- nically making it single-character motion generation, our work focuses on decision independence in two-character interactions.
Reactive motion generation. Reactive motion generation is subfield of human motion gener- ation that focuses on generating human motion in response to external agents. Men et al. (2022) introduces semi-supervised GAN system with part-based LSTM module to model temporal sig- nificance. Chopin et al. (2023) employs a transformer network, enhanced by an interaction distance module using graphs. Siyao al. (2024) proposes GPT-based model to predict discrete motion
2

---

Published as a conference paper at ICLR 2025
History Encoder
Encoder
causal mask
What my nest move?
observation
Transformer Encoder
MEP
Bext pose
MLP
Z
Transformer Encoder
Reaction Policy P
Diffusion
2
Next Latent Predictor
Aug
Ricis-4.71
Online Motion Decoder
Figure 2: Pipeline overview. Given boxing scene at the leftmost figure, where the blue agent is thinking about its next move. The reaction policy (Section 3.2) follows these steps: first, based on the observations, the history encoder encodes the current state and observations; then, the next latent predictor predicts the upcoming motion latent; and finally, an online motion decoder decodes this motion latent into the actual next pose. The same reaction policy can be applied to the pink agent. Through a streaming process for both agents, our reaction policy enables the continuous generation of two-character motion sequences without length limit (Section 3.3).
tokens, enhanced by an off-policy reinforcement learning strategy. Xu et al. (2024b) utilizes a diffusion-based model with an explicit distance-based interaction loss. Ghosh al. (2024) employs a diffusion-basee model with a combined spatio-temporal cross-attention mechanism. However, Xu al. (2024b) and Ghosh et al. (2024) require the complete motion sequence of the other agent, and cannot generate reactive motions online. In contrast, our method enables online interaction generation by leveraging auto-regressive diffusion models, as inspired by Li et al. (2024).
Two-character motion generation. While multi-character motion generation focuses producing motion for groups with social relationships (Zhu et al., 2023b; Peng et al., 2023; Le et al., 2023; Lim et al., 2023; Jeong et al., 2024), two-character motion generation focuses generating closer in- teraction between two characters. Starke et al. (2020) proposes the local motion phase for complex, contact-rich interactions. Starke al. (2021) combines motion generator with task-dependent control modules. Both Starke et al. (2020) and Starke et al. (2021) require user control signals. Physically-baser methods (Won al., 2021; Zhu al., 2023a; Younes al., 2023) ensure the phys- ical plausibility of character interactions but still struggle to generate natural and diverse motions. Liu et (2019); Fieraru al. (2020); Yin et al. (2023); Liang et al. (2024); Xu et al. (2024a) pro- vide datasets with rich annotations that enable advancements in interaction modeling two-character scenarios. Recent advances text-driven two-character motion generation have introduced diffu- sion models (Tevet et al., 2023) to enhance realism and control (Tanaka & Fujiwara, 2023; Gu et al., 2024; Ruiz-Ponce et al., 2024; Cai et al., 2024). Despite these advancements, these methods jointly generate whole sequences for both characters and ignore the dynamic feedback inherent in real-life two-person interactions.
3
METHOD
Our aim is develop intelligent reaction policy capable of generating online motions that dynam- ically respond to counterpart's movements. We begin by outlining the formulation of the one-step reaction in Section 3.1. Then, introduce the design of the reaction policy in Section 3.2. We
3

---

Published as a conference paper at ICLR 2025
plain how this policy drives the generation of two-character motions in Section 3.3. In Section 3.4, we explain the training loss and implementation details.
3.1 PROBLEM FORMULATION
We first introduce the problem formulation. There are characters, the agent A and the oppo- nent O. The goal is to generate the future agent's 's motion A, based on the opponent's movements Ozelf-wa W.f) and the agent's previous motion A.,))-
A = P(O - -w Arelswwnn).
where the generative reaction policy, and W is the visible past window size. We use the same root coordinate definition in Starke et al. (2020; 2021), as shown in Figure 2. Each agent's motion A; is defined as:
A, [rom R², rair ?R, Ops Rx33 O Rxx6,
O R2
where r.f and rair are the horizontal trajectory (excluding the y-axis) positions and directions relative to the (i (1-1)-a 1 frame agent's 's root coordinate, respectively. The O O and O are the positions, 6D rotations and velocities relative to the i-th frame agent's root coordinate, respectively. is the number of body joints. Each opponent's motion O; is defined as:
O, =(//-
pos E ERYS
θ R 6
ER)
(3)
which includes the positions, 61) rotations and velocities relative to the i-th frame agent's (A's) root coordinate, respectively.
3.2 REACTION POLICY
In this section, present reaction policy, which predicts motion in the latent space. As illus- trated in Figure 2, the history encoder module is responsible for encoding the observations of the current state. Based on this historical information and the opponent's 's motion, the next latent predic- tor module predicts the future motion latent codes. Finally, the online motion decoder generates the future motions using the predicted latent and the current agent's 's state.
Latent motion representation. Converting data to latent space has become popular approach in generation pipelines (Rombach et al., 2022), and VQ- has been proven to be effective in learning disentangled representations of human motion data (Zhang al., 2023; Jiang et al., 2024; Lee et al., 2024; Starke et al., 2024). In this paper, we adopt a similar architecture as in Zhang et al. (2023) to encode the raw motion data into latent sequences. Given a sequence of agent motion (A, i [0, f), E Z). the VQ-VAE motion encoder encodes it to Z {Z, i E [0, LLJ), E Z} with downsampling factor d = 4, and each latent code Z; is quantized through the codebook C to find the most similar element:
Ż, = arg min || I2 - Cx ELEC
(4)
History encoder. First, we need to encode the past information of both the agent and the opponent. To represent the agent's past motion A, we could directly utilize the VQ- motion encoder to
compress them into latent variables 2. The opponent's 's historical motion is also downsampled by factor of d = 4 and then passed through single-layer MLP, which encodes it into feature vector with the dimension of Ż, ensuring consistency for subsequent processing.
Next latent predictor. We then predict the next motion latent in auto-regressive manner based on the historical information. Our approach employs a transformer-basee condition encoder to fectively capture all visible information. This encoded data is then passed to the diffusion- based motion latent predictor, which generates the next motion latent based on the encoded conditions.
Specifically, we begin by constructing transformer-basee encoder to encode the information acces- sible to the reaction policy. As illustrated in Figure 2, the transformer's input consists of the motion latent code and the opponent's 's feature obtained from the history encoder. As a result, each output of the transformer-haste encoder P, encapsulates the information preceding the f-th frame.

---

Published as a conference paper at ICLR 2025
Next, we introduce the diffusion process, which predicts the future motion latent codes Z, based on P at the f-th frame. We use conditional diffusion models (Tevet et al., 2023; Ramesh et al., 2022) and P, serves the conditioning input, shown in Figure 2. We employ single-layer MLP the generative model g, ensuring that our model can operate in real-time. The predicted motion latent code Zy is then used to generate the future motion through the online motion decoder.
Compared to previous methods that rely predicting motion tokens' probabilities and supervising GPT models with cross-entropy loss, our approach of predicting motion latent using a diffusion- based model preserves the smooth and continuous nature of motion. This results in fewer cumulative errors and less deviation from the intended motion over time, offering greater stability and accuracy than predicting token probabilities with GPT models.
Online motion decoder. remaining problem to decode the predicted motion latent code 2, into the future agent motion A online. We propose an online motion decoder that takes a few previous frames and two consecutive motion latent codes to generate the next motion frames. As shown in Figure 2, we use transformer as the online motion decoder. The inputs to the online motion decoder are the past agent motions Aie(s-d.f)- root information ( the last motion latent code 7 and current motion latent code Z from the next latent predictor. Here, each root information R2 is defined as:
(5) which includes the agent's horizontal trajectory (excluding the y-axis) positions and directions rel- ative to the opponent's s root coordinate, and the distance between the agent's ' root and the center of the boxing ring. The output of the online motion decoder are the future agent motions Aself.+t)) and the future root information Rig(s,/d)) In contrast to VQ-VAE decoder (Zhang et al., 2023), which requires the entire sequence tokens before decoding, our method decodes motion latent into explicit motion sequences in real-time using only few tokens and historical data, enabling online generation.
R (rof E R22 dir E RR² rais ER))
3.3 ONLINE TWO-CHARACTE& MOTION GENERATION
Our reaction policy, as described in Section 3.2, enables the generation of the next motion frame by leveraging both the opponent's past motion and the agent's past motion. To implement online two-character interaction generation, we use the same reaction policy P for the two characters.
Starting with an initial input of = 4 frames of poses, both characters use the policy P to gen- crate the next d frames by considering their own initial motion and the opponent's 'ss motion. These generated motions are then added to history buffer H. with maximum size of W. After this initial phase, both characters continuously update their predictions by incorporating the newly gen- erated frames and the accumulated motion history. The interaction generation process operates in a streaming fashion. Motion that exceeds the buffer size W is discarded as outdated information. Through this approach, both characters dynamically respond to the other, ensuring coherent and natural interactions while enabling two-character motion generation without a length limit.
3.4 TRAINING LOSS AND IMPLEMENTATION DETAILS
The training process is divided into two stages: (1) pre-training the VQ-V model and (2) jointly training the next latent predictor model and the online motion decoder. Details about the network architecture are provided in Appendix C.
Stage 1. We pre-train the VQ-VAE model following the approach in Zhang et (2023); Razavi et al. (2019) for 40k iterations, using motion sequences cropped to 64 frames. The batch size is set to 128, with a codebook size = 512, codebook feature dimension = 512, and a downsampling rate of d 4. The codebook is updated with the exponential moving average (EMA) method (Razavi et al., 2019), as a replacement for the codebook loss. Finally, the loss is defined as:
Liquae Cree + a |llls|/|/2
[2] Z||
where Cree is the L2 reconstruction loss, || 2 refers to stop-gradient operation, and a 0.1.
Z 2 is the commitment loss, the operator sg
5

---

Published as a conference paper at ICLR 2025
Stage 2. Next, we train the next latent predictor and online motion decoder jointly for 40k itera- tions while keeping the VQ-VAE motion encoder fixed. During training, we applied causal masks (Figure 2) to ensure that the model can only access the current and previous inputs, preventing in- formation from leaking into future time steps. We use ground truth y instead of the predicted - in Figure 2 during training. For this phase, motion sequences are cropped to W 60 frames (2 seconds) for training. The batch size is set to 32, with time step T 1000, and we employ DDIM (Song et al., 2020) to sample only 50 steps during inference. The loss is defined as:
L =
Ldiffusion + B| A
+
7 R R12,
where B = 1.0, 7 = 1.0. Luiiission is defined as:
Laiiiusionn Feg[1.T]xx [| ((x,, c)]12),
where Xo ' is the ground truth next latent, 9 is the generative model, c = Py is the condition, and (x,, L, c) 2 is the predicted next latent, All models are trained using the Adam optimizer (Kingma & Ba, 2014) with learning rate of 0.0001 on single Nvidia RTX 4090 GPU.
4 EXPERIMENTS
4.1 DATASET, EXPERIMENTAL SETTING, AND EVALUATION METRICS
Dataset. To evaluate our method, we collect a high-quality dataset, DuoBox, using the OptiTrack Mocap system' equipped with 12 cameras. We invited three boxing enthusiasts to perform various boxing movements, recording multiple sequences of their actions. Please refer Appendix A for the details of the data collection. In total, DuoBox consists of 63.4 minutes of motion data (approx- imately 457k frames) captured 120 FPS. For our experiments, we split the dataset into training (80%) and testing (20%) subsets, and downsample the original data to 30 FPS for training purposes. To further enrich the dataset, we apply the augmentation by swapping the roles of the agent and the opponent during training.
Experimental setting. We evaluate our method in three scenarios: reactive motion generation, two- character interaction generation, and long-ter fong-term two-character interaction generation. In the reactive motion generation, we use the ground truth for the opponent's 's motion as input. For the two-character interaction generation, we provide only the initial 4 frames of poses for both characters. In both test scenarios, the motion of individual characters follows the procedure outlined in Section 3.2, while the generation of two-character motions adheres the process described in Section 3.3.
Evaluation metries. We evaluate the generated motion sequences using the following metries: (1) Frechet Inception Distance (FID). We follow Dou et al. (2023) calculating per-frama per- transition and per-clip FIDs. A lower FID indicates the generated motion is more similar to the real data. (2) Jitter. We evaluate motion jittery following Shen et al. (2024). A closer value to the ground truth indicates better motion quality. (3) Root Orient (RO). To assess the long-term consistency of the generated motion, we propose a new metric RO. It calculates the percentage of frames where the facing direction between the two agents exceeds 45 degrees. This is motivated by the nature of boxing, where the athletes typically face to each other throughout the match. Lower deviation from the ground truth means the generated motion aligns better with the expected interactive behavior. (4) Foot Sliding (FS). Foot sliding (Ling et al., 2020) measures the average sliding distance when the foot close to the ground (< 5em). A value closer to the ground truth indicates better motion quality. Minimal foot slide may suggest that the generated motion remains stationary. In addition, we provide the inference speed in Appendix E and motion diversity analysis in Appendix F.
4.2 COMPARISON WITH BASELINES
We compare our method against several baselines, including InterFormer (Chopin al., 2023), CVAE-AR, CAMDM (Chen et al., 2024), T2MGPT-online (Zhang et al., 2023), and Duolando- offline (Siyao et al., 2024). Among these, InterFormer is deterministic model and lacks the ability to generate diverse results. CVAE-AR is CVAE-based approach that we construct specifically for
hhttps://optitraat

---

Published as a conference paper at ICLR 2025
Table I: Comparison with baselines. We compare our method with five baselines (Section 4.2) in the two scenarios: reactive motion generation and two-character motion generation. Among them, bold indicates the best results, means lower is better. means closer to the real data is better.
Reactive
Two- churacter
Methods
FID
Jitter
RO
FS
HDD
Jinerr
RO
FS
Por-framc
Per-trans.
Per-elip
Per-frame
Por-trnss
Per-clip
Real
21.331
24.7%
0.97
21.332
24.7%
0.97
InterFormer
0.724
1.993
15.061
6.712
40.3%
1.08
2.498
4.590
47.194
6.117
31.5%
1.05
CVAE-AR
4.233
46.2%
1.13
5.405
1.99
CAMDM
1.606
4.037
28.503
53.622
40.1%
1.96
4.162
7.483
70.416
53.994
22.6%
2.01
T2MGP
1,721
1.567
30. 159
78. 292
50.4%
2,65
8.249
3.347
122 413.463
78.334
46.2%
2,63
Duolando-oflline
5.862
Ours
0.535
0.995
9.599
17.825
34.7%
1.02
1.394
2,105
25,283
16,844
24.1%
0.97
Table 2: Quantitative results of long-term two-character motion generation. We compare our method with four baselines (Section 4.2). The generated motion lengths are set to 1800 frames.
FID
Methods
Jitter
Per-frame
Per-transition
Per-clip
Real
21.332
24.7%
0.97
InterFormer
5.628
5,936
87.697
4.784
44.6%
0.90
CVAE-AR
7.325
11.717
110.458
36.7%
0.75
CAMDM
7.557
13.465
109.654
25.237
13.7%
0.86
T2MGPT-online
21.27
4.457
273.086
72.481
70.4%
2.34
Ours
2.388
2.375
36.755
19,204
31.3%
0.83
comparison. For CAMDM, we modify its input to make it function as a reaction policy, T2MGPT- online refers decoding each newly generated token immediately into raw motion using a VQ- VQ-VAE (Siyao et al., 2024) decodes the tokens
decoder as soon as the token is produced. Duolando-offlin after the entire sequence has been generated. We retrain all these methods on DuoBox using similar training configurations for fair comparison. Details can be found in Appendix D.
Reactive motion generation. We begin by evaluating our method in the context of generating reactive motion, where the opponent's motion provided as ground truth. The results, presented in Table 1 (left: reactive), show that our method significantly outperforms the baseline across multiple metries, including per-frame, per-transition, and per-clip FID scores, as well as reducing foot slid- ing. Additionally, the RO of our method closely matches the performance of the offline Duolando generation. Notably, the root prediction Duolando is relative to the opponent, preventing drift over time. Among the online prediction methods, InterFormer may generate motions with implau- sible root position and orientation. T2MGPT, which decodes GPT-predicted tokens online, exhibits jätter and discontinuity. Both CVAE-AR and CAMDM produce motion that is easy get stuck over time. We also provide qualitative results in Figure and the supplementary materials.
Two-character interaction generation. Our method enables the simultancous generation of mo- tion for both agents. Starting with the first four frames, each agent's subsequent motion is generated by leveraging the interaction between their own and their opponent's 's past motions. In contrast, Duolando-offline cannot generate both agents' motions simultaneously. as it predicts the root posi- tion relative to the opponent's 's at the same frame and requires future information through a looking- ahead mechanism. Moreover, it relies on having all tokens available before decoding, preventing online generation. As shown in Table (right: two-character), our method significantly outperforms the baseline across all metrics in this more complex scenario. We also provide qualitative results in Figure and the supplementary materials.
Long-term two-character interaction generation. We demonstrate our method's ability to gen- erate extended sequences of two-character motion, highlighting its reduced error accumulation and superior motion quality over long durations. Our method is capable motion for 1800 frames. The results are shown in Table 2. The FID of ours is much better than all the baselines. To show the effect of error accumulation, we plot the face direction relative to time in Figure 3. T2MGPT-online and InterFormer will quickly generate motions that face away from each other after some time, and CAMDM generates motions that are ovor-smoohhed. Please refer to the supplementary material for more visualizations.
generating two-character
7

---

Published as a conference paper at ICLR 2025
Nwww
150
Naturar
Resetive Motion Generation
GT
Ours
Baselincs
Bro-charmcter Mistion Genzration
Visualization of the face direction relative to time. We compare our method with baselines in two scenarios described in Section 4.2. The x-axis represents the frame number while the y-axis shows the angle between the facing directions of the two characters (in degrees). An angle of O indicates that the two agents are facing each other, whereas 180 means they are facing away from each other. The green lines represent the ground truth, the blue lines represent method, and the red lines represent the baselines.
Figure 3:
Table 3: Ablation study. We compare our method with five variants to validate main design choices (please refer to Section 4.3 for details). Among them, bold indicates the best results. means lower is better. >> means closer to the real data is better.
Reactive
Two-churacter
Methods
FID
FID
RO-
FS
RO
FS -
Per-frame
Per-trans.
Per-clip
Per-frame
Per-trans.
Per-clip
Real
24.7%
0.97
24.7%
0.97
use VAF enender
01525
1,290
10.423
36.5%
1.00
1.446
3,002
28.268
22.2%
1.01
w/o motion encoder
0.693
1.339
24.860
54.0%
1.48
7.203
3,012
126.991
52.3%
1.47
use GPT
0.892
1.781
16.212
37.4%
0.99
2,418
3,536
41.144
20.9%
0.97
w/o online decoder
5.215
11.020
78.933
44.9%
1.06
11.279
23.189
157.253
16.0%
0.91
w/o in decoder
0.496
1.074
10.315
43.3%
0.93
3,370
2.275
54.503
38.2%
0.92
0.535
0.995
9.5998
34.7%
1.02
1.394
2.105
25.283
24.1%
0.97
4.3 ABLATION STUDY
As shown in Table 3, we compare method with five main ablated versions: (1) use VAE as encoder. To demonstrate the stability of our method with different motion latent encodings, we replace the VQ-VAE with VAE as the motion encoder. As shown in the table. using VAE the motion encoder does not significantly affect the results. (2) w/o motion encoder. To highlight the importance of predicting motion latent codes rather than motions, motion encoder and directly predicted the pose sequence. The results show that directly predicting the raw pose sequence significantly degrades the motion quality. (3) use GPT. We replace the diffusion model with GPT to predict the next token probabilities. The results show decline in motion quality, with noticeably worse FID scores. (4) w/o online decoder. To validate the necessity of training new online motion decoder, we directly apply the VQ-VAE decoder to decode the motion latent codes into motion sequences. Using the VQ-VAE decoder at each step results in discontinuous motion, which in turn leads to worse FID score. (5) w/o R. in decoder. We remove the root sequence R Equation 5 as the input the online motion decoder. Without R, the model can easily predict motions with the wrong root facing direction.
remove the VQ-VAE
In summary, we demonstrate the importance of different components our model. More ablation studies and visual results can be found in Appendix G and the supplementary materials.
5 APPLICATION: GENERATING REACTIVE MOTION WITH SPARSE SIGNALS
Introducing sparse control is essential for making method practical in real-world applications, particularly VR online interactive environments. In these settings, capturing detailed and dense input data can be challenging due to hardware limitations, computational costs, comfort. Sparse control addresses this by allowing the system to generate high-quality motion based on min- imal input signals.
8

---

Published as a conference paper at ICLR 2025
Table 4: Quantitative results of generating reactive motion from sparse signals. We compare our method with CAMDM. Among them, bold indicates the best results. + means lower better. means closer to the real data is better. Our method outperforms the baseline in terms of all metrics.
FID
Methods
Jitter
FS
Pos. Err. J
Rot. Err.
Per-frame
Per-tramsition
Per-clip
Real
21.332
0.97
CAMDM
0.697 0.249
1,506 0.263
15.169
47.229
2.25
14.52
22.40
Ours
4.086
21.163
1.06
2.72
4.39
OURS
+ f=16
1:332
+ 1-48
1:60
f-96
CAMDA
I-16
T-32
F-48
1:60
f=80
F-96
Figure 4: Qualitative results of generating reactive motions from sparse signals. We compare our method with CAMDM. Our approach successfully generates realistic motion while effectively adhering to the sparse signals (anmotated by red dots in the figures). contrast, CAMDM struggles to achieve the same level of responsiveness and accuracy, shown in the red circles.
To demonstrate that our method well-suited for VR online interactive environments, we also con- ducted experiments showing that it can be controlled by sparse signals. The sparse signals the head and two-hand positions and rotations relative to the previous frame's 's agent root coordinate. To enable the controlling feature, we retrain the reaction policy by adding the sparse signals as conditions to the two transformer models in Figure 2. The loss and other training settings remain unchanged. We evaluate the quality of the generated motion using FID scores, motion jitter, foot sliding, and position and rotation errors to highlight the controllability of our approach. We compare our method with CAMDM (Chen et al., 2024), an auto-regressive method that generates diverse mo- tions based on control signals. The results, presented in Table 4, show that our method consistently outperforms the baseline across all evaluated metries. We also provide qualitative results Figure and in the supplementary materials.
6 DISCUSSION AND CONCLUSION
In this work, we have introduced a novel reaction policy that can be applied to two-character boxing interaction generation. The reaction policy includes diffusion-based predictor for forecasting the next motion latent, paired with an online motion decoder optimized for the online generation. Our method effectively reduces error accumulation, enables real-time online generation, and can produce extended motion sequences. Additionally, it offers controllability through sparse signals, making it well-suited for online interactive environments.
However, several limitations method is primarily designed for interactions between two individuals, and extending it to handle multi-person scenarios remains challenge. Second, with the environment or objects, which are essential for many real-world applications. Overcoming these limitations will be crucial for making the method more versatile and applicable to wider range of scenarios.
approach should be addressed in future work. First, the current
method does not account for interactions
ACKNOWLEDGEMENTS
This work was partially supported by the NSFC (No. 62322207, No. 62172364, No. 62402427), Zhejiang Provincial Natural Science Foundation of China (No. LD25F030001), Ant Group Research Fund and Information Technology Center and State Key Lab of CAD&CG, Zhejiang University.

---

Published as a conference paper at ICLR 2025
the two characters too close, leading to penetration
InterFormet
P-35
1+50
f-85
F-Q0
f+110
the blue character is inactive and getting stuck
CAMDM
P-35
1+50
f+85
f-00
1+110
orientation
Doolando
OURS
f+35
1+50
f+80
Figure 5: Qualitative results of generating reactive motions. Given the same ground truth op- ponent motion, InterFormer produce reactive motion that is too close to the opponent, leading to penetration. CAMDM tends to get stuck, while Duolando may result in human motion with incorrect orientation after certain period.
wrong
Interoormee
f-50
f+50
f-110
[+140
f+170
f+200
stuck
CAMDM
6-50
1+80
f-110
[+140
f-170
1+200
far from other
T24GPT
OURS
Figure 6: Qualitative results of generating two-character motions. Given the same initial four frames for both characters. InterFormer tends produce human motion with incorrect orientation. CAMDM often results in the characters getting stuck, while T2MGPT can cause the two characters drift apart due to accumulated errors.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Zhongang Cai, Jianping Jiang, Zhongfei Qing, Xinying Guo, Mingyuan Zhang, Zhengyu Lin, Haiyi Mei, Chen Wei, Ruisi Wang, Wanqi Yin, al. Digital life project: Autonomous 3d characters with social intelligence. In CVPR, 2024. 3
Zhi Cen, Huaijin Pi, Sida Peng, Zehong Shen, Minghui Yang, Shuai Zhu, Hujun Bao, and Xiaowei Zhou. Generating human motion in 3d scenes from text descriptions. In CVPR, 2024. 2
Rui Chen, Mingyi Shi, Shaoli Huang, Ping Tan, Taku Komura, and Xuelin Chen. Taming diffusion probabilistic models for character control. In SIGGRAPH, 2024. 2, 6,
Baptiste Chopin, Hao Tang, Naima Otherdout, Mohamed Daoudi, and Nicu Sehe. Interaction trans- former for human reaction generation. IEEE Transactions on Multimedia, 2023. 2,
Zhiyang Dou, Xuelin Chen, Qingnan Fan, Taku Komura, and Wenping Wang- C- ase: Learning conditional adversarial skill embeddings for physics-based characters, In SIGGRAPH Asia, 2023.
2.
Mihai Fieraru, Mihai Zanfir, Elisabeta Oneata, Alin-Ionut Popa, Vlad Olaru, and Cristian Sminchis- escu. Three-dimensional reconstruction of human interactions. In CVPR, 2020. 3
Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik. Recurrent network models for human dynamics. In ICCV, 2015. 2
Anindita Ghosh, Rishabh Dabral, Vladislav Golyanik, Christian Theobalt, and Philipp Slusallek. Remos: 3d motion-conditioned reaction synthesis for two-person interactions. In ECCV, 2024.
3
Dongjun Gu, Jaehyeok Shim, Jaehoon Jang, Changwoo Kang, and Kyungdon Joo. Contactgen: Contact-guided interactive 3d human generation for partners. In AAAI, 2024. 2.
Félix G Harvey, Mike Yurick, Derek Nowrouzezahraii and Christopher Pal. Robust motion in- betweening. TOG, 2020.
Gustav Eje Henter, Simon Alexanderson, and Jonas Beskow, Moglow: Probabilistic and controllable motion synthesis using normalising flows. TOG, 2020. 2
Daniel Holden, Jun Saito, and Taku Komura. deep learning framework for character motion synthesis and editing. TOG, 2016. 2
Daniel Holden, Taku Komura, and Jun Saito. Phase-functioned neural networks for character con- trol. TOG, 2017. 2
Jaewoo Jeong, Dachee Park, and Kuk-Jin Yoon. Multi-agent long-term 3d human pose forecasting via interaction-aware trajectory conditioning. In CVPR, 2024.
Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as foreign language. NeurIPS, 2024.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv, 2014.
Nhat Le, Thang Pham, Tuong Do, Erman Tjiputra, Quang D. Tran, and Anh Nguyen. Music-driven group choreography. CVPR, 2023. 3
Taeryung Lee, Fabien Baradel, Thomas Lucas, Kyoung Mu Lee, and Grégory Rogez. T2lm: Long- term 3d human motion generation from multiple sentences. In CVPR, 2024.
Peizhuo Li, Kfir Aberman, Zihan Zhang, Rana Hanocka, and Olga Sorkine Sorkine-Homunng Ganimator: Neural motion synthesis from single sequence. TOG, 2022.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. arXiv, 2024.
Han Liang. Wenqian Zhang, Wenxuan Li, Jingyi Yu, and Lan Xu. Intergen: Diffusion- multi- human motion generation under complex interactions. IJCV, 2024.

---

Published as a conference paper at ICLR 2025
Donggeun Lim, Cheongi Jeong, and Young Min Kim. Mammos: Mapping multiple human motion with scene understanding and natural interactions. In CVPR, 2023.
Hung Yu Ling, Fabio Zinno, George Cheng, and Michiel Van De Panne. Character controllers using motion vaes. TOG, 2020. 2,
Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling- Yu Duan, and Alex C Kot. Ntu rgb+ 120: A large-scale benchmark for 3d human activity understanding. IEEE transactions pattern analysis and machine intelligence, 2019. 3
Julieta Martinez, Michael J. Black, and Javier Romero. On human motion prediction using recurrent neural networks. CVPR, 2017.
Qianhui Men, Hubert PH Shum, Edmond SL Ho, and Howard Leung. Gan-based reactive motion interaction. Computers & Graphics,
synthesis with class-aware discriminators for 2022. 2
Xiaogang Peng, Siyuan Mao, and Zizhao Wu. Trajectory-aware body interaction transformer for multi-person pose forecasting. In CVPR, 2023. 3
Mathis Petrovich, Michael J. Black, and Gül Varol. TEMOS: Generating diverse human motions from textual descriptions. In ECCV, 2022. 16
Huaijin Pi, Sida Peng, Minghui Yang. Xiaowei Zhou, and Hujun Bao. Hierarchical generation of human-object interactions with diffusion probabilistic models. In ICCV, 2023. 2
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text- Conditional Image Generation with CLIP Latents. arXiv, 2022.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. NeurIPS, 2019. 5
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022. 4
Pablo Ruiz-Ponce, German Barquero, Cristina Palmero, Sergio Escalera, and José Garefa- Rodríguez. in2in: Leveraging individual information to generate human interactions. In CVPRW, 2024. 3,
Zehong Shen, Huaijin Pi, Yan Xia, Zhi Cen, Sida Peng. Zechen Hu, Hujun Bao, Ruizhen Hu, and Xiaowei Zhou. World-grounder human motion recovery via gravity-view coordinates. In SIGGRAPH Asia, 2024.
Yi Shi, Jingbo Wang, Xuekun Jiang, Bingkun Lin, Bo Dai, and Xue Bin Peng. Interactive character control with auto-regressive motion diffusion models. TOG, 2024. 2
Li Siyao, Tianpei Gu, Zhitao Yang, Zhengyu Lin, Ziwei Liu, Henghui Ding, Lei Yang, and Chen Change Loy. Duolando: Follower gpt with off-policy reinforcement learning for dance accompaniment. In ICLR, 2024. 6,7
Jiaming Song, Chenlin Meng, and Stefano Ermon Denoising diffusion implicit models. arXiv, 2020. 6
Sebastian Starke, Yiwei Zhao, Taku Komura, and Kazi Zaman. Local motion phases for learning multi-contact character movements. TOG, 2020. 3, 4
Sebastian Starke, Yiwei Zhao, Fabio Zinno, and Taku Komura. Neural animation layering for syn- thesizing martial arts movements. TOG, 2021. 3,4
Sebastian Starke, Paul Starke, Nicky He, Taku Komura, and Yuting Ye. Categorical codebook matching for embodied character controllers. TOG, 2024. 4
Mikihiro Tanaka and Kent Fujiwara. Role-aware interaction generation from textual description. In ICCV, 2023. 2,3
12

---

Published as a conference paper at ICLR 2025
Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano.
Human motion diffusion model. In ICLR, 2023. 3,5
Jungdam Won, Deepak Gopinath, and Jessica Hodgins. Control strategies for physically simulated characters performing two-player competitive sports. TOG, 2021." 3
Yiming Xie, Varun Jampani, Lei Zhong, Deqing Sun, and Huaizu Jiang. Omnicontrol: Control any joint at any time for human motion generation. arXiv, 2023. 2
Liang Xu, Xintao Lv, Yichao Yan, Xin Jin, Shuwen Wu, Congsheng Xu, Yifan Liu, Yizhou Zhou, Fengyun Rao, Xingdong Sheng, et al. Inter-x: Towards versatile ysis. In CVPR, 2024a.
interaction anal-
Liang Xu, Yizhou Zhou, Yichao Yan, Xin Jin, Wenhan Zhu, Fengyun Rao, Xiaokang Yang, and Wenjun Zeng. Regennet: Towards human action-reaction synthesis. In CVPR, 2024b. 2,
Yifei Yin, Chen Guo, Manuel Kaufmann, Juan Zarate, Jie Song, and Otmar Hilliges. instance segmentation of close human interaction. CVPR, 2023.
Hi4d: 4d
Mohamed Younes, Ewa Kijak, Richard Kulpa, Simon Malinowski, and Franck Multon. Maaip: Multi-agent adversarial interaction priors for imitation from fighting demonstrations for physics- based characters. Proceedings of the ACM on Computer Graphics and Interactive Techniques, 2023. 3
Ye Yuan and Kris Kitani. Dlow: Diversifying latent flows for diverse human motion prediction. In ECCV, 2020. 16
He Zhang, Sebastian Starke, Taku Komura, and Jun Saito. Mode-adaptive neural networks for quadruped motion control. TOG, 2018. 2
Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xi Shen, and Ying Shan. Generating human motion from textual descriptions with discrete rep- resentations. In CVPR, 2023. 2, 4, 5, 6, 15
Qingxu Zhu, He Zhang, Mengting Lan, and Lei Han. Neural categorical priors for physics-based character control. TOG, 2023a. 3
Wentao Zhu, Jason Qin, Yuke Lou, Hang Ye, Xiaoxuan Ma, Hai Ci, and Yizhou Wang. Social motion prediction with cognitive hierarchies. In NeurIPS, 2023b. 3
13

---

Published as a conference paper at ICLR 2025
APPENDIX
A DATASET DETAILS
A.1 DATA COLLECTION DETAILS
Figure 7 illustrates our MoCap system. We use 12 highrresolution cameras with 120 FPS around the persons during our capture process. Figure (a) and (b) shows the bird-view and the side-view of the camera positions. We apply 50 markers illustrates the motion capture setup and environment. The two actors wear motion capture suits with markers.
person shown in Figure 7 (c) and (d). Figure
(a) Biew Cameras
(b) Side-view Cameras
(c) Front-view Markers
(d) Rear-view Markers
Figure 7: Cameras and markers. We have 12 cameras around the persons: (a) cameras from bird-
view and (b) cameras from side-view. We put 50 markers on one person as (c) front-view and (d) rear-view of markers.
Figure 8: Data collection. During data collection, the actors motion capture suits with markers and perform boxing in the center of the area.
A.2 DATASET STATISTICS
Table Dataset staties. Types of movements in the DuoBox.
# Sequences
Types of Movements
25
Simple punches. Jab, cross, (lead, rear) uppercut, (lead, rear) hook, defense, dodge, evade, counter, counterattack, parry.
19
Combinations punches. Jab-jab; jab-cross-rear uppercut; jab-lead uppercut;; jab-cross-cross; jab-cross-lead uppercut; jab-cross-jab-rear uppercut; jab-cross -rs-crostttepleat hook; jab-rear uppercut-lead upper cut- cross; jab-rear uppercut-jab-reat uppercut; jab face-cross body-jab face; jab-lead hook-cross;
72
Free sparring.
14

---

Published as a conference paper at ICLR 2025
We recorded 1 116 sequences in total. The detailed statistics presented in Table 5. In 25 sequences, the participants were instructed to perform simple punching actions. In 19 sequences, they were asked to execute specific boxing punch combinations In the remaining 72 sequences, they engaged in free-style boxing, simulating an actual match.
A.3 ETHICAL CONSIDERATIONS
The participant group consisted of three male individuals aged 18-22, who were boxing enthusiasts recruited from boxing club. All participants provided informed consent prior to participation and were fully informed of the research purpose. Since we used optical motion capture equipment, no photographic images of the participants were collected. ensure privacy and confidentiality, all data was anonymized during processing, and any identifying information was removed.
B ONLINE TWO-CHARACTER MOTION GENERATION ALGORITHM
We present a pseudo-algorithm in Algorithm outlining the process of generating two-character motions using reaction policy, described in Section 3.3.
Algorithm 1: Online Two-Character Motion Generation with Reaction Policy
Input: Initial poses: {p' Jimm for Agent, {ro) for Opponent; Reaction policy: P(O,,A; History buffer size: W; Frames per round: d 4:
Output: Generated motions: ( for Agent, (ub) for Opponent;
Initialize history buffers: BA + (P'M Bo t K
Initialize time index: t 5- d;
for each round until T frames generated do (ritidd Jimi f- P(Bo, BA);
(PO Fi=1 + - P(BA,
dd Append BA with (p't 1i=1:
Append Bo with {Po J
|B > W then
Remove the oldest element from BA and keep length of W; end
|Bl > W then
Remove the oldest element from Bo and keep length of W;
end
Update time index l d;
end
C DETAILS OF NETWORK ARCHITECTURE
The VQ- VAE network architecture follows Zhang et al. (2023). Both the encoder and decoder con- sist of ID convolution and residual blocks. Temporal downsampling is achieved using convolutions with stride 2, while upsampling is performed via nearest interpolation. The downsampling and upsampling are applied twice, resulting in a total downsampling rate of d 22 4.
The two transformer models in Figure 2 use 8 transformer layers with 4 heads. The diffusion model consists of single-layer MIP.. We also tried various types and numbers of layers of the MLP the produce lower FIDs. An additional ablation towards the diffusion network design can be found at Appendix G.3. During training, the next latent predictor and the online motion decoder are trained together. The output next latent shaped with (B, L, C) are reshaped to (BxL, C) and passed to the online motion decoder.
diffusion process, but we found that single-layer MI. is sufficient and effective
httpps:/ www.wwww.be..coo....
YC7K
15

---

Published as a conference paper at ICLR 2025
D IMPLEMENTATION OF THE BASELINES
Here, we provide the details of the baseline implementations used for comparison with our method, which evaluated against several state-of-the-ar motion generation baselines. (1) ForerFormer. InterFormer is an online deterministic reactive motion prediction method. We retrain this model on our dataset. (2) CVAE-AR. We designed CVAE-based method specifically for comparison, utilizing the standard CVAE pipeline with KL and reconstruction loss. The encoder and decoder architectures follow those from TEMOS (Petrovich al., 2022). (3) CAMDM. CAMDM is an auto-regressive diffusion model that generates motion sequences with root control. We adapt the model by using the opponent's motion in place of root control and retrain it on our dataset. (4) T2MGPT-online. T2MGPT is originally a text-to-motion model. To make it online method, we modify it to immediately decode each newly generated token into raw motion using a VQ-VAE decoder. The opponent's motion used input instead of text. (5) Duolando-oftine, Duolando is designed for the duet dance generation. We adapt its network architecture but exclude its RL finetuning for the comparison.
E INFERENCE SPEED ANALYSIS
We also test the inference speed and the results are shown in Table 6. Although our model is the slowest, can achieve real-time inference and the lowest FID scores.
Table 6: Inference speed. We compare the inference time per frame with other methods. All methods are tested single Nvidia RTX 4090 GPU.
Methods
InterFormer
CVAB-AR
CAMDM
12MGPT-online
Duolando-ofline
Ours
Inference time per frame L
2
17
22
Reactive per-clip FID
15.061 47.194
26.010
28.503
30.159 122,463
13.606
9,599
Two-character per-clip FID
92.978
70.416
25.283
F MOTION DIVERSITY ANALYSIS
For each sequence input in the test set, we generate 10 samples and calculate Average Pairwise Distance (APD) scores to evaluate the motion diversity Yuan & Kitani (2020). The results are shown Table 12. A greater value means the generated motions are more diverse. Among the methods, InterFormer is deterministic and therefore lacks diversity. We observed that other basclines show greater diversity, primarily due to their higher levels of error accumulation.
Table Motion diversity, We compare the generated motion diversity with baselines.
Methods
InterFormer
CVAE-AR
CAMDM
T2MGPT-online
Duolando-oftline
Ours
Reactive APD
0.000
2.375
3,385
3.728
1.726
2.154
Reactive per-olip FID 1
26.010
28.503
30.159
13.606
9,599
Two-character APD
0.000
4.431
6.300
7.143
3.709
Two-character per-clip FID 1
47.194
92.978
70.416
122.463
25.283
G MORE ABLATION STUDY
G.1 ABLATION STUDY OF THE TRAINING PROCESS
We also conduct a more specific ablation study on the training process. We compare our method with two variants: (1) split training: split the training of the next latent predictor and the online motion decoder. The predicted next latent is not fed into the online motion decoder while training. (2) stop gradient: stop the gradient flow from the online motion decoder to the next latent
16

---

Published as a conference paper at ICLR 2025
predictor (the diffusion model). The results are shown in Table 8. From the table, observe that split training outperforms our method in reactive per-frama metrics, RO, and FS, but performs poorly in the two-character scenarios. We conclude that supervising the online motion decoder with the ground truth pose allows for joint optimization of both the next latent predictor and the online motion decoder, leading better overall performance compared to the ablation methods.
Table Ablation study of the training process. We conduct an ablation study towards the training process. Among them, bold indicates the best results. means lower is better. + means closer to the real data is better.
Reactive
Two-character
Methods
FID,
FID
RO-+
FS
RO-
FS-
frame
Per-trans.
Per-clip
Per-frame
Per-trans.
Per-clip
Real
24.7%
24.7%
0.97
split training
0.522
1.203
10.004
33.4%
0,96
1.618
2.679
29.574
19.8%
0.94
stop gradient
0.536
1.190
11.064
36.3%
1.01
1.822
3.589
34.797
26.1%
0.98
Ours
0.535
0.995
9.5998
34.7%
1.02
1.394
2.105
25.283
24.1%
0.97
G.2 ABLATION STUDY OF THE VISIBLE WINDOW SIZE
We conduct an ablation study on the visible past window size W stated in Section 3.1. We compare our setting W = 60 with W = 40 and W = 80. The results are shown in Table 9. Finally, we select W 60 for its better performance on FID scores.
Table 9: Ablation study of the visible window size. We conduct an ablation study towards the visible window size W. Among them, bold indicates the best results. + means lower is better. -> means closer to the real data is better.
Reactive
Two-churacter
Methods
FDD
FDD
RO-+
FS
RO-
FS ++
Per-frame
trans.
Per-clip
Per-frame
Per-trans.
Per-elip
Real
24.7%
0.97
24.7%
0.97
W=
40
0.610
1.416
11.631
33.7%
0.99
1.982
3.483
34.530
19.0%
0.94
W
80
0,583
1,038
10.994
35.1%
1.02
1.625
2.680
30.319
21.2%
0.97
W = 60 (Ours)
0.535
0.995
9.5998
34.7%
1.02
1.394
2.105
25.283
24.1 %
0.97
G.3 ABLATION STUDY OF THE DIFFUSION NETWORK DESIGN
As stated in Appendix C, we explored various network architectures during the design phase. How- ever, we ultimately found that a single-layer MLP not only offers faster inference but also achieves lower FID. To support our statement, we conduct an ablation study on the diffusion network design. We compare our single-laye MLP with five variants: (1) 2-MLP: using 2 layers of MLP. (2) 4- MLP: using 4 layers of MLP. (3) 1-ResNet: using layer of ResNet. (4) 2-ResNet: using 2 layers of ResNet. (5) 4-ResNet: using layers of ResNet. The ResNet is adopted from Stable Diffsuion's and we make some modifications: (1) we use linear norm, (2) we use linear layer instead of convolution. The results are shown in Table 10.
From the table, we can conclude that the 1-MLP design achieve better performance generally. We considered several possible reasons to explain this phenomenon. First, more complex networks may overfit the training dataset faster, potentially leading to poorer performance the test dataset. Second, since the diffusion denoising process involves 1000 steps, simpler transformations at each step might already suffice achieve the desired results.
Phttps://ee thub. Sttssstttilllttltttett A/ggbeiratt ve
17

---

Published as a conference paper at ICLR 2025
Table 10: Ablation study of the diffusion network design. N-MLP means using N layers of MLPs. N-ResNet means using N layers of ResNet. Our method use the 1-MLP design.
Reactive
Two-character
Methods
FIDI
FID4
RO
FS-+
RO
FS +
Per-frame
Per-trans.
Per-clip
Per-frume
Per-trans.
Per-clip
Real
24.7%
24.7%
0.97
2-MLP
0.812
1.682
14.525
35.2%
2.111
3.594
35.970
20.5%
0.87
4-MI.P P
0.763
1.472
14,258
35.3%
0.92
2,030
3.474
36.831
21.2%
0.86
I-ResNet
0.656
1.324
12.653
34.5%
0.94
1.727
3.000
32.142
21.5%
0.92
2-ResNet
0.624
1.235
11.650
33.4%
0.95
1.750
2.750
31.887
20.9%
0.93
4-ResNet
0.703
1.518
13.610
1.925
3.256
34.499
0.89
1-MLP (Ours)
0.535
0.995
9,5998
34.7%
1.02
1.394
2.105
25.283
24.1%
0.97
H APPLYING OUR METHOD TO THE INTER-2 X DATASET
We selected three actions from the Inter-X with varying contact frequencies--hhat kick, and dance in increasing order of contact frequency. We use the first 22 joints from the SMPLX skele- ton. The network and the training protocols remain unchanged. For the two-charaeter FID score, calculate the per-frame, per-transition, and per-clip features for each individual, alternating between the "agent" and the "opponent". We removed the RO metric, because in actions unlike boxing, the two characters do not necessarily need to always face each other.
Results shown in Table 11. From the tables, forms the selected baselines, demonstrating its ability to generalize to different types of motions effectively.
observe that method generally outper-
Table 11: Comparison with baselines on the Inter-X dataset. We compare our method with CAMDM and T2MGPTon the Inter-X dataset.
Quantitative results on Inter-X dataset with action category "Chat".
Reactive
Two-character
Methods
FID
FS
FIDI
FS
Per Frame
Per
Per-clip
Per. framfrumme
Per-trans.
Per.clip
0.20
CAMDM
1.693
0.405
21.024
0.67
1.472
0.357
18.931
0.70
T2MGPT-oaline
17.757
0.207
Ours
1.133
0.120
15.372
0.30
1.148
0.10
15.215
0.31
(b) Quantitative results the Inter- X dataset with action category of "Kick".
Reactive
Two character
Methods
FID
FS-+
FID
FS
Per-frame
Per-trans.
Per-olip
Per-frame
Per-trans.
Per-elip
Real
0.76
0.76
CAMDM
1.127
18.427
1,209
19,484
1.99
T2MGPT-oalinc
1-465
0.883
22.283
1.89
1.770
0.914
26.250
1.93
Ours
0.757
0.593
12.861
0.750
0.558
12,828
0.83
(c) Quantitative results on the Inter-X datasct with action category "Dance".
Reactive
Two charcaert
Methods
FID, Per-trans.
FS-+
FID
FS
Per-frame
Per-clip
Per-frame
Per-trans.
Per-clip
Real
0.68
0.68
CAMDM
2.239
1.251
34.199
1.02
2.128
1.192
34.548
1.21
T2MGPT-ellie
0.404
22.380
2.714
0.275
Ours
1.699
0.653
21.300
0.58
1.065
0.435
18.646
0.53
18

---

Published as a conference paper at ICLR 2025
I PENETRATION ANALYSIS
Our skeleton representation is based on positions and rotations exported from motion capture soft- ware. In the main paper, we visualize the predicted rotations by mapping them onto an "Xbot" model for demonstration purposes. Note that the Xbot skeleton differs from the one we use in our method, which makes penetration evaluations based on Xbot less reliable. As a result, penetration cannot be directly evaluated at the mesh level.
To evaluate penetration between skeletons, we approximate the bones using triangular prisms with distance from the centroid frame by frame between the two body meshes by using trimesh. Trimesh.Trrmess Since boxing typically involves instantaneous contact, the proportion of frames with penetration is expected to be very low. Therefore, we report the total number frames with penetration across the entire test set, and the mean penetration volume in cm
vertex 5 em (as shown in Figure 9) and calculate penetration
function.
From the table, we can see that in the reactive setting, our method achieves comparable results to Duolando-offine, which predicts setting, our method achieves a lower mean penetration volume compared to all the baselines, demon- strating its effectiveness in avoiding penetration.
positions relative the opponent's root. In the two-character
Figure 9: Skeleton mesh that is used to compute penetration. We create approximated skeleton meshed using triangular prism with distance from the centroid vertex 5 cm.
Table 12: Penetration analysis. We measure the number of penetration frames and mean penetra- tion volume (in cm²) in the test set. | means lower is better.
Setrings
Metrics
Interformer
CVAE-AR
CAMDM
T2MGPT-anline
Phulande-offiiee
Ours
GT
Reactive
Mean Penetration Volume.
200.43
237.51
200.16
218.78
118.51
161.38
21.37
Two-charneter
Penetration Frames
2617
1858
469
962 365
141
Mean Penctration Volume
222.83
218.15
129.14
111.84
21.37
J CONTACT IMPACT ANALYSIS
Table 13: Cotact impact analysis. All contact analysis is conducted under the condition that one's hand is close enough to the other. #OF represents the number of frames that the opponent moves forward. #OF AB represents the number of frames that the agent moves backward when the opponent moves forward. Ratio #OF AB - means closer to the real data is better.
#OF
Settings
Metries
InterFormer
CVAE-AR CAMDM T2MGPT-online
Duolando-offine
Ours
GT
Reactive
OF DF-AB
1163
997
532
1145
402
761
243
615
505
260
613
272
397
125
Ratio
52.80%
50.60%
48.80%
53.50%
23.30%
52.10%
51.40%
Two-churacter
OF
1179
1484
371
539
816
243
OF-AB
921
716
193
267
420
125
78.10%
48.20%
52.00%
51.40%
19

---

Published as a conference paper at ICLR 2025
To analyze the impact of sparse contact motion generation, first detect contact and then assess whether the agent moves backward or forward response. Specifically, we identify contact by calculating the distance between one character's skeleton mesh and the other's hand joints. If the distance is less than 5 cm, we consider that the hand has successfully made contact with the other. This could include the opponent's s hand hitting any part of the agent vice versa. For these identified contact frames, we examine whether the agent's 's root moves backward when the opponent's root moves forward and compute the proportion of such occurrences. For each frame, we switch the roles of "agent" and "opponent". Results are shown in Table 13. A higher alignment with the ground truth indicates better performance.
From the table. we observe that our method achieves performance closer to the ground truth, demon- strating ability to produce more realistic reactions.
20


Reviews:
[{"review_id": "v4We1PmnRx", "reviewer": "Reviewer_odxr", "summary": "This paper introduces a new two-person boxing dataset to support three tasks: (1) reactive motion generation, (2) two-character interaction generation, and (3) long-term two-character interaction generation. The authors demonstrate that their method can generate motions lasting over one minute and outperforms existing methods across various tasks. Interestingly, the authors verify that their approach can be used in VR scenarios.", "questions": "This paper investigates a very interesting problem. However, the paper does not consider the modeling of contact information during two-person interactions, which seems to limit its application in real robots. Nevertheless, the authors demonstrate the potential of their approach in VR applications. Overall, I have given my current score based on these considerations.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed DuoBox dataset is a strong dataset for two-person interaction, which is highly beneficial to the community. It provides a new test benchmark for three novel tasks.\n\n2. The method proposed in the paper can generate impressive results, significantly outperforming GPT-based methods and capable of generating longer action sequences.", "weaknesses": "1. Given that there is not much actual contact between individuals in this scenario, with contact occurring only at specific moments, have the authors analyzed how the occurrence of such contact influences subsequent motion generation?\n\n2. Since the visualizations provided by the authors seem to be based on some kind of simulation environment, have the authors quantified metrics such as penetration (clipping) or foot sliding to assess the physical plausibility of the generated results?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B7iCwzzwe7", "reviewer": "Reviewer_T1pE", "summary": "This work proposes an autoregressive diffusion model to generate motion for human boxing matches. The main innovation lies in an autoregressive policy that handles both the human actor's and opponent's policy concurrently, ensuring there is an \"action\" and \"reaction\" aspect for each player. To train the proposed method, a new dataset, DuoBox, is collected. Results show that the proposed method can generate long and realistic boxing motions.", "questions": "Does the model generalize beyond boxing? There have been several two-person interaction datasets proposed (e.g., [1]).\n[1] Xu, Liang, et al. \"Inter-x: Towards versatile human-human interaction analysis.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This work addresses an important problem in two-person interaction motion generation: handling interaction. The proposed autoregressive approach is sound and intuitive.\n- The proposed online motion decoder is a well-thought-out solution for applying VQ-VAE styled motion representation in the autoregressive motion generation scheme.\n- The special focus on root orientation prediction is crucial for the boxing task, based on the ablation.\n- The generated motion is of high quality and visually pleasing. Quantitative results also show that the generated motion is of high quality and outperforms prior arts.\n- The study on sparse VR input is a welcome addition, showcasing the method's applicability in animation.", "weaknesses": "- At video 0:45 (of the gray box), it is clear that the reaction of the other character can contain large penetration and unrealistic interaction. These should be properly handled. In fact, from watching the video, I would say that while the motion quality of the two characters is high, the \"reaction\" part is not well-demonstrated. There are many cases where punches are thrown, but the opponent does not evade and has no discernible reaction.\n- I feel that given the study's scope, the title should reflect the focus on \"boxing\". Many proposed techniques (such as root orientation) focus on boxing. Extending to two-person sports (fencing, dancing, etc.) could be an interesting direction, but the proposed method has not been shown to handle generic two-person interactions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n6jBKYCkpT", "reviewer": "Reviewer_qtLW", "summary": "The paper present a new method for human reactive motion generation. The method contains a history encoder to encode historic motion using VQ-VAE or MLP, a latent predictor using Transformer encoder and diffusion and a latent decoder using Transformer decoder. The proposed architecture can be use for reactive motion generation but also for two persons interaction generation and support long term generation (1800 frames). The proposed method outperforms the state of the art quantitatively and qualitatively on a new boxing dataset containing more than 1 hour of 3D motions.", "questions": "- In figure 2 \"Tranformer\" is used in both the latent predictor and the motion decoder. from what I understood of the text there is only a Transformer encoder in the latent predictor and a transformer decoder in the motion predictor. If it is the case the authors should update the figure to reflect this. Using the term \"Transformer\" alone make it seems it contains both encoder and decoder.\n\n- Notation error: \"d\" is used both for the down-sampling factor (line 198) and for the initial number of frame (line 248)", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method is relatively simple yet performs very well and is very versatile as it can be used for reactive motion generation but also for two persons interaction generation.\n\n- The method is efficient even for long term generation.\n\n- The paper presents extensive experiments and ablation.\n\n- The authors propose a new interaction (boxing) dataset with long sequence. This is very welcome due the the scarcity of long interaction datasets.\n\n- The authors provided qualitative results in video format.", "weaknesses": "- The experiments are performed on only one dataset and this dataset is the new one proposed in the paper. It would have been better to also have experiment on another dataset e.g one of those used for Duolando or Interformer.\n\n- The paper does not precise if the dataset will be released to the public.\n\n- While I understood the reasoning behind using diffusion with MLP for the latent predictor, I am surprised that it works so well with only a single layer. What was the intuition behind the choice of this single layer MLP, did the authors try other configurations for the denoising network ? This would have been interesting to explore this more in details.\n\n- A few things are still not clear to me in the methodology:\n\n    - Line 176: what is the reasoning behind the exclusion of the y-axis when extracting r_off and r_dir?\n\n    - Why does the opponent input doesn't contain r_off and r_dir when the agent input uses them?\n\n    - Also on the subject of differences between agent and opponent, why use a VQ-VAE for the AGENT but a MLP for the opponent?\n\n    - How does the two-character motion generation works exactly? Does this work in a auto-regressive manner? i.e. generate agent motion T -> generate opponent motion T+1 -> generate agent motion T+1->... . Does it need two separately trained networks (one for agent and one for opponent) or only one ? More details are necessary.\n\n- For the two-character motion generation experiments the authors do not compare with Duolando. This is understandable since it was build to generate the motion of only one person in an interaction. However this seems to also be the case for Interformer yet the authors were able to compare against it. Were some modifications done to the network architecture or to the training/inference protocol to make it possible? this should be clarified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cur8c136Vt", "reviewer": "Reviewer_TLUt", "summary": "This paper introduces a novel approach to generating real-time interactions between two characters. The proposed \"Ready-to-React\" method focuses on creating dynamic, continuous motions for each character based on their counterpart’s actions. The key innovation lies in incorporating a diffusion head into an auto-regressive model, which helps mitigate error accumulation and improves the naturalness of long-term interactions. The approach is demonstrated on a boxing task, where it outperforms existing methods in generating coherent and plausible interactions over extended sequences. This technique has potential applications in VR and online interactive environments.", "questions": "1. You mention in the introduction that \"incorporating a diffusion head into an auto-regressive model, which can respond to the counterpart’s motions in a streaming manner while ensuring the naturalness and diversity of the motions\". How can this method help diversity?\n2. Can you use a few baselines that are not deep learning, such as those executed by programmed programs? Because in this scene (such as ), I think pre-programmed is quite smooth.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Experimental adequacy: The experimental result is persuasive compared with existing baselines and ablation studies are sufficient to prove the effectiveness of the proposed method.\n2. The insight into error accumulation mitigation in two-character motion generation is interesting.", "weaknesses": "1. Lack of significance: The background of this work is too narrow and I do not think it is important to study. Also, from my perspective, this method is for interaction generation. However, there has been much work studying video generation which seems to be a lot more complicated than this paper. \n2. Lack of novelty: The proposed method simply uses diffusion to generate representations instead of directly generating actions which is lack of novelty and contribution. \n3. This method is only tested in one dataset which is not sufficient although you mention this scene is a difficult one.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)