Think Then React: Towards Unconstrained Action-to-Reaction Motion Generation
Authors: Wenhui Tan
Boyuan Li
Chuhao Jin
Wenbing Huang
Xiting Wang
Ruihua Song
Venue: ICLR 2025 Poster (2025)
Keywords: Human Reaction Generation
3D Human Motion
Large Language Model
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Modeling human-like action-to-reaction generation has significant real-world applications, like human-robot interaction and games.
Despite recent advancements in single-person motion generation, it is still challenging to well handle action-to-reaction generation, due to the difficulty of directly predicting reaction from action sequence without prompts, and the absence of a unified representation that effectively encodes multi-person motion.
To address these challenges, we introduce Think-Then-React (TTR), a large language-model-based framework designed to generate human-like reactions.
First, with our fine-grained multimodal training strategy, TTR is capable to unify two processes during inference: a thinking process that explicitly infers action intentions and reasons corresponding reaction description, which serve as semantic prompts, and a reacting process that predicts reactions based on input action and the inferred semantic prompts.
Second, to effectively represent multi-person motion in language models, we propose a unified motion tokenizer by decoupling egocentric pose and absolute space features, which effectively represents action and reaction motion with same encoding.
Extensive experiments demonstrate that TTR outperforms existing baselines, achieving significant improvements in evaluation metrics, such as reducing FID from 3.988 to 1.942.

Published as a conference paper at ICLR 2025
THINK-THEN-REACE TOWARDS UNCONSTRAINED HUMAN IETION-TO-REATIONN GENERATION
Wenhui Tan, Boyuan Li, Chuhao Jin, Wenbing Huang, Xiting Wang & Ruihua Song"
Gaoling School of Artificial Intelligence
Renmin University of China
Beijing, China
(tanwenhui liboyuan, hainchuhaa,,hwniiee xi xitingwang, rsong) @ruc. edw.ccdducc..
The person slands still, the other person stands tront heshim.
Timestep
Action
Thinking Results
Reaction
person reaches her hand, graeps the other person's hands and they shake twice."
3
Stne
ThinkcThen-React
right hand. extends herfhis hands and give hen/him high-five.
penson
person
people raise
They gave other high-five above their beads."
Molion Tokenizer
Pio
Action
C=2
t=3
t=
C=
C=6
t=7
Figure 1: Given human action as input, our Think-Then-Reaee model first thinks by generating an action description and reasons out a reaction prompt. It then reacts to the action based on the results of this thinking process. TTR reacts in real-time manner at every timestep and periodically re-thinks at specific interval (every two timesteps in the illustration) to mitigate accumulated errors.
ABSTRACT
Modeling human-like action-to- to-recctionn generation has significant real-world ap- plications, like human-robo interaction and games. Despite recent advancements in single-person motion generation, it is still challenging to well handle action-to- reaction generation, due to the difficulty of directly predicting reaction from action sequence without prompts, and the absence of unified representation that effec- tively encodes multi-person motion. To address these challenges, we introduce Think-Then-React (TTR), large language- magguage-moeelbbasee framework designed to generate human-like reactions. First, with our fine- tine-grained multimodal training strategy. TTR is capable unify two processes during inference: thinking pro- cess that explicitly infers action intentions and reasons corresponding reaction description, which serve semantic prompts, and reacting process that pre- dicts reactions based on input action and the inferred semantic prompts. Second, effectively represent multi-person motion language models, we propose a unified motion tokenizer by decoupling egocentric pose and absolute space fea- tures, which effectively represents action and reaction motion with same encod- ing. Extensive experiments demonstrate that TTR outperforms existing baselines, achieving significant improvements evaluation metrics, such as reducing FID from 3.988 to 1.942.
INTRODUCTION
Predicting human reaction to human action in real world scenario is an online and unconstrained task, i.e., future states and text prompts are inaccessible, and has board applications in virtual ality. human--rbou interaction and gaming. Recently, significant advancements have been achieved
"Corresponiigg author: Ruihua Song (rsong rww..cd..ccceee

---

Published as a conference paper at ICLR 2025
in the domain of human motion generation especially single- person motion generation, conditioned (Guo et al. 2024: 2022b; Zhang et al. 2023) and action labels (Xu et al. 2023
on text prompts
Guo et al.
2020 Leveraging well -mnottaed human motion datasets (Xu et al. 2024a| Guo et al.
2022a Liu et al.
2020: Plappert et al. 2016), these models employ various generative frameworks,
such as Diffusion Models (Ho et al. 2020; Liang al. 2024 Zhang et al., 2022), Variational Au- loencoders (VAEs) (Kingma| 2013 Petrovich et al. 2021b), and Generative Adversarial Networks (GANs) (Goodfellow et al. 2014 Men et al. (022). to capture cross-modality distributions for bet- ter motion generation. Furthermore, Large Language Models (LLMs) have been applied to human motion generation, demonstrating superior performance (Jiang et al. 2023 Zhang et al. 2024).
However, generating human reaction in multi -prison scenario presents a more challenging task due to two primary factors. First, directly predicting reaction from action sequence is a difficult task with unstability. As shown in Figure given first two action steps, it is ambiguous to distinguish whether the action is "shake hand" or "high five", and this would lead to accumulated error to consequent predicted reactions. Second, dissimilar to single-person motion representation that can adopt an egocentric view, representing human motion in multi-person scenario necessitates both egocentric and absolute information.
Several works have focused on human interaction domain. For instance, InterFormer (Chopin et al., 2023) proposes injecting human skeleton priors into transformer attention layers for effective spatial modeling. InterGen (Liang et al., 2024) introduces a mutual attention mechanism within diffusion process for joint action-reaction generation. However, these methods are not directly applicable to real-world applications, as they rely on extra prompts to condition the generation process. ReGen- Net (Xu al., 2024b), similar to our approach, acknowledges the online and unprompted nature of reaction generation, and proposes a diffusion-based model for online reaction generation. It ob- serves that given the action's 's intention as condition explicitly, the model can achieve superior performance compared to unprompted settings, highlighting the necessity of understanding inter- action semantics for reaction generation. However, ReGenNet directly models action-to-reaction generation process, without inferring action intention, thus achieving subpar performance.
To address these challenges, we propose Srink-Then-Reee model (TTR), an L.MM-based model designed to predict human reactions in online and unprompted settings with the following innova- tions: First, to unifiedly represent human motion in multi-person scenario, we propose decoupled space-pose tokenizers that separately handle egocentric pose features and absolute space features. Specifically, we train a VQ- VAE (Van Den Oord et al.] 2017) to encode egocentric human pose se- quences (i.e., the space features are normalized, to ensure codebook utilization) into L.LM-reddable tokens. To maintain spatial features which are crucial in multi-person interaction scenarios, we propose space tokenizer that encodes positions and orientations space tokens. We concatenate initial space tokens as prefixes to pose sequences, indicating the initial absolute state of an egocentric motion sequence. Second, to stabilize reaction prediction process, we introduce a novel framework that is capable to automatically infer text prompts for reaction generation. Specifically, TTR unifies two processes within one model: thinking process that infers action intent and reasons reaction description, and a reacting process that takes both the action motion and inferred prompts as in- put, to generate precise and semantically appropriate reactions. Third, to adapt a language model to motion modality, we design multi-task and multi-stage training pipeline consisting of motion- text, space-pose and motion motion-motion generation tasks. With our proposed training strategy, TTR is capable to effectively build correlations between text, motion and space modalities.
In summary, our main contributions are as follows:
We introduce unified motion tokenizer that effectively represents both absolute space and egocentric pose features into LLM-readable tokens in multi-person scenario.
We propose a novel framework Shink-Thee-React with fine- -reigra training strategy, en- abling the adaptation of language model to a multi-modal model encompassing two pro- cesses: inferring action intention and reasoning reaction description, and predicting reac- tion, within one model, thus ensuring generation quality.
Through extensive experiments, we demonstrate that our approach surpasses existing base- lines by substantial margins, achieving an FID improvement from 3.988 to 1.942, along with other ranking metrics!"
Project page: https:///mntt / Then- React, .i gith.. 10/
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORK
2.1 HUMAN MOTION REPRESENTATION
Representing human motion can be mainly categorized into two norms: continuous and discrete rep- resentation. Human motion be intuitively represented in continuous space as joint positions of 3D human skeleton extracted with SMPL (Loper et al., 2015). However, simply using joint position lacks enough information like joint velocity and rotation. Guo al. (2022a) proposes redundant representation, consisting human root angular velocity, root linear velocity, root height, joint posi- tion. joint joint rotation, and foot-ground contact signals. This representation focuses on egocentric view in single--erse person scenario. Based on this, several works propose leveraging VQ-VAE (Van Den Oord al. 2017) to encode human motion into discrete tokens, which can be fed into language models, adapting motion prediction task to language modeling task (Jiang et al.) 2023] Guo et al., 2024 Zhang et al. 2023] Guo et al. 2022b). This technique proved be quite effective especially in the era of LLMs.
Representing human motion in multi-person scenario more complicate than in single-person do- main, as it is required to simultaneously representing egocentric pose and absolute space features (i.e., distance and orientation among multiple persons). Contrary to use normalized position and ori- entation in egocentric view, Liang et al. (2024) proposes non-canonical representation that directly takes global signals (joint positions and velocities) as continuous motion representation, to main- tain absolute information in multi-person person scenario. Similar Guo et al. (022a)), it combines joint position, velocity, rotation and foot-gro goot-ground contact as continuous motion feature. Based on previ- ous works, we propose unified tokenizer, which decouples space and pose tokenization process, enabling effective adaptation of discrete motion representation to multi-person domain.
2.2 HUMAN MOTION GENERATION
The field of Human Motion Generation focuses on creating realistic and diverse 3D human motion
from various input modalities, including text (Zhang et al.) 2023 Guo et al. 2022b Tiang et al.,
2023 Guo et al. 2024. [Liang et al. 2024), action labels (Guo et al. 2020 Xu et al. 2023 Petrovich et al. 30.1a). and human motion (Chopin et al. 2023 Liang et al. 2024 Xu et al. 2024b). Most re- search has concentrated on text-conditioned single-person motion generation (lext-to-motion) tasks. In this area, several works have utilized generative models commonly used the vision domain, such GANs, VAEs, and Diffusion Models, to generate human motion sequences. Another promi- nent approach (Zhang et al., 2023) Guo et al. [202bb] Jiang et al. 20233] Guo et al., (022) employs VQ-VAR to encode human motion sequences into one-hot tokens, which are then processed by auto-regressive models. This method converts the high-dimensional generation task into a next- token prediction task, effectively leveraging pre-trained large language models for more accurate text prompt understanding and diverse motion generation.
Recently, there has been growing interest in generating human motion in multi-person scenarios. InterGen (Liang et al. 2024) introduces a dual-person interaction dataset with detailed textual de- scriptions and a diffusion-based model for jointly generating multi-person interactions conditioned on text input. InterFormer (Chopin et al.) (023) utilizes temporal and spatial attention with human skeleton priors to generate human motion sequences reacting to input action sequences. The latest work, ReGenNet (Xu et al., (024bb) employs a diffusion model to generate human reactions based on human actions in a unconstrained and online manner, and points out that given action's 's intention as a condition, the model can achieve superior performance compared to unconstrained settings. However, it directly predicts reaction motion without analyzing semantics of action motion. Our work unifies two processes: thinking process that infers action semantics, and reacting process that predicts reaction motion based on action motion and the thinking results, ensuring to generate reaction with appropriate semantics.
3

---

Published as a conference paper at ICLR 2025
Absolute Space Tokenizer
= 1
t=
t=1 t=
Spanals
Input
rX
Action
22
32
918>
444
4p72
Output Reactions
< ملكة >
P32
118
P56
P81
Egucentric Pose Tokenizer
Plain Tokenizer
Figure 2: Illustration of our decoupled tokenizer and the plain tokenizer.
3
METHOD
3.1 OVERVIEW
For the task of action-to-reaction generation, given human action a {a=) over Ny frames, aim is generate corresponding reaction b {br without any input prompts. Most previous works leverage input prompts but they are often inaccessible in unconstrained interaction setting. As the example in Figure I shows, when robot/avatar meets a human, it can only observe the human behaviors, try to understand her/his intents, and think what the robot/avatar is expected to react. There is prompt available to tell what they are going to do.
To address the above problem, we propose a unified framework, Think-Then-BBe Then-React (TTR), for both action understanding and reaction generation. First, we propose unified tokenizer convert both egocentric poses and absolute spatial location and orientation information into tokens (Section 3.2) Then, we propose unified Large Language Model (LLM) based model that are pre-trained on three categories of motion and language related tasks, such as describing a motion, and then fine-tune the model with instructions of predicting a reaction from a given action (Section 3.3).
To avoid confusion, we define pose as human body posture or movement within brief time interval, such as "taking one step forward", and a pose can be represented as single token. A motion refers to sequence of poses, starting with an initial spatial state represented by space features. For example, a motion could be "a person walks three steps".
3.2 UNIFIED MOTION REPRESENTATION
To represent one two persons (denoted by pl and p2) in absolute coordinate system, where the x-z plane represents the horizontal plane and y-axis represents the vertical direction, we nor- malize their centers at the origin while facing positive axis. Then for each frame, we extract the 3D skeletons' joint position, velocity and rotation as normalized (or egocentric) pose feature. Be- fore normalizing, we keep the two persons' pelvis 2D coordination æ, and body orientation r maintain absolute space features. The y-axis (vertical) is not included, as few motions begin in "flotinng" state. Based on pose and space features of pl and p2, we propose a unified tokenizing pipeline to convert them into LLM-readable tokens.
3.2.1 EGOCENTRIC POSE TOKENIZER
Our aim is to convert continuous pose features into discrete pose tokens like "<pps"" < < >.." To achieve this, adopt VQ-VAE (Van Den Oord et al.] 2017), similar to Jiang et al. (2023), as the egocentric pose tokenizer. The pose tokenizer consists of an encoder and a decoder D. first encodes continuous motion features, i.e., the 22 joints' position, rotation and velocity vector
4

---

Published as a conference paper at ICLR 2025
= {m,) into N, discrete pose tokens, i.e., Nr timesteps, which is downsampled from Ny- Specifically, & and "D are ID convolution networks with downsample and upsample blocks.
We first obtain the latent pose representation of motion sequence p E(m). Then, a learnable codebook for human poses P E TR,,, with Np entries in size dz. A quantization
up
operation Q--) is applied on the encoded motion latent features by replacing each row vector p; with its nearest codebook entry Pk. The process is formulated as:
Pquantized = (()) = (arg min llp: -Pxl) E Ryxd,
(1)
P&EC
Then, we obtain the reconstructed pose feature ın through the decoder mn = D(P,). The overall process of the VQ- (AE can be formulated as:
m ((Q(E(m)))
This is trained via a reconstruction loss with codebook commitment loss. Noting that the argmin operation is non-differentiable, we simply copy the gradients from D to & as the estimated gradient. Furthermore, for smoother reconstructed motion and a stable training process, we add an extra veloc- ity regularization in the reconstruction loss and employ exponential moving average (EMA) Hunter Hunter (1986) with codebook reset techniques, following ZZhngg et al. (2023). More details about this sec- tion are provided in Section A.I
3.2.2 ABSOLUTE SPACE TOKENIZER
For better generalization capability, all motions, including actions and reactions, are normalized to the original point and same direction before being tokenized. Therefore, absolute space information, ie., the human body 2D position and orientation of each person, is omitted. To extend egocentric pose tokens with absolute space information, we propose converting position and rotation of a per- son's center point into
tokens.
As shown Figure 2 before normalizing a human motion, we first extract the center point's fea- tures, i.e., the position and and orientation We then compute the range of 2, 2, and across the dataset to get the maximum and minimum values. These ranges are uniformly divided into N₂ bins, converting each continuous value discrete tokens. For example, x 0.55 will be represented as "<Z15>" if all the x positions are in range [-1,]] with N₂, = 20 bins.
Finally, we use unified coding system represent action, reaction, and their relative information. Specifically, for each motion, we apply absolute space tokenizer to encode initial x, z, and into egocentric pose tokens, and apply pose tokenizer to encode the following pose sequence, i.e., the following motion, into pose tokens. Such tokens enable training model that can understand and generate motion and language simultaneously effectively and efficiently in the subsequent phase.
3.3
UNIFIED LLM BASED MOTION UNDERSTANDING AND GENERATION
3.3.1 PRE-TRAINING
To adapt a language model into a maniun-language model, we first pre-train the model with multiple tasks in diverse formats. The pre-training tasks can be categorized into three main types:
(1) Motion Text. To enable the model to understand and generate human motion, we com- bine the action and reaction token sequences to construct prompts, which are then fed into the model to generate corresponding textual descriptions, and vice versa. For example, the input se- quence could be "Describe the interaction. Action: <<>>><2>><<> <r>>>>>>>>>>>> <p>>>>>>>>>> <p?>... Reaction: < <p">...""". and the target response is: "One waves the right hand, and the other waves back". However, reaction motions not given during the inference phase. There- fore, the reaction motion is randomly dropped during the training phase to enable the model to infer the interaction from the action motion solely. We also pre-train model by the instructions predicting action and reaction token sequences from an interaction description prompt in text.
(2) Puse Space. Spatial information is represented by orthogonal one-hot tokens, but it may be helpful to infuse auxiliary spatial information into the model. Specifically, we design two tasks: i) Egocentric pose to absolute space: Given space token and subsequent pose tokens of & timestep,
5

---

Published as a conference paper at ICLR 2025
train the model to predict the space tokens of t + timestep. For example, given input space token <212> and a pose token <pss>, which represents "stepping forward". the target output should be <213>,, denoting spatial transition. ii) Absolute space to egocentric pose: Similarly, given space tokens of / and i + timestep, the model is trained to predict pose tokens between them.
To capture more fine-grained action-reaction relationships, half of the action sequence and the second half of the reaction sequence, along with their corre- sponding initial spatial tokens, as input. The model is then pre-trained complete the remaining motion clips. For example, given a sequence spanning ten timesteps 11:10, we feed the first half of the action 01:5 and the second half of the reaction b6:10. supervising the model predict 06:10 and 61.5. Alternativly, we feed b1s5 and 06:10 to predict 01:5 and b6:10-
(3) Motion Motion.
the first
During pre-training, we jointly train all the tasks in non-causal manner for better efficiency. Own- ing to our unified motion and language architecture and space-pose token representation, single person motion and text data can be seamlessly integrated into the training process. We adopt Hu- manML3D (Guo et al.) (022a)), large scale single person motion-text dataset to facilitate pre- training. To avoid overfitting, we prepare 20 prompt templates for each task and randomly mask out 15% of tokens to be predicted during training. In addition, we adopt random clipping of motions as augmentation. We also find that text generation tasks converge much faster than motion genera- tion tasks. To balance different training tasks, use the validation losses of the tasks sampling weights to dynamically select the training source for each epoch.
3.3.2 FINE-TUNING
After pre-training, the motion-languna model is well-structured with knowledge of pose, space, and text. To make the model applicable to online eetiottto-reactionn generation, we fine-tune it in causal manner, focusing on two tasks: thinking and reacting.
The thinking task involves understanding action motion, e-g. "the person is waving hand", and inferring its possible interaction, e.g., two persons wave goodbye to each other", or reaction, "the other person waves back". At each training iteration, we randomly choose the first quarter, half, or the entire action sequence as input to predict the entire interaction caption. However, the entire action motion is not given in the early stage of inference, thus the inferred description based on action motion clips may not be accurate, thus we adopt periodical re-thinking in the inference phase for each N, action tokens given, to dynamically adjust the prompt for reaction generation. We define N, re- thtikiiniig interval.
For the reacting task, we aim to supervise the model to generate reaction motions conditioned on the generated descriptions during the thinking process. However, in the early stages of fine-tuning, the inferred interaction descriptions are not accurate enough to guide the reaction generation process. Thus, we adopt teacher forcing approach. In the early stages, the model takes the ground-truth text prompt as a condition to generate the entire reaction sequence. Mcanwhile, we monitor the validation loss and text generation metrics. When the metrics tend to converge, we begin to sample predicted interaction captions by the model and use them as reaction generation conditions. This process ensures alignment between training and inference, as ground-trui -ruth prompts are inaccessible during inference.
4 EXPERIMENT
We evaluate our proposed method with strong baselines and further analyze contributions of different components, and the impact of key parameters.
4.1 EXPERIMENT SETUP
Dataset. We evaluate all the methods Inter-X dataset, which consists about 9K training sam- ples and 1,708 test samples. Each sample an textual description. As supplementation. text dataset HumanML3D (Guo et al., 2022a), which consists more than 23K annotated motion sequences. We uniformly sample frames for both datasets to 30 FPS.
sequence and three corresponding mix pre-training data with single person motion-

---

Published as a conference paper at ICLR 2025
Evaluation Metrics. Following single-person motion generation (Zhang al. 2023 we adopt the these metrics to quantitatively evaluate the generated motion: R-Precision measures the ranking of Euclidean distances between motion and text features. Accuracy (Acc.) assesses how likely generated motion could be successfully recognized as its interaction label, like "high-five". Frechet Inception Distance (Heusel al.] 2017) (FII) evaluates the similarity in feature space between pre- dicted and ground-truth motion. Multimodal Distance (MMDist.) calculates the average Euclidean distance between generated motion and the corresponding text description. Diversity (Div.) mea- sures the feature diversity within generated motions. All the metrics reported are calculated with batch size set to 32, and accumulated across the test dataset, and we evaluate each method for 20 times with different seeds to calculate the final results at 95% confidence interval.
Evaluation Model. Every metric mentioned above requires an encoder M to extract motion fea- ture. For single person text-to-motion generation tasks, a motion-text matching model are commonly trained as human motion feature extractor. A simple way to transfer this method to interaction do- main is to directly train an interaction- to-toon-ttteetttttt-ee matching model M(a, b, tezt), where action sequence a and predicted reaction sequence b together is regarded as a generated interaction sequence, or a reaction-tor tootoon--tteet----o text match model MMb, text). However, the former one may focus too much on the ground-truth action imput, leading insufficient discriminative power of b's quality, while the latter one lacks semantics provided by action, thus leading to subpar matching capability.
To address the issue, simply uniformly mask off large portion of a, obtaining down-samples action motion sequence a' (downsampled to I FPS in our setting), which serves as a semantic hint for the matching process while not introducing too much emphasis input action sequence. The final evaluation model consists of an masked interaction encoder and a text encoder. We use contrastive loss following CLIP (Radford et al.) 202T). which encourages paired motion and text features to be close geometrically. In addition, we add a classification head after the predicted motion features, to simultaneously predict interaction labels, such as "high-five".
Baselines. To evaluate the performance of our method TTR on online and unconstrained setting, we compare TTR with the following baselines: 1) InterFormer (Chopin et al.) 2023) is a trans- generation model that leverages human skeleton as prior knowledge for efficient attention process. 2) MotionGPT (Jiang et al.] 2023) a motion-language model that leverages an I.LM for motion and text generation. We extend the motion tokenizer of MotionGPT motion, while keeping other settings unchanged. 3) InterGen (Liang et al.)
former based action-to-reactio
to encode 2024) proposes mutual attention mechanism within diffusion process for human interaction gener- ation, we reproduce and adapt IngerGen to state-of-the-art ahe model on action-to-reaction generation. It adopts transformer de- coder based diffusion model, which directly predicts human reaction given action input in uncon- strained and online manner ours.
generation. 4) ReGenNet (Xu et al.]
2024b) is latest state-of-
(Chung et al., 2024 Raffel et al., 2020) as our base model, with extended vocabulary. We warm up the learning rate for 1,000 steps, Both
Implementation Details. For the LLM,, we adopt
peaking at Ic-4 for the pre-training phase, and use the same learning rate for fine-tuning the pre-training and fine-tuning phases are trained on a single machine with 8 Tesla V100 GPUs. The training batch size is set to 32 for the LLM and we monitor the validation loss and reaction generation metrics for early- stopping, resulting about 100K pre-training steps and 40K fine-tuning steps. We set the re-thinking interval N, to 4 tokens and divide each space signal into N₂, 10 bins.
4.2 COMPARISON TO BASELINES
As shown in the upper side of Table T our method TTR significantly outperforms baseline meth- ods in terms of ranking, accuracy, FID and multimodal distance, showing superior human reaction generation quality. Compared to MotionGPT, which adopts similar lanionnl architecture, TTR expresses stronger performance, which we attribute to our unified representation of motion via space and pose tokenizers, enabling effective individual pose and inter-person spatial relationship representation. TTR also surpasses the thik-tthe--ract architecture, improving generated motions by describing observed action and rea- soning what reaction is expected semantic level. In addition, ReGenNet and MotionGPT get closer diversity to the real than our model. We mainly attribute to that, TTR may conduct multiple re-thinking processes during inference, and the inferred semantics may bring higher diversity.
methods, InterGen and ReGenNet, with our
7

---

Published as a conference paper at ICLR 2025
Table 1: Comparison to state-of-the-art basclines and ablation studies of our method on Inter-X dataset. 1 or denotes higher or lower value is better, and + means that the value closer to real better. We use + represent 95% confidence interval and highlight the best results in bold. For ablation methods (in grey), PT, M, P. S, and SP are abbreviations for pre-training, motion, pose, space, and single-person data, respectively.
R-Precisiont Top
Methods
Top
Acc.f
HDD
MMDist
Div.->
Real
0.511 003
0.682 002
0.776
0.463 000
0.000 000
5.348#.000 002
2.498
Inserformer
0,172 012
0.292 013
0.343 012
0.171 008
400-468-.00 021
7.831 4: 4-11
3.505
MarionLiP'
0.238
u.304 004
U.441
0.180
5.824 0088
6,211
2.615
Intertion
0.326 036
0.4234 063
0.525 033
018 0.297
5.506 257
6.1824: 0088
009
ReGimNett
0.384
0.483#
0.572
3.988 0088
5.867
2.502 001
TTR (Ours)
0.428*t000
0.50g--000
0.698 0.693t.00s
f:00s
0.318
1.942 011
6.648 1648*t.000 0008
2.629 10.20 000
weo Think
0.36722003
0.49122-22 027
0.584
0.230
3.828 122g2.00
6.186 055
2.6092:000
0.398 007
0,531 002
0.628
3.467 113
5,822 003
2.909 003
weo M.M PL.
0. 1.408850000 005
0.563 004
0.646
0.293
2.874 020
6.736
2.353#.000
1.41770004
0.682*-000 004
0.664 7.04 0004
0.308 .008
2.685 + .024
6.609 6.609+.004 +004
2,859 1.59990000 007
who M-T PT:
0.406
0.5572.000
0.6372.004
0.304=-000
2.580
5.8222:.00
2.88922.000
w/o SP Data
0.414 D04
0.592 005
003
0.315 008
2.007 015
5.6872.000 0003
2.611 2: 005
Iinee-X Action
HannsML33
30
T-pose
Standing still
-5
-50
-75
Massage
Being pulled
Figure 3: Visualization of a person's 's motion sequences in Inter-X dataset and HumanML3D dataset.
4.3 ABLATION STUDY OF KEY COMPONENTS
To evaluate the effectiveness of proposed key designs, we conduct detailed ablation studies by removing each of them to observe how much drop compared to the full version of our TTR method. contribution. The results are shown in gray lines of Table Accord-
The larger drop indicates ing to the drops in FID, all designs, including thinking, pre-training tasks and using single person data in pre-training, have positive contributions the final performance, and thinking contributes the most. Some detailed findings and analyses are as follows.
First, we skip thinking stage during inference, and find the performance drops significantly in FID from 1.9 to 3.8. This supports the necessity of our proposed thinking process before reacting. We also notice decreasing diversity of generated samples, as the model relies solely on input action, and cannot explicitly capture and infer action's 's intent, thus leading to more rigid motion in some cases.
Second, to evaluate the effectiveness of pre-training, we omit the pre-training stage, and directly train our model TTR for thinking and reacting tasks. As shown in Table T our model's performance deteriorates without fine-grained pre-training phase from 1.9 to 3.4 in FID. This indicates that pre- training can effectively adapt a language model (Flan-T5-basee) into motion and language model. We further removing three kinds pre-training tasks: motion--otion (M-M PT.). pose-space (P- S PT..) and motion-text (M-T PT.). The results show that the without any task, the performance obviously gets worse, from 1.9 to 2.5 2.8 in FID, indicating their positive contribution to the final performance and complementary values to each other.
8

---

Published as a conference paper at ICLR 2025
(a) Two people stand facing each other. One person approaches and opens her/his arms to embrace the other person's back and waist, while the other person imitates the same action.
(b) The first person pushes the second person heavily the back with both hands, causing hen'him be pushed forward several steps.
(c) The first person runs towards the other and knocks herfis left shoulder against the right shoulder, and the second person is forced step back.
(d) The first person grabs the other person's waist, the second person wrestles with the first person.
Figure 4: Visualized cases of our predicted reactions (in green) to input action (in blue) and corre- sponding thinking results. We also provide failure in figure (d), where TTR misunderstands the input action as "wrestling". which should be "embracing".
Third, to see how much single-person data helps reaction generation, we remove single person motion-text data, i.e., the data from HI33 dataset, from our training set. The result (w/o Data) shows that the model performs worse without training HumanMI.ID, which proves that our unified motion encoder and motion-language architecture can leverage both single- and multi- person data, alleviating the insufficiency of training data. However, the benefit from single-person data not as large as we expect.
4.4 ANALYSIS ON OVERLAPPING BETWEEN SINGLE- AND MULTI-PERSON MOTIONS
To investigate the reason small contribution from single-person data, we further visualize mo- tion sequences of single-person motion (HumanML3D), two-person action (Inter-X Action) and reaction (Inter-X Reaction) in the same space, as presented in Figure 3 Specifically, we use t-SNE tool Van der Maaten & Hinton 2008) to project motion token sequence features into two-dimension. As shown in Figure 3 the single- and two-person motion sequences have little overlap. When doing case studies, will never be used single-person motion. Similarly, most single-person motions are unique too, e.g., T-pose, and seldom appear in multi-person interaction. There are only few overlapped mo- tions, e.g., standing still. In addition, when comparing action and reaction sequences in multi-person interaction, we have some interesting findings. When reactions are close actions, the motion usu- ally belongs to symmetrical interactions, e.g., pulling or being pulled; whereas, when actions are far from reactions, the motion usually belongs to asymmetrical interaction, e.g., massage.
find that most two-person motion unique, e.g., massage and being pulled, and
4.5 IMPACT OF DOWN-SAMPLING PARAMETER IN MATCHING MODEL FOR EVALUATION
As described Section 4.1 we propose downsampling action motion sequence avoid matching models for evaluation pay too much attention to input action rather than output reaction. We con- duct an experiment change the downsampling parameter frame rate and calculate the difference between taking ground-truth action and random action as the input of M. in terms of summed rank- ing scores (Top-1, Top-2, Top-3 and Acc.). As presented in Figure 5 difference is lowest when FPS equals to O, which meaning only match generated reaction motion with text. It goes up to the peak when FPS equals and quickly goes down to low values, even close to the lowest when FPS about 15. This indicates that it is necessary to concatenate input action with generated reaction to compose meaningful interaction evaluation, otherwise the motion-text matching model cannot well recognize the interaction. However, only FPS is enough. With larger FPS, the matching

---

Published as a conference paper at ICLR 2025
1.6
3.3
1.60
2 3.0
1gg
1.52
1,8
Treeer
800%
1.4
201 Impar Action FPS
20
90
50
Ro-hink Interval
Figure 5: Impact of input ac- tion FPS summed ranking score differences.
Figure 6: Impact of re-thinking in- terval to FID and average inference time per step (AITS).
Figure 7: User preference be- tween TTR and ReGenNet on different motion duration.
models will be disturbed by imput action rather than the generated reaction. Thus, we choose FPS, corresponding to the largest difference, as our final setting.
4.6 IMPACT OF RE-THINKING INTERVAL
We change the re-thinking interval N, from about to 100 timesteps (about 0.1 to 10 seconds) and observe how it impacts generative quality measure FID. As shown in Figure 6 FID falls down first until N2- 4 (about 0.5 second) and then continues rising up. This indicate that the best time interval is about 0.5 second. When the time interval is too short, our TTR model cannot get enough information to re-think what the input action means and will bring some randomness into predicting appropriate reaction. When the time interval gets too long, our TTR model give slow responses to the input action sequences and generates coarse-graines reaction.
We also evaluate the average inference time per step (AITS) with respect to the re-thinking interval. As shown in Figure(6 the inference time significantly decreases as the re-thinking interval increases, eventually converging to approximately 10 milliseconds per step (100 FPS). our setup, we opt to re-think every four steps, resulting in inference time of less than 50 milliseconds, which meets the requirements for a real-time system.
4.7
USER STUDY
To further evaluate our model qualitatively, we conduct a user study on TTR vs. the latest SOTA method ReGenNet, and the results are shown in Figure We randomly sample 100 action sequences from Inter-X dataset, which are fed into TTR and ReGenNet to predict reactions, and ask four real human to choose the better ones. It can be seen that TTR surpasses ReGenNet on all the duration range, and the winning rate rises significantly when motion duration is longer. mainly contribute this to our explicit thinking and re-thinking procedure, which ensures semantics matching and alleviates accumulated errors.
CONCLUSION
In this paper, we propose novel framework Think-Then-React (TTR) to address the action-to- reaction motion generation problem. First, we propose unified motion encoder that tokenizes person's 's starting location and following poses separately. Then we design motion and text related tasks to pre-train a large language model backbone understand and generate both language and motion. We also fine-tune the model to think what the input action means and what an appropriate reaction is, and then generate reaction motions. Experimental results show that our proposed TTR method outperforms all basclines in all metrics except for diversity, Our proposed thinking phase and all pre-training tasks contribute to the best performance. We find that although our proposed unified motion encoder enable leveraging kingle-pesson data in pre-training, it brings limited benefit due to the little overlapped poses between single-perser motion and multi-person interaction. In the future, plan to explore more effective method for
and multi-person dataset.
Acknowledgment This work is supported by the National Natural Science Foundation of China (No. 62276268) and Kuaishou Technology.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Baptiste Chopin, Hao Tang, Naima Otberdout, Mohamed Daoudi, and Nicu Sebe. Interaction trans- former for human reaction generation. IEEE Transactions on Multimedia, 25:8842 8854, 2023.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. Scaling catructioo-finetune lan- guage models. Journal of Machine Learning Research, 25(70):1-53, 2024.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farke Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. Action2motion: Conditioned generation of 3d human motions. In Proceedings the 28th ACM International Conference on Multimedia, pp. 2021-222220 2021-2029, 2020.
Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng, Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5152-5161, 2022a.
Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng. Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts. In European Conference on Computer Vision, pp. 580-597. Springer, 2022b.
Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. Momask: Gener- ative masked modeling of 3d human motions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1900-1910, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851. 2020.
J Stuart Hunter. The exponentially weighted moving average. Journal of quality technology, 18(4): 203-210, 1986.
Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as a foreign language. Advances in Neural Information Processing Systems, 36:20067-20079, 2023.
Diederik Kingma. Auto-encoding variational bayes. arXiv preprint arXiv: 2013.
Han Liang, Wengian Zhang, Wenxuan Li, Jingyi Yu, and Lan Xu. Intergen: Diffusion-based multi- human motion generation under complex interactions. International Journal of Computer Vision, pp. 1-21, 2024.
Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling- Yu Duan, and Alex C. Kot. NTU RGB+D 120: A Lcage-Scale Benchmark for 3D Human Activity Understanding. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 42(10))2684-277) 2701, October 2020.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia), 24(6))248:12248 16, October 2015.
Qianhui Men, Hubert PH Shum. Edmond SL Ho, and Howard Leung. Gan-based reactive motion synthesis with class- aaase discriminators for human human interaction. Computers & Graphics, 102:634-644 645, 2022.
11

---

Published as a conference paper at ICLR 2025
Mathis Petrovich, Michael J Black, and Güi Varol. Action-conii conditiond 3d human motion synthesis with transformer vac. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10985-10995, 2021a.
Mathis Petrovich, Michael J Black, and Gil Varol. Action-conditioned 3d human motion synthesis with transformer vae. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10985-10995, 2021b.
Matthias Plappert, Christian Mandery, and Tamim Asfour. The kit motion-language dataset. Big data, (4))236-252, 2016.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, (1(110))1-611 2020.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine
learning research, 9(11), 2008
Liang Xu, Ziyang Song. Dongliang Wang. Jing Su, Zhicheng Fang. Chenjing Ding, Weihao Gan, Yichao Yan, Xin Jin, Xiaokang Yang, et al. Actformer: A gan-based transformer towards general action-conditioned 3d human motion generation. Proceedings of the IEFF/CVF International Conference on Computer Vision, pp.
2023.
Liang Xu, Shaoyang Hua, Zili Lin, Yifan Liu, Feipeng Ma, Yichao Yan, Xin Jin, Xiaokang Yang, and Wenjun Zeng. Motionbank: A scage-scale video motion benchmark with disentangled rule- based annotations. arXiv preprint arXiv:2410. 22444..37990 /3790, 2024a.
Liang Xu, Yizhou Zhou, Yichao Yan, Xin Jin, Wenhan Zhu, Fengyun Rao, Xiaokang Yang, and Wenjun Zeng. Regennet: Towards human action-reaction reaction synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1759-1769, 2024b.
Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xi Shen, and Ying Shan. Generating human motion from textual descriptions with discrete representations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14730-14740, 2023.
Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208. 15001, 2022.
Yaqi Zhang, Di Huang, Bin Liu, Shixiang Tang, Yan Lu, Lu Chen, Lei Bai, Qi Chu, Nenghai Yu, and Wanli Ouyang. Motiongpt: Finetuned ilms are general-purpose motion generators. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 7368-7376, 2024.
12

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 MOTION REPRESENTATION AND POSE TOKENIZER
For motion representation, we use the same strategy a Liang et al. (2024), which combines local joint positions, rotations, velocities, and foot-ground contact the feature of human motion. Re- garding the tokenizers, we adopt a temporal down-sample rate of four and N, - 256 for motion tokens, each motion token in dp 512 in the codebook. We divide all space tokens into N₂ 10 bins. The motion VQ- VAE is trained for 150K steps with batch size set to 256 and learning rate fixed le-4 on single Tesla V100 GPU. We adopt a similar architecture to Guo et al. (2022a) as our pose tokenizer. The encoder/decoder consists of two dwwn-sample/up-sanne sample ID convolution layers and three ID ResNet blocks He et al. (20T6). We set the width of the auto-encoder to 512. We train the model on both the Inter-X and HumanML3D datasets for 200,000 steps, with batch size set to 256, and learning rate set to le-4, We apply L1-loss on both pose feature and velocity reconstruc- tion, and commitment loss for the embedding process. The weight set to velocity loss is 0.5 and commitment loss 0.02.
A.2 MATCHING MODEL
For the motion-text matching model, we adopt a similar architecture to InterCLIP (Liang et al., 2024). which consists an eight-layer motion transformer encoder and an eight-layer text trans- former encoder. The hidden size is set to 768 and attention heads is set to 8. We add a learnable token the motion encoder and extract its feature in the last layer of motion encoder as the pooled motion feature. To perform motion classification, we add classification head (an MLP) after the pooled motion feature. We use the text embedding layer from saop--it-arrg-aattee 14 (Radford al.] 2021), which is frozen during training. We train the model for 40 epochs with batch size set to 128. The learning rate warmed-up to 0.001 in the first 1,000 steps.
A.3 EVALUATION ON MOTION CAPTIONING TASK
Table 2: Motion captioning results on Inter-X dataset. TTR* denotes feeding both action and reac- tion motion into TTR for captioning. TTR (x%) denotes only the first 2% of action motion is fed into TTR for captioning.
RAEs
SeqGan
Seq2Seq
TM2T
TTR*
TTR
TTR (50%)
TTR (25%)
Bleu-1
28.6
45.4
53.8
56.8
60.2
55.6
54.1
52.2
Bleu-4
9.7
14.1
18.5
21.6
25.4
20.3
18.9
16.6
Rouge
34.1
36.8
45.2
48.2
50.5
46.4
45.3
43.0
We also evaluate our TTR model on motion captioning task, and the results are shown Figure The results of baselines are from -nter-X paper Section A.1. As the baseline methods all take both action and reaction as input, while in our setting, our thinking process is only accessible to ground- truth action, we first align TTR's setting with the baselines'. denoted as TTR". can be that, with our fine-grained training and effective motion representation, TTR" achieves the best captioning performance in all metrics.
Then we evaluate TTR on real-world settings, i.c., only partial of the input action is visible to our model. We take the first 25%. 50% and full action as input of TTR for the action-to-text generation process. It can be seen that even though only quarter of input action is given, TTR is still capa- ble of accurately predicting the corresponding action and reaction description, showcasing strong generalization capability.
A.4 ABLATION STUDY ON THINKING PROCESS
To evaluate the necessity of the Thinking process, conduct ablation study different prompts provided to the Reacting process. First we fed ground-truth prompt to the Thinking process, and can be seen that the overall quality of predicted reaction is significantly improved. Then
13

---

Published as a conference paper at ICLR 2025
Table 3: Ablation study on how does thinking process influence model performance. GT denotes
ground-truth, and Thinking" denotes using a better motion-to-text model for the thinking process.
Methods
FID
Top-1
Acc.
w/ GT Prompt
1.584=.016
0.458#.005 0005
0.361 2000
w/ Thinking"
1.882 014
0,429 004
0.331 4.003
w/ Thinking
1.942
0.423 005
0.318-.0009
w/o Thinking
3.828
0.367 003
0.230
leverage enhanced Thinking model as mentioned in Section A.3] and the FID decreases from 1.94 to 1.88, proving that better thinking process leads could promote the following Reacting process. Moreover, when discarding the Thinking process, our model dramatically deteriorates in reaction generation quality, as Thinking and re- re-thinking process is crucial to guide reaction generation and reduce accumulated errors.
14


Reviews:
[{"review_id": "BFGtgy5HI0", "reviewer": "Reviewer_X2o2", "summary": "The submission handled the problem of human reaction generation. Specifically, the submission proposed to utilize a unified motion token representation encompassing both egocentric and global spaces. Then, an LLM-based online reaction generation framework TTR is proposed to leverage textual information for training. TTR involves action intention inference as the first step and precise reaction token prediction as the second step. Impressive quantitative improvements are shown.", "questions": "- In L219, it is unclear how a pair of persons should face the positive z-axis. It is more understandable that the reactee faces the positive z-axis.\n\n- In L221, are (x,z) corresponding to the pelvis 2D coordinates? \n\n- Terms in L231 are inconsistent, making it hard to understand.\n\n- Why the evaluation model from Inter-X is not adopted, which has a high ACC. over 90% compared to the 46.3% of the adopted model?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Explicit decomposition of egocentric poses and global transformations is reasonable, especially considering human reactions.\n\n- Introducing explicit intention modeling as an intermediate stage for reaction generation has been proven helpful.", "weaknesses": "- The overall writing clarity could be improved. Please refer to Questions.\n\n- The evaluation settings could be ambiguous. Please refer to Questions.\n\n- The effectiveness of the proposed unified motion tokenizer is not specifically discussed in experiments. Is it feasible to treat TTR w/o ALL P.T. as MotionGPT w/ unified representation? To understand this, details on how MotionGPT is adapted to multi-person scenarios should be provided.\n\n- Despite the considerable quantitative improvements, the improved quantitative metrics are essentially indirect proxies of the real reaction generalization performance. A qualitative comparison of how the proposed method outperforms previous SOTAs is still expected. Animated motions would be a better form of visualization.\n\n- Moreover, a failure case study is also expected if possible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pGd1Snc9uC", "reviewer": "Reviewer_De2Y", "summary": "The author trains an online complicated framework to generating reaction motion based on the given action motions.  Firstly, the author introduces a unified space-pose representation that effectively converts both absolute space and egocentric pose features into discrete tokens, facilitating LLM training. In addition, the authors introduce the think-then-react framework to get better results.", "questions": "1. Could the author provide more video visualization to show the results? It's tough for me to see the generation results from images.\n2. Could the author explain further why the thinking part works? i.e. why we need the text as a bridge to connect the input action and output action? From my general understanding, the translation from input action to text has already introduced the information loss. \n3. Did 1 FPS and 0.5-second time interval really work? This may lead to heavy leggy and jitter. Can 0.5 seconds really work for an online system?\n4. Since the performance depends on the text generation process, could the author try some motion caption benchmarks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.  The figure and render visualization is good. \n2. The authors introduce a unified space-pose representation that effectively converts both absolute space and egocentric pose features into discrete tokens, facilitating LLM training.", "weaknesses": "1. Poor writing. This paper is hard to follow. It's hard for me to distinguish the differences between special words, like “motion”, \"pose\", \"action\", \"action motion\", etc. \n2. The training process is complicated. Besides the pertaining and fine-tuning, to improve the training stability, the author also needs to check the validation loss manually and switch the training source.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Jd39i5BXK4", "reviewer": "Reviewer_enFt", "summary": "The paper presents Think-Then-React, a novel method for action-to-reaction motion generation. It introduces a unified motion encoder that tokenizes the initial position and subsequent poses. Besides it employs a large language model for understanding and generating both motion and text. The model is pre-trained on various tasks, therefore it enhances its performance in generating coherent reactions based on observed actions.", "questions": "- I am confused by the dataset used for training. Do other methods also use mixed dataset? If they only use one dataset but you use the mixture of two, it will cause unfairness. Please provide the information about the dataset used by other methods in the paper and the details about dataset mixture strategies.\n\n- Does your paper include a comparison involving evaluations of the generated actions by real humans (i.e., a user study)? In the generation field, the importance of user study results is comparable to that of numerical metrics.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper aims at improve the quality of action-to-reaction generation, which is an important issue.\n- The paper introduces a unified space-pose token representation and proves its effectiveness.\n- The significant improvement in performance compared to the baseline is a major advantage.", "weaknesses": "- The training data utilized a mixture of two datasets, which may lead to an unfair comparison.\n- The experiments only presented numerical metrics on the test set and did not include evaluations of the generated actions by real humans.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9xO2fAO5xU", "reviewer": "Reviewer_wKay", "summary": "The paper present a new method for generating a reaction motion based on an action motion in a two persons interaction. The proposed architecture uses LLMs to understand the input action motion and then infer possible reactions to this action. The action motion is first converted into tokens that are used by the LLM to infer the type of action and predicts the reaction and the corresponding token that are then projected back into motion space. The method uses several training schemes to achieve a better understanding of the action and generate a better reaction. The method outperform the state of the art quantitatively on the Inter-X dataset.", "questions": "- Qualitative comparison are very important in these type of works. Including them is always a good idea as they help in showing the performance of the method. Also since the work is on motion video qualitative results are always appreciated.\n- Typo line 231 : \"number of frames or original motion\"", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of splitting the task of reaction generation between a \"thinking\" process using LLM and a \"reacting\" process using VQ-VAE is very interesting. This method will probably be able to scale well as new human interaction datasets are released.\n- The architecture and training scheme as well as the intuitions behind them are sound.\n- Treating the global pose with a separate tokenizer is a good idea and allow for more flexibility while staying accurate.\n- \"re-thinking\" is also interesting to correct the generation in an online manner.\n- Method beats the state of the art by a good margin on most metrics.\n- Extensive ablation on the pretraining.", "weaknesses": "- Very few qualitative results. There is no qualitative comparison against the state of the art and no videos to show the actual motions. the lack of qualitative results makes it very difficult to evaluate the model qualitatively and to see if quantitative improvements correspond to similar qualitative improvements.\n- The \"pose tokenizer\" described in the main paper (3.2.1) is different from the one described in the appendix. Is the tokenizer from the appendix the \"space tokenizer\" ? This should be clarified.\n- Sec 3.3.1 (3) \"we take the first half of action and the second half of reaction sequence of tokens as input\". Is the opposite operation also performed ? i.e. \"take the SECOND half of action and the FIRST half of reaction sequence of tokens as input\" ? The text says \"and vice versa\" but it is not clear what it refers to.\n- The authors do not explain why they use only the 2D positions for the \"absolute pose\" and not the 3D positions.\n- There are some issues with notations:\n    - line 204 \"k\" is never defined and why take \"k-l\" as input ?\n    - line 231 \"M\" is never defined.\n    - line 231 why change the notation from \"Nf\" to \"L\" for the number of frame ?\n    - line 266 why use \"t\" for timestep when it was defined as \"l\" (or maybe \"k\") on line 204 ?\n- Some of the writing could use improvement:\n    - lines 203-206 : phrasing is very confusing and makes it difficult to understand what the authors want to say.\n    - lines 266-268 : same as above\n    - The way \"pose\" and \"motion\" have been defined and are then used is sometime confusing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)