Social Motion Prediction with Cognitive Hierarchies
Authors: Wentao Zhu
Jason Qin
Yuke Lou
Hang Ye
Xiaoxuan Ma
Hai Ci
Yizhou Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: multi-person motion prediction
TL;DR: We present a new dataset, a novel formulation, and a cognition-inspired framework for the social motion prediction problem.

Abstract: Humans exhibit a remarkable capacity for anticipating the actions of others and planning their own actions accordingly. In this study, we strive to replicate this ability by addressing the social motion prediction problem. We introduce a new benchmark, a novel formulation, and a cognition-inspired framework. We present Wusi, a 3D multi-person motion dataset under the context of team sports, which features intense and strategic human interactions and diverse pose distributions. By reformulating the problem from a multi-agent reinforcement learning perspective, we incorporate behavioral cloning and generative adversarial imitation learning to boost learning efficiency and generalization. Furthermore, we take into account the cognitive aspects of the human social action planning process and develop a cognitive hierarchy framework to predict strategic human social interactions. We conduct comprehensive experiments to validate the effectiveness of our proposed dataset and approach.

Social Motion Prediction with Cognitive Hierarchies
Wentao Zhu 1- Jason Qin 1+ Yuke Lou'
Hang Ye
Xiaoxuan Ma Hai Ci
Yizhou Wang 1,2
Center on Frontiers of Computing Studies, School of Computer Science, Peking University Institute for Artificial Intelligence, Peking University
Abstract
Humans exhibit remarkable capacity for anticipating the actions of others and planning their own actions accordingly. In this study, we strive to replicate this ability by addressing the social motion prediction problem. We introduce a new benchmark, novel formulation, and a cognition- eegnitioninnspree framework. We present Wusi, a 3D multi-person motion dataset under the context of team sports, which features intense and strategic human interactions and diverse pose distributions. By reformulating problem from multi-agent reinforcement learning perspective, we incorporate behavioral cloning and generative adversarial imitation learning boost learning efficiency and generalization. Furthermore, we take into account the cognitive aspects of the human social action planning process and develop cognitive hierarchy framework to predict strategic human social interactions. We conduct comprehensive experiments validate the effectiveness of our proposed dataset and approach.
1 Introduction
Human beings are inherently social creatures. Concretely, individuals unconsciously anticipate the actions of others and make informed decisions about their own behaviors in social contexts [13] [1]] 154, which enables individuals to cooperate and compete with others in a variety of scenarios, from team sports [64 50] Notably, task experts demonstrate exceptional skill
pedestrian traffic (5 44
in predicting others' movements in advance 37
To better understand and replicate this ability, the research community has proposed the task of future prediction for multiple interacting agents given their historical observations. The majority of prior work in this area focuses on modeling and predicting agent interactions at the trajectory
level 42 39 52 5 34 58
75
59 43 72 and has demonstrated promising results applications 49
17 36 16 27). However, trajectory-baat approaches can only reflect coarse- geariee interactions (e.g., collision avoidance, social distancing) and fail to capture the rich and fine- -ee--graind human actions. To this end, some studies have investigated the multi- person motion prediction problem, which aims to forecast both trajectories and poses for a group 73. Despite recent advancements, there remain several critical challenges this field. Firstly, existing multi-person motion datasets are primarily designed for human pose estimation tasks 665 | B1], and consequently, do not place particular emphasis human interactions. Individuals in these datasets tend to move casually and interact with others at random, making future predictions both difficult and less meaningful. Secondly, the majority of prior methods concentrate on developing neural network architectures for end-to-end supervised training while overlooking the cognitive aspects of human social action planning. These two challenges are closely related and necessitate a comprehensive solution.
such autonomous driving [38
of people 2 3 67
68
22
63
48
'Equal Contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Offense
Defense
Pass
Steal
Fake Move
Figure I: Example sequences from Wusi dataset. Three players in purple form the offensive team, and players in orange form the defensive team. Left: a successful pass for the offensive team. Middle: successful steal for the defensive team. Right: a successful pass following fake pass.
In this work, we propose novel perspectives this problem to address the aforementioned limitations. We begin by constructing a large-scale multi- pertii 3D motion dataset featuring intense and strategic interactions among participants. To achieve this, we turn team sports, which offer several inherent advantages: 1) Well-defined game rules and global rewards implicitly constrain and guide individual actions. 2) Participants develop intricate interaction strategies based on their roles, such as planning cooperative actions with teammates while acting adversarially towards opponents. Skilled players employ sophisticated techniques like deception, etc. 3) Human motions exhibit greater dynamism in terms of pose diversity and motion intensity, making motion prediction more challenging than in previous datasets.
Additionally, we present a new formulation of the multi-person motion prediction task as multi- agent reinforcement learning (MARL) problem. Specifically, we model the task using imitation learning, where the objective is to learn a policy from expert demonstrations. We employ behavioral cloning (BC) 7 51 imitate expert behaviors from the dataset. To improve learning efficiency and generalization, we utilize generative adversarial imitation learning (GAIL) [26), aiming to render propose framework for
agents' policy indistinguishable from experts' policy. Furthermore, human social interactions based on the cognitive hicrarchy theory [10]. In particular, we assume that people base their decisions on their predictions regarding the likely actions of others, while others engage in similar dacision-making processes from their perspectives. By considering the reasoning steps recursively, where & represents the depth of strategic thought. Building on this insight, we develop a computation model that can be elegantly integrated with the MARL formulation.
posit that level l agent takes actions based the level agent actions,
summarize our contributions as follows: 1) We present Wusi, the first large-scale multi-human 3D motion dataset featuring intense and strategic interactions. We demonstrate that our dataset has greater motion diversity than existing datasets and poses a more significant challenge for the social motion prediction problem. 2) We propose novel MARL formulation for the problem and develop an imitation learning baseline that combines behavioral cloning and generative adversarial imitation learning. 3) We introduce cognitive hierarchy framework to model the strategic and came-theo human social interactions. Our approach outperforms the state-of-the-an methods in challenging long-term social motion predictions.
2
Related work
2.1 Multi-agent trajectory prediction
The multi multi-agent trajectory prediction problem has been extensively studied, especially for interact- ing traffic participants, e.g., pedestrians and vehicles. There exist multiple industrial benchmarks including Argoverse [12], nuScenes [ Waymo Open Motion Dataset [18]. Meanwhile, various generation methods have been explored to model the interaction among different agents and predict
their future trajectories 42 39
75 59 4 72 In this work, we aim to predict intense and strategic interactions for group of humans with inc-graiied body motions.

---

2.2 Multi-person motion dataset
The existing multi-human motion datasets can be categorized into 2D and 3D. The Pose Track dataset [ [29] offers video sequences with manually annotated 2D keypoints. However, 2D keypoints fail to accurately represent the proximity of individuals in the real world, and manual annotation difficult to scale. Mcanwhile, 3D mulli-pperoo motion datasets can be effectively constructed with mo- tion capture (mocap) systems, including marker- based solutions and markerless ones. 3DPW [65] uses the inertial measurement units (IMUs) to obtain high-quality motion reconstructions. UMPM [6]] employs a set of reflective markers to identify the joint positions. However, wearable sensors may struggle capture complex movements and become expensive the number of individuals increases. Other datasets, e.g., CMU-Mocap [. Panoptic [1. MuPoTs-3D [41] ExPI [22), employ markerless mocap systems with multi-view cameras address the occlusion problem and obtain 3D human motion through triangulation. In this work, we construct a multi-person motion dataset featuring high motion diversity and intense social interactions using multi-view cameras. We provide a comparison with previous datasets in Section 3.3
2.3 Multi-person motion prediction
Joo el al. [32] propose to predict human motion conditioned on other individuals' kinesic signals in triadic haggling scenario. Adeli al. [2] introduce the 2D social motion forecasting (SoMoF) benchmark, which aims to predict multi-person trajectory and pose on the PoseTraak dataset [6]. They also propose a baseline model using shared GRU encoder and pooling layer incorporate social clues. TRiPOD 3 employs attention graphs to characterize the spatiotemporal social interactions. It also considers the joint occlusion and body invisibility issues arising from 2D observations. Futuremotion [6] provides a simple baseline using no social context, yet achieving competitive performance on SoMoF. As 2D multi-person motion prediction faces challenges such as data scarcity, depth ambiguity, and occlusions, more recent research has shifted towards exploring the multi-person motion prediction problem in 3D. Wang el al. (68] propose a multi-range Transformer separately encode the local and global motion history. Guo et al. [22] design a cross- interiacioon
(XIA) module model close interactions between pairs in duo dance scenarios. Vendrow er al. [63] introduce joint-aware attention and joint-wise query to predict the entire future sequence without recurrence. Peng et al. [88] present a social-aware motion attention mechanism that models both inter- and intra-individur motion relations. DuMMF (73) frames the problem as dual-level generative task and instantiates it with various generative models. In contrast to previous work, we adopt a MARL perspective and model the strategic social interaction process using cognitive hierarchies.
2.4 Cognitive hierarchy theory
Cognitive hierarchy theory (CHT) is a model in behavioral economics and game theory that aims describe human decision processes in strategic games. Several foundational works introduce the concept of CHT [10,, and provide experimental evidence [45 14, 23
57 56 supporting the existence of cognitive hierarchies in human decision-making processes. Taking it step further, Wright et al. [71) develop Poisson cognitive hierarchy model to predict human behavior in normal-foo games. Li et al. [35] summarize how the information structure guided by the cognitive hicrarchy supports belief generation and policy generation in thee-the multi-agent learning. In this work, we propose a cognitive hierarchy framework in conjunction with MARL, and demonstrate its effectiveness in real-world social motion prediction scenario.
3 Dataset
3.1 Overview
We present a multi-person 3D motion dataset with a special focus on strategic interactions called Wusi (Wusi Basketball Training Dataset). In the following, first introduce the contents of the dataset, then outline the data collection pipeline, and finally provide statistical analysis and comparison with regard to existing datasets.
Our dataset captures the no-dribble-3-on-2 baskethall drills performed by team of professional basketball athletes. In each drill, three offensive players possess the ball, while another two players
3

---

Table 1: Dataset comparison. We compare our dataset with existing multi-person motion datasets employed by previous works on the multi-perse person motion prediction task. denotes multi person subset as utilized previous works
Dataset
2D/3D
Frames
Duration (min)
No. of people Interaction
PoseTrrack
Multiple
Weak
CMU-Mocap
3D
34K
4.9
2
Weak
3DPW
65
3D
5K
6.1
2
Weak
MuPoTs-3D
3D
8K
4.4
2-3
Weak
ExPI
20
3D
30K
20.0
2
Cooperative
Wusi
3D
60K
40.3
5
Strategic
are on defense. The offensive team aims to accomplish as many successful passes as possible within the given time, while the defensive team strives to get steals, deflections. and slow down the offense. Since dribbling is prohibited, the drill requires the offensive players to make better passing decisions and foresee the defense before they pass; conversely, the defensive players need anticipate the passing directions order to steal the ball. Figure showcases the diverse and dynamic human interactions present in dataset. On the left, the offensive player successfully passes the ball to a teammate. In the middle, the defensive player accurately anticipates the passing trajectory and successfully steals the ball. On the right, the offensive player pretends pass the ball to the right, then swiftly executes a pass to the left.
3.2
Data collection
Our motion capture system consists of 11 synchronized and calibrated wide-baseline cameras. Our setup ensures that the players are sell-surro by the cameras from elevated shooting angles, enabling each body joint to be covered by at least four different camera views. We utilize markerless multi-person 3D pose estimation algorithm 774 with JDETracker [69]. Subsequently, we apply post-processing pipeline that includes automatic failure detection and temporal filtering [1] A final
manual check is conducted to guarantee the integrity the data sequences.
3.3 Data analysis
In this section, we examine the statistics of Wusi dataset and compare it with existing datasets for multi-person motion prediction. As outlined in Table I current datasets exhibit two primary limitations for the multi-person motion prediction task. Firstly, high-quality 3D motion data is limited in terms of scale (duration and the number of people). Consequently, prior works 68
48
63
73
resort to randomly mixing the motion sequences 1 65 41 order to generate more multi-person motion sequences for training. However, such practice inevitably causes unnatural human interactions. Secondly, the interaction strength in existing datasets restricted, mainly featuring simple actions such walking together 66] [ 65 41]. More recently, the ExPI dataset [22] introduces cooperative 3D motions of two Lindy-hop dancers. In contrast, our dataset offers 2 to 9 times the video duration of existing datasets and includes strategic global interactions among individuals.
Table 2: Comparison of pose diversity based on different thresholds.
Threshold
50mm
100mm
Human3.6M
28
24%
12%
ExPI P
52%
23%
CMU-Mocap
|I
20%
9%
MuPoTs-3D
41
37%
19%
Wusi
53%
27%

---

CMLRMROND
M.PuT=-3D
68
62
Rvet
Lifles
Let
Righing
Right Amnn
bet
Cieet
Haad
Lohinad
Righ obvuttr Rilr-low
Rejerwwn
Figure 2: Comparison of motion intensity.
Additionally, we provide a quantitative comparison with existing datasets, emphasizing two critical aspects: pose diversity and molion intensity. Firstly, we examine the pose diversity within the datasets
by calculating the ratio of unique poses to the total number of poses following [28] (22]. Table 2 shows that our dataset surpasses the previous datasets in terms of overall pose diversity. We further analyze the motion intensity by computing the average velocity for all the body joints. As illustrated in Table our dataset exhibits more dynamic movements across all body joints. We suppose that these characteristics make Wusi not only challenging for motion prediction, but also potentially useful for other related tasks, e.g. motion generation [76].
4 Method
4.1 Formulation
The multi-person motion prediction problem can be formulated as follows: given the motion histories spP of length from human subjects, predict their future motion {- frishtrtt PSPSP of length T", where each z represents 3D human pose.
We model the problem using Markov Decision Process (MDP) [60] by dividing the prediction sequence into steps with even step length m I following (66), and further extend the formulation to multiple agents. We use bold symbols to represent the collective variables of all agents following the convention MARL. At each step i. The state 8; is defined as the aggregated motion history { I 1 for all the agents. The action for the p-th agent a" is defined as sequence of velocities (o ()(st)))mmSSSTiimmm.m where uf x x.. Therefore, the joint action all P agents a; deterministically transition the MDP into the new state 8,-1 according to:
x... = 2 -T
1+0--)MM
of, T (i- m ST m. SSPPP. (1)
The goal of this objective is to learn policy π² for each agent that maps each possible state to agent action a a² Ts)).. In the real world, people optimize their ~² to maximize an implicit team reward r. Since we have no access to or the environment, we aim to learn T² from expert demonstrations D via imitation learning.
We parameterize nP with a set of parameters 0. In practice, we follow 68 to employ a global-range Transformer Fg encode global state feature 8g F.g (s), and local-range Transformer E, to encode local state feature sf = (x))) for each agent See Figure 3 for an overview. This approach helps extract multi-level state features and enables all agents share policy network parameters.
4.2 Behavioral Cloning
A straightforward method in imitation learning is Behavioral Cloning (BC) 7 51, i.e., using expert demonstrations minimize the difference between the actions produced by the policy and those taken by the experts:
argming Σ
E (TT
(+),
(s)).
(2)

---

w-th agant cb
Local Eiioder y
Policy
m0
Polkyy Mal
%
9x-11
Policy
100
4000
Clobal Encodar 5
(0-1
nction lnve--O
joint action Invel-
Figure 3: Framework overview. For the p-th agent, two state encoders first extract its local and global state features of and Sy, from which the level-0 policy network produces an initial action ooo-- The level-k policy network produces action d(k) based on sy and the joint actions of the previous level a(k-1) (k 1).
where is sampled from expert demonstrations D, no and Hº are policies of the model and the expert for agent p. is distance function defined in the action space. While BC enjoys the benefit of being both computation-effiiien and sample-efficient [53], it also faces several drawbacks. For instance, policies tend be overfitted with respect the state distribution encountered by the experts, leading to suboptimal generalization ability [66 30 and cognitive hierachy reasoning.
To cope with this challenge, we further introduce GAIL
4.3 Generative Adversarial Imitation Learning
Generative Adversarial Imitation Learning (GAIL) (26] is model-free imitation learning approach. It leverages the adversarial training framework of Generative Adversarial Networks (GAN) 21
and the concept of inferring reward functions from Inverse Reinforcement Learning (IRL) 46
Direct estimation of reward signals from expert demonstrations can be very difficult; therefore, GAIL. transforms the IRL problem into its equivalent dual problem of occupancy measure matching. Specifically, the policy network is regularized to match its state-action pair distribution to that of the experts' policy through adversarial training. We implement GAII. using global discriminator D parameterized by which differentiates the state-action pairs produced by agents' joint policy π and experts' joint policy N. The optimization objective can be formulated as:
min max Ex, [log D., (s, Te (s)) + E+ [log (1 D. (5, (3))))
(3)
where and å sampled from expert demonstrations 'D independently.
4.4 Cognitive hierarchies
Furthermore, notice that in real-world situations, individuals would predict others' behaviors by forming beliefs about others' policies and then acting accordingly to maximize their own payoffs (20 9. This can be formulated as hierarchical, rame-theorette decision-maie -mkiing process, in which lower-level agents adopt straightforward strategies, while higher- players anticipate the strategies of lower-level agents and respond accordingly. We explicitly model this recursive reasoning process to learn more interpretable and robust agent policies. We adopt a specific type of cognitive hierarchy model [T0] called level-k thinking [57] 45 [5] to represent this process. Specifically, agents each level (except for the lowest one) assume that others are reasoning at the previous level. As shown in Figure straightforward policy is to take actions based on local and global state features sf and 8g. which we denote as level-0 policy (o)- Level-0 actions are thus defined as:
0 = To (s) 6roy 8z).
(4)
where t2(0) is the level-0 policy network. Then for level-k agent (A: 2 1), it takes actions based on the joint agent actions a(k-1) the previous level and its local state features sP:

---

dxx)
a(k-))), (() 1
(5)
where (1)) is the level-k policy network, K' is the maximum strategic depth.
4.5 Training objectives
Finally, we introduce our training objectives which effectively integrate the aforementioned compo- nents. For level-k (1 S K) agents, we apply GAIL to regularize the distance between their joint policy and the experts' policy:
K CCAIL Σ
Ex(a
[log D (s, maa (+))] + E+ [log (1 - D (5, (8)))
(6)
In addition, we employ BC on level-K agent policies:
p CBC >> D pm1
(T(K)
(8),
We train the discriminator D and the policy networks & alternatielly, where D aims maximize CGAIL. and 6 is optimized to minimize a linear combination of LBc and LGAIL using a policy gradient algorithm following [30]:
L LBC + ALGAIL:
where À is constant for balancing the loss terms.
Experiments
5.1 Setup
Implementation details. We implement the presented framework to train and test the proposed Wusi dataset. We employ Transformer encoder (62) for both the local and global state encoders, well Transformer decoders for the policy networks. Each Transformer consists of 3 layers with 8 attention heads. We share parameters for policy networks Φ(1)- ... ((K)- We set the strategic reasoning depth K 3 unless otherwise stated. For all the methods, we provide Is motion history and predict future 1s motion. For additional experimental details, please refer to the appendix.
Evaluation metrics. We compute the Mean Per Joint Position Error (MPJPE) between the predicted future motion and ground prediction, further calculate the mean root position error and mean local pose error (MPJPE after root alignment). We report all the errors in millimeters.
In order to disentangle the results of trajectory prediction and pose
5.2 Evaluation and comparison
In this section, we evaluate the proposed dataset and method, comparing with the prior works. We conduct experiments with the following baseline methods including a naive baseline and multiple state-of-the-arr approaches: 1) Frozen is naive baseline that simply replicates the last frame of the input motion. 2) Social-STGCNN [2]
multi-agent trajectory prediction method using spatio-tempone graph convolutional networks. For trajectory prediction methods, we train and test using the root (global) trajectories and replicate the last frame for the root-relative (local) pose. 3) Vocia/VAE [72) is multi-agent trajectory prediction method using timewise variational autoencoder. 4) History Repeats Itself (HRI) [40] a single-person motion prediction method that allows absolute coordinate inputs. 5) Multi-Range Transformers (MRT) [68] predicts multi-person motion utilizing local-range encoder and a global-range encoder. 6) Social Motion Transformer (SoMoFormer)
employs the joints of all people as queries and predicts their future motion in parallel.

---

CMU-Mocap
Frozen
HRI
Wusi
MRT
20
40
60
80
100
mm
Figure 4: Comparison of baseline methods on CMU-Mocap [ and our dataset. We compute the 1s mean local pose prediction errors. The baseline methods are trained and tested on two datasets separately.
Table Performance comparison with the baseline methods. We compute the mean prediction errors of global pose, local pose, and root, respectively.
milliseconds
Global
Local
Root
400
600
1000
400
600
800
1000
400
600
800
1000
Frozen
119.2
165.7
207.3
244.5
62.8
82.8
98.9
112.1
99.8
139.3
175.0
207.6
Social-STGCNN
42
188.2
246.4
298.8
346.5
62.8
82.8
98.9
112.1
173.1
225.6
273.7
317.9
SocialVAE 72
84.0
127.2
171.7
215.1
62.8
82.8
98.9
112.1
52.6
89.7
130.3
170.8
IRRI
40
50,0
106.8
145.5
189.2
39.5
62.9
78.4
94.1
40.1
88.0
119.6
157.7
MRT
66.9
103.2
140.2
176.4
68.8
85.4
99.5
50.6
79.0
109.3
140.1
SoMoFormer
63
53.2
88.8
124.9
160.0
42.3
62.6
79.7
93.9
42.5
70.7
100.8
131.2
Ours
54.6
86.2
119.3
152.5
43.7
60.8
74.6
86.6
41.7
66.9
94.8
124.0
5.2.1
Quantitative results
We first benchmark three baseline methods to assess the motion prediction difficulty of our dataset in comparison to the commonly-use used existing dataset [T. Figure reveals that for the CMU-Mocap dataset [I]. merely repeating the last-frame pose (Frozen) leads significantly lower errors than the model predictions. This finding indicates that the motions in this dataset are relatively "static", making it less challenging and informative for the social motion prediction task. In contrast, the dynamic nature of our dataset gives rise to considerably higher prediction errors of the naive Frozen baseline. The results demonstrate that the proposed dataset serves as more convineing benchmark for social motion prediction and introduces more challenges to the field.
Furthermore, conduct quantitative evaluation of the proposed framework's performance and compare it with results with SOTA approaches. It worth noting that the single-person method HRI [40] demonstrates respectable short-term motion prediction accuracy; however, its performance over longer horizons is constrained by the absence of global context, particularly in root trajectory prediction. In contrast, our approach significantly surpasses SOTA methods in long-term social motion prediction, highlighting its effectiveness in modeling long-range human interactions. In the subsequent section, will demonstrate the critical role that social context plays in predicting long-term human motion via examples.
approaches. As illustrated in Table our method achieves competitive
5.2.2 Qualitative results
In order to better understand the performance improvement of our method, visualize its motion prediction results and compare with the baseline methods qualitatively. Figure 5 illustrates an representative result in which the input motion shows that defensive player (highlighted) is switching the target of defense. Our approach correctly predicts its future direction and orientations as the offensive player the other side will likely be covered by the teammate. In contrast, other approaches fail to reason this, predicting that the player would continue to move along the input trajectory or simply stop the middle.
In Figure three players to noticed are highlighted in rose, purple and blue (we also call them this way below). Additionally, visualize the actions produced by the intermediate policy networks to

---

HRI
x A
1 &
18 $
1 &
MRT
x A
1 7
1 11
1 117
SoMoFormer
x 1
1
1x 1
1x $
Ours
fx
1 r
GT
A R
x 1
Input
Output
Figure 5: Qualitative comparison with existing methods. Left two columns are past motion and right three columns are future motion. We highlight the prominent player with bright colors. Real motion are shown in green, and model predictions are shown in purple.
Level
Level
f
A
p
1
Sanaing 1 11
Level
f
Co-op
17..99
GT
f AM
1
98
Imput
Output
Figure 6: Actions produced by policy networks at different levels. Left two columns are past motions and right three columns are future motions.
interpret the decision-making process. Shown on the top row, player purple at level 1 policy network simply follows the past trend to receive the ball. As displayed in the second row, defender rose at level 2 gains knowledge from the first level that player purple is receiving the ball, and implements steal attempt. Nevertheless, such aggressive actions could create defensive gaps. Defender blue at level 3 senses such gaps caused by defender rose from level 2, therefore executes cooperative defense upwards to fill the gap.
5.3 Ablation study
Finally, we conduct ablation studies to investigate the influence of each component. In Table 4 (a)-(e) we change the policy network depth K. K = 0 corresponds to producing the final action directly without any recursive reasoning. We find that introducing the recursive reasoning process (K > 0) clearly lowers the prediction errors, proving the effectiveness of our framework design based on cognitive hierarchy. From (f) to (h), we keep K and study the influence of training regularizations. (f) removes the GAIL regularization for intermediate levels 1... K 1. and (g)

---

Table Ablation study of cognition levels and architecture designs.
milliseconds
Global
Local
Root
400
600
800
1000
400
600
800
1000
400
600
800
1000
(4) K = 0
60.1
94.3
129.2
163.7
47.4
66.2
82.0
95.3
45.7
72.7
101.8
131.8
(b) K 1
55.2
88.0
121.9
155.5
44.3
62.1
77.0
89.7
42.3
68.0
96.3
125,4
(c) K = 2
54.8
87.2
120.7
154.2
43.5
60.6
75.0
87.3
41.7
67.6
95.9
125.3
(d) K
54.5
86.3
119.7
153.5
43.6
60.5
74.7
86.6
42.0
67.3
95.5
125.2
(c) K = 3, full
54.6
86.2
119.3
152.5
43.7
60.8
74.6
86.6
41.7
66.9
94.8
124.0
(1) w/ mid GAIL
54.9
87.5
121.4
155.4
43.7
60.4
74.5
86.4
41.8
67.8
96.6
126.9
(g) w/o GATI.
54.6
87.1
121.1
155.1
43.4
60.4
74.7
86.8
41.9
68.1
97.0
127.0
(h) who weight sharing
57.1
91.0
125.8
160.2
44.4
62.2
77.1
89.7
43.6
70.3
99.5
129.5
Table 5: Quantitative comparison of actions produced by policy networks at different levels.
milliseconds
Global
Local
Root
400
600
800
1000
400
600
800
1000
400
600
800
1000
Level-1
76.8
122.4
170.3
217.7
50.1
69.0
85.2
99.1
61.4
100.4
143.2
186.9
Level-2
61.0
99.9
144.6
192.5
47.1
66.0
82.5
96.6
46.5
78.0
116.2
158.8
Level-3
54.6
86.2
119.3
152.5
43.7
60.8
74.6
86.6
41.7
66.9
94.8
124.0
completely removes GAIL. Training without GAIL increases the long long-term trajectory prediction errors, while slightly lowers the local pose prediction error. We also observe that models without GAIL tend to produce motions that appear less natural (e.g. limb twisting), leading to inferior visual effects. In (h), we use different policy network instances for levels .... K, and it turns out that sharing the policy networks leads to better generalization.
In addition the qualitative visualization of actions generated by policy networks at different reasoning levels, we also provide a quantitative comparison of their distance from ground-truth. Table demonstrates that policy networks different levels learn different actions, and policy networks at higher reasoning levels produce actions with less prediction errors. We hypothesize that, through the proposed recursive reasoning process, policy networks different levels display a range of cognitive types, with behavior spanning from relatively naive to substantially rational. Higher-level policies more closely approximate an equilibrium strategy, which usually alignment with expert demonstrations. We provide several example motion sequences in the video demo to further illustrate this observation.
Conclusion
In this work, propose the first large-scale multi-person 3D motion dataset featuring strategic human social interactions. This dataset surpasses the existing ones in scale, diversity, dynamics, and interaction, thus posing new challenges the social motion prediction problem. We reformulate the problem using MARL perspective. Building on this, we propose a framework that effectively combines behavioral cloning (BC), generative adversarial imitation learning (GAIL), and cognitive hierarchy. Our approach demonstrates strong generalization capabilities and improved interpretability for modeling strategic human social interactions.
Limitations: Our approach based on the cognitive hierachy assumption and may not hold true for all type of social interactions. This work do not explicitly model social roles, which can be explored as future work.
This work was partially supported by National Key R&D Program of China (2022ZD01 14900). We thank Xuesong Fan and Rujie Wu for their generous support during the data collection process.
Acknowledgment
10

---

References
[1] Cmu graphics lab motion capture database. http://
cmu. edu/
[2] Vida Adeli, Ehsan Adeli, Ian Reid, Juan Carlos Niebles, and Hamid Rezatofighi. Socially and contextually aware human motion and pose forecasting. IEEE Robotics and Automation Letters, 5(4):6033-600 6040, 2020.
[3] Vida Adeli, Mahsa Ehsanpour, Ian Reid, Juan Carlos Niebles, Silvio Savarese, Ehsan Adeli, and Hamid Rezatofighi. Tripod: Human trajectory and pose dynamics forccasting in the wild. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 13390-13400 2021.
[4] Nasir Ahmed, T_ Natarajan, and Kamisctty R Rao. Discrete cosine transform. IEEE transactions on Computers, 10009)) 1974.
[5] Javad Amirian, Jean- Bernadd Hayet, and Julien Pettré. Social ways: Learning multi-modal distributions of pedestrian trajectories with gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0-0, 2019.
[6] Mykhaylo Andriluka, Umar lqbal, Eldar Insafutdinov, Lconid Pishchulin, Anton Milan, Juergen Gall, and Bernt Schiele. Posetrack: A henehmark for human pose estimation and tracking. In Proceedings of the IEEE conference computer vision and paltern recognition, pages 5167-5176, 2018.
[7] Michael Bain and Claude Sommut. A framework for behavioural cloning. Machine intelligence, 15(15):103,
1999.
[8] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Frin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 11621-1 2020.
[9] Colin F Camerer. Behavioral game theory: Experiments in strategic interaction. Princeton university
press, 2011.
[10] Colin F Camerer, Teck-Hua Ho, and Juin-Kuan Chong- A cognitive hierarchy model of games. The Quarterly Journal of Economics, 119(3):861-898, 2004.
[11] Géry Casiez, Nicolas Roussel, and Daniel Vogel. filter: A simple speed-based low-pass filter for noisy input in interactive systems. In Proceedings of the SIGCHI Conference Human Factors in Computing Systems, CHI '22 page 25272530, New York, NY, USA, 2012. Association for Computing Machinery. [12] Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, al. Argoverse: 3d tracking and forecasting with rich maps. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8748 8757, 2019.
[13] Tanya L Chartrand and John A Bargh. The chameleon effect: the perception behavior link and social interaction. Journal of personality and social psychology, 76(6);893, 1999.
[14] Miguel A Costa-Gomes and Vincent P Crawford. Cognition and behavior in two-person guessing games: An experimental study. American economic review, 96(5):1737-1766. 2006.
[15] Vincent P Crawford and Nagore Iriberri. Level-k auctions: Can nonequilibrium model of strategic thinking explain the winner's 's curse and overbidding private-value auctions? Econometrica, 75(6): 1721))77221770 2007.
[16] Henggang Cui, Vladan Radosavljevic, Fang-Chich Chou, Tsung-Han Hann--Hanlie Lin, Thi Nguyen, Tzu-Kuo Huang, Jeff Schneider, and Nemanja Djuric. Multimodal trajectory predictions for autonomous driving using deep convolutional networks. In 2019 International Conference on Robotics and Automation (ICRA), pages
2090-2096. IEEE, 2019.
[17] Nachiket Deo and Mohan M Trivedi. Convolutional social pooling for vehicle trajectory prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 1468- 1476, 2018
[18] Scott Ettinger. Shuyang Cheng, Benjamin Caine, Cheaxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp. Charles R Qi. Yin Zhou, et al. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9710 7710-9719, 2021.
[19] Chris D Frith and Uta Frith. The neural basis of mentalizing, Neuron, 50K4):531-534, 2006.
[20] Drew Fudenberg, Fudenberg Drew, David K Levine, and David K Levine. The theory of learning in games, volume 2. MIT press. 1998.
[21] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.
[22] Wen Guo, Xiaoyu Bie, Xavier Hameda--Pinedaa and Francesc Moreno-Noguer. Multi-person extreme mo- tion prediction. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 13053-13064, 2022.
[23] Erann Haruvy, Dale O Stahl, and Paul W Wilson. Evidence for optimistic and pessimistic behavior in normal-form games. Economics Letters, (3(3):255-25) 259, 1999.

---

[24] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778. 2016.
[25] Dirk Helbing. Péter Molnár, Illés J Farkas, and Kai Bolay. Sell-organizing pedestrian movement. Environ- ment and planning B: planning and design, 28(3):361-383, 2001.
[26] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in neural information processing systems. pages 1565-4572 2016.
[27] Zhiyu Huang, Haochen Liu, and Chen Lv. Gameformer: Game-theoretic modeling and learning of transformer- based interactive prediction and planning for autonomous driving. arXiv preprint arXiv:2303.088 057600. 2023,
[28] Catalin lonescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.6M 66: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions on pattern analysis and machine intelligence, 36(7):1325- 1339,,13,22220 2014.
[29] Umar lqbal, Anton Milan, and Juergen Gall. Posetrack: Joint multi-person pose estimation and tracking. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2011 -2200, 2017.
[30] Rohit Jena, Changliu Liu, and Katia Sycara. Augmenting gail with be for sample efficient imitation learning. In Conference on Robot Learning, pages 80-90. PMLR, 2021.
[31] Hanbyul Joo, Hao Liu, Lei Tan, Lin Gui, Bart Nabbe, lain Matthews. Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh. Panoptic studio: A massively multiview system for social motion capture. In Proceedings of the IEEE International Conference
Computer Vision, pages 3334-3342, 2015.
[32] Hanbyul Joo, Tomas Simon, Mina Cikara, and Yaser Sheikh. Towards social artificial intelligence: Nonverbal social signal prediction in triadic interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10873-10883 2019.
[33] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICIRR, 2015.
[34] Vineet Kosaraju, Amir Sadeghian, Roberto
Ian Reid, S Hamid Rezatofighi, and Silvio Savarese. Social-bigat: Multimodal trajectory forecasting using bicycle-gan gan and graph attention networks. arXiv preprint arXiv: 1907 03395, 2019.
[35] Tao Li, Yuhan Zhao, and Quanyan Zhu. The role of information structures in game-thcoree multi- ageet learning. Annual Reviews Control, 2022.
[36] Xin Li, Xiaowen Ying, and Mooi Choo Chuah. Grip: Graph-basce interaction-awar trajectory prediction.
In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pages 3960-390 3966. IEEÉ, 2019.
[37] Florian Lofting and Rouwen Canal-Brulande Anticipation in sport. Current opinion in psychology, 16:6-11, 2017.
[38] Yuanfu Luo, Panpan Cai, Aniket Bera, David Hsu, Wee Sun Lee, and Dinesh Manocha. Porca: Modeling and planning for autonomous driving among many pedestrians. IEEE Robotics and Automation Letters, 3(4):3418-325,, 2018.
[39] Karttikeya Mangalam, Harshayu Girase, Shreyas Agarwal, Kuan-Hui Lee, Ehsan Adeli, Jitendra Malik, and Adrien Gaidon. It is not the journey but the destination: Endpoint conditioned trajectory prediction. In European Conference on Computer Vision, pages 759-776. Springer, 2020.
[40] Wei Mao, Miaomiao Liu, and Mathieu Salzmann. History repeats itself: Human motion prediction via motion attention. In Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pages 474-489. Springer, 2020.
[41] Dushyant Mehta, Oleksandr Sotnychenko, Franziska Mueller, Weipeng Xu, Srinath Sridhar, Gerard Pons- Moll, and Christian Theobalt. Single-shot multi-person 3d pose estimation from monocular rgb. In 2018 International Conference on 3D Vision (3DV), pages 120-130. IEEE, 2018
[42] Abduallah Mohamed, Kun Qian, Mohamed Elhoseiny, and Christian Claudel. Social-stgenn: A social spatio-temporal graph convolutional neural network for human trajectory prediction. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14424 14432, 2020.
[43] Abduallah Mohamed, Deyao Zhu, Warren Vu, Mohamed Elhoseiny, and Christian Claudel. Social- implicit: Rethinking trajectory prediction evaluation and the effectiveness of implicit maximum likelihood estimation. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23 27, 2022, Proceedings, Part XXII, pages 463 479. Springer, 2022.
[44] Mehdi Moussaid, Niriaska Perozo, Simon Garnier, Dirk Helbing, and Guy Theraulaz. The walking behaviour of pedestrian social groups and its impact on crowd dynamics. Ploš one, 10047, 2010.
[45] Rosemarie Nagel. Unraveling in guessing games: An experimental study. The American economic review, ((5):1313-1326, 1995.
[46] Andrew Y Ng. Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In Icml, volume 1, page 2, 2000.
[47] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
12

---

[48] Xiaogang Peng. Yaodi Shen, Haoran Wang, Binling Nie, Yigang Wang, and Zizhao Wu. Somoformer: Social-aware motion transformer for multi-person motion prediction. arXiv preprint arXiv:2208.09224 2022.
[49] Pongsathom Raksincharoensak, Takahiro Hasegawa, and Masao Nagai. Motion planning and control of autonomous driving intelligence system based on risk potential optimization framework. International Journal of Automotive Engineering, -366 2016.
[50] Anell Ric, Carlota Torrents, Bruno Gonçalves, Jaime Sampaio, and Robert Hristovski. Soft-assembled multilevel dynamies of tactical behaviors in soccer. Frontiers in psychology, 7:1513, 2016.
[51] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635, 2011.
[52] Amir Sadeghian, Vineet Kosaraju, Ali Sadeghian, Noriaki Hirose. Hamid Rezatofighi, and Silvio Savarese. Sophie: An attentive gan for predicting paths compliant social and physical constraints. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1349-1388, 2019.
[53] Caude Sammut. Behavioral cloning. Encyclopedia of Machine Learning, pages 93-97, 2010.
[54] Natalie Sebanz, Harold Bekkering, and Günther Knoblich, Joint action: bodies and minds moving together. Trends in cognitive sciences, 10(2):00-7, 2006.
[55] Khurram Soomro, Haroon Idrees, and Mubarak Shah. Online localization and prediction of actions and interactions. JEEE Transactions on Pattern Analysis and Machine Intelligence, 2018.
[56] Dale O Stahl and Paul W Wilson. On players' models of other players: Theory and experimental evidence. Games and Economic Behavior, 10(1):218-2554 1995.
[57] Dale O Stahl 11 and Paul W Wilson. Experimental evidence on players' models of other players. Journal of economic behavior & organization, 25(3):309-327, 1994.
[58] Jianhua Sun, Qinhong Jiang, and Cewu Lu. Recursive social behavior graph for trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 660 -69, 2020.
[59] Jianhua Sun, Yuxuan Li, Hao-Shu Fang, and Cewu Lu. Three steps to multimodal trajectory prediction: Modality clustering, classification and synthesis. arXiv preprint arXiv:2103.0788 2021. Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[60]
[61] NP Van der Aa, Xinghan Luo, Geert-Jan Giezeman, Robby T Tan, and Remco Veltkamp. Umpm benchmark: A multi- person dataset with synchronized video and motion capture data for evaluation of multi-person articulated human motion and interaction. In 2011 IEEE international conference on computer vision workshops (ICCV Workshops), pages 1264-1269. IEEE, 2011.
[62] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[63] Edward Vendrow, Satyajit Kumar, Ehsan Adeli, and Hamid Rezatofighi. Somoformer: Multi-person pose forecasting with transformers. arXiv preprint arXiv:2208. 14023, 2022.
[64] Luis Vilar, Duarte Araújo, Keith Davids, and Chris Button, The role of ecological dynamics in analysing performance in team sports. Sports Medicine, 42: 1000,,02200 2012.
[65] Timo Von Marcard, Roberto Henschel, Michael J Black, Bodo Rosenhahn, and Gerard Pons-Moll, Re- covering accurate 3d human pose in the wild using imus and a moving camera. In Proceedings of the European conference on computer vision (ECCV), pages 601-617, 2018.
[66] Borui Wang. Ehsan Adeli, Hsu-kuang Chiu. De-An Huang, and Juan Carlos Niebles. Imitation learning for human pose prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7124-7133, 2019.
[67] Chenxi Wang, Yunfeng Wang, Zixuan Huang, and Zhiwen Chen. Simple baseline for single human motion forecasting. In Proceedings of the IEEÉ/CVE International Conference on Computer Vision, pages
2260-2265, 2021.
[68]
Jiashun Wang, Huazhe Xu, Medhini Narasimhan, and Xiaolong Wang. Multi-person 3d motion prediction with multi-range transformers. Advances in Neural Information Processing Syxtems, 34:6036-600 (06666609,222 2021.
[69] Zhongdao Wang. Liang Zheng, Yixuan Liu, and Shengjin Wang. Towards real-time multi-object tracking.
The European Conference on Computer Vision (ECCV), 2020.
[70] A Mark Williams, Paul Ward, John M Knowles, and Nicholas J Smeeton. Anticipation skill in real-world task: measurement, training, and transfer in tennis. Journal of Experimental Psychology: Applied, 8(4):259, 2002.
[71] James Wright and Kevin Leyton-Brown. Beyond equilibrium: Predicting human behavior in normal-font games. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 24, pages 901-907, 2010.
[72] Pei Xu, Jean-Bernard Hayet, and Joannis Karamouzas. Socialvae: Human trajectory prediction using timewise latents. In Furopean Conference on Computer Vision, pages 511-528 Springer, 2022.
[73] Sirui Xu, Yu-Xiong Wang, and Liangyan Gui. Stochastic multi-person 3d motion forecasting. In The Eleventh International Conference on Learning Representations, 2023.
person
13

---

[74] Hang Ye, Wentao Zhu, Chunyu Wang, Rujie Wu, and Yizhou Wang. Faster voxelpose: Real-time 3d human pose estimation by orthographic projection. In Computer Vision-ECCV 2022: 17th European Conference, Aviv, Israel, October 23-27. 2022, Proceedings, Part VI, pages 142-159. Springer, 2022.
[75] Cunjun Yu, Xiao Ma, Jiawei Ren, Haiyu Zhao, and Shuai Yi. Spatio-temporal graph transformer networks for pedestrian trajectory prediction. In European Conference Computer Vision, pages 507-523. Springer, 2020.
[76] Wentao Zhu, Xiaoxuan Ma, Dongwoo Ro, Hai Ci, Jinlu Zhang, Jiaxin Shi, Feng Gao, Qi Tian, and Yizhou Wang. Human motion generation: A survey, arXiv preprint arXiv:2307.10894 2023.
14

---

A Appendix
A.1
Dataset
A.1.1 Data Collection
We record basketball drill videos from 11 camera views at a frame rate of 25 FPS. Multi-person 3D poses are estimated using Vextel-V 774. To obtain 2D heatmaps, we first resize the image to 960 x 512 pixels and employ the ResNet [24] backbone. The 3D space size is set to 16m x 16m 3m, with the granularity of human detection networks at 80 x 80 20. To estimate the joint locations of each subject, we set the human bounding box size to 2m x 2m x 2.4m and use voxel size of 64 x 64 x 64. We further apply JDETracker [69] with a tracking frame rate of 25FPS and a threshold of 0.2m. Next, we conduct an automatic failure detection process. We first discard motion clips with incorrect person counts and then filter outliers with root position changes exceeding a 0.4m threshold compared to neighboring frames. Subsequently, we adopt the one-euro filter []] for temporal smoothing. We set different parameters for different body joints because they exhibit different moving patterns. For upper limb joints such as wrists and elbows, we set seutoff 0.0287 and 3 = 0.5; for lower body joints including ankles and knees, we set Sautoft = 0.0274, 3 = 1; and
for the rest of the body, we set Seutoff = 1, B = 1. Finally, we perform a manual check to ensure the correctness of the data sequences.
A.1.2 Data Analysis
Comparison of Two Datasets
40 35
30
25
20
15
10
0 Count
5
10
15
20
25
30
35
40
45
Figure 7: Clip length distribution comparison between our dataset and CMU-Mocap
We visualize the clip lengths of our dataset and compare them with CMU-Mocap M. Figure 7 shows that our dataset features a longer overall duration and a larger number of long sequences.
Elbow
Waist
Shoulder
Knee
0.40
0.40
De sel
1.0
0.32
0.32
0.8
0.8
0.6
0.24
0.24
0.4
0.4
0.16
0.16
0,2
0.08
0.08
0.0
0.0
0.00
(a) Overall joint angle distribution
(b) Intra-motion joint angle std distribution
Figure 8: Distribution comparison between our dataset and CMU-Mocap 11. (a): We plot the joint angle distribution over the entire dataset. (b): We calculate the joint angle standard deviation for each motion sequence, then plot the std distribution.
Furthermore, we assess pose diversity and motion intensity utilizing joint angle distributions. As Figure 8(a) shows, our dataset encompasses broader range of plausible body joint angles, indicating
15

---

superior pose diversity. We then investigate motion intensity using intra-motion variation. Figure ((b) illustrates that motion sequences in our dataset have higher joint angle variances, indicating more dynamic motions.
A.2 Implementation details
We represent human motion with 3D skeleton sequences following previous works Specifically, each x E R represents a 3D human pose for person p at timestep t. containing the Cartesian coordinates of skeleton joints.
68
48
We apply Discrete Cosine Transform (DCT) to the input motions and Inverse Discrete Cosine
Transform (IDCT) for the decoded actions following 40 688. We employ local-range Transformer encoder and a global-range Transformer encoder [68] as the state encoders. We utilize Transformer
decoders the policy networks. We concatenate the local state features sy with global state features 8n (for level-0) or joint agent actions at the previous level a(k-1j (for level-k, k 2 1). For GAIL training, implement a discriminator D that takes a pair of state and joint actions a as input and aims to classify whether the actions are from the policy network or expert demonstrations. The discriminator consists of global-range Transformer encoder, a temporal average pooling layer, and an MLP with one hidden layer. We implement the proposed framework with PyTorch [4 using Linux machine with NVIDIA V100 GPU. We À 0.002, batch size 32, learning rate 0.0001, and train for 60 epochs using Adam
33 optimizer.
A.3 Ablation Study
Table 6: Ablation study of X.
milliseconds
Global
Local
Rool
400
600
800
1000
400
600
800
1000
400
600
800
1000
À 0.01
61.3
94.1
127.5
160.2
50.3
69.3
84.9
97.4
45.6
71.1
99.2
128.0
À 0,005
54.8
86,9
120.5
154.6
43.8
61.1
74.9
87.4
41.8
67.4
97.6
126.2
0.002
54.6
86.2
119.3
152.5
43.7
60.8
74.6
86.6
41.7
66.9
94.8
124,0
0.001
54.3
86.1
119.5
153.5
43.1
60.0
74,2
86.1
41.4
66.8
95.2
125.3
0.0002
54.2
86.5
120.2
43,0
59.9
96.2
126.1
We provide an additional ablation study on loss weight À in Table
A.4 Discussions
Broader Impacts Our approach does not rely on explicit biometric markers like unique fingerprints, retina scans, or body markers. The skeletal structure representation utilized in our dataset is based on generalized joint positions and movements. Our approach should only be employed with legally obtained human motion data. Meanshile, our approach may contribute to more interpretable decision- making processes and holds potential for applications in sports tactical analysis and education.
Assets To collect our dataset, we acquire written consent from the participating athletes and coaches. We plan to release the processed 3D human motion sequences under license agreement that permits their use for non-commercial scientific research purposes.
Human Subjects We inform the participating athletes about the data collection process and its intended uses, ensuring that we obtain their written consent. The instructions are provided by their coach, and the collected drills are part of their regular training routines. All participants receive compensation in accordance with their standard practices. To ensure privacy, we have implemented rigorous measures. Our data release strategy does not involve disclosing raw video data that may contain identifiable information, such as facial features. Instead, we will exclusively release processed 3D human motion sequences derived from the video data, guaranteeing the anonymity of the athletes.
16


Reviews:
[{"review_id": "SHvwJQvWUn", "reviewer": "Reviewer_EKS7", "summary": "--", "questions": "--", "limitations": "--", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "--", "weaknesses": "--", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to provide your feedback. However, we noticed that the specific details of your review were not included. To better understand your viewpoint and address any potential issues, it would be immensely helpful if you could provide more detailed feedback."}, {"review_id": "NVbcS48QPG", "reviewer": "Reviewer_X2WW", "summary": "This paper aims to predict multiperson human motion. The main contributions of this paper are:\n1. The paper presents a large-scale multi-human 3D motion dataset with intense, strategic interactions.\n2. The paper formulates the multiperson human prediction problem as MARL and presents a hierarchy framework to model complex social interactions.\n", "questions": "Please check 1. - 3. in Weaknesses.\nIn addition, why was the best performance achieved when K=3? if the claims of the cognitive hierarchy are true, I was wondering what happened after having more levels of decision. \n\n", "limitations": "The visualization of the output from each level shows some existence of cognitive hierarchies, but it should be more clear if the paper and the demo video could include the visuals in meshes (with ball movement if possible) like Figure 1.\nOverall, the author's response to the concerns in the Weakness section is needed to make the final decision. I am happy to increase the rating if my concerns are addressed.\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper has several strengths:\n1. Overall, the paper is well-written with a clear and well-motivated introduction. The hierarchy framework is introduced to consider the recursive decisions in the interaction.\n2. The paper proposed a large-scale multi-person motion dataset\n3. Evaluation on the proposed dataset demonstrates that the method outperforms state-of-the-art methods.\n", "weaknesses": "1. In L32-34, the paper claims that most existing methods employ end-to-end supervised training which overlooks the cognitive aspects. Considering the state (past motion) and action (future velocity) space defined in this paper, could the author provide more insight on the difference between the BC, GAIL used in the paper and GAN-based supervised learning in related work? \n2. While this work qualitatively demonstrates the utility of the proposed framework for tasks specific to motions like sports, it would be beneficial to provide additional comparisons on existing benchmarks in addition to Figure 4.\n3. Other multi-person datasets in sports also exist, such as KTH Multiview Football datasets. The paper may compare the proposed dataset and method with them.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's constructive and positive feedback. We are grateful that you recognize our main contributions and find it 'well-written with a clear and well-motivated introduction'. In the following, we aim to address your questions and concerns:\n\n**Q1: Provide more insight on the difference between the BC, GAIL and GAN-based supervised learning.**\n\nA1: Wang _et al._ [57] initially introduces an imitation learning formulation for single-person human motion prediction with BC and GAIL compared to end-to-end supervised training. Our work extends conceptually from [24, 57] as it adapts BC and GAIL for a multi-agent imitation learning context. Our approach is distinct from GAN-based supervised learning in two primary ways:\n\n- The GAIL objective focuses on making the state-action pairs produced by the joint policies of agents and that of experts indistinguishable. This is in contrast to the GAN framework (_e.g._, [59]), where the objective is to make the predicted future motion indistinguishable from the real ones.\n\n- Our MARL formulation enables a more principled integration with the cognitive hierarchy theory, while existing works overlook the cognitive aspects of human social action planning. By modeling the motion prediction process with chained policy networks, we derive the BC and GAIL regularization for each level accordingly, and propose to share parameters for policy networks $\\phi_{(1)} \\dots \\phi_{(K)}$, both of which lead to performance improvements as shown in Table 3 of the submitted manuscript.\n\n**Q2: It would be beneficial to provide additional comparisons on existing benchmarks.**\n\nA2: Following your suggestion, we train and test our method on the CMU-Mocap (UMPM) dataset [40]. Please refer to Table 1 of the rebuttal PDF for the results. Our approach outperforms the previous methods [32, R2, 59] and is comparable with a concurrent work [40]. Notably, our method exhibits particular strength in predicting long-term root trajectories and global motions.\n\n**Q3: Other multi-person datasets in sports also exist, such as KTH Multiview Football datasets.**\n\nA3: Please refer to General Response Q1 for the discussions. We will make it more clear in the revision.\n\n**Q4: Why was the best performance achieved when K=3?**\n\nA4: Our cognition-inspired framework aims to learn joint policies from real-world expert demonstrations and to discover the explainable cognitive hierarchies. The finding that $K=3$ provides the best fit for the real-world dataset may suggest that this particular depth of strategic reasoning most aptly encapsulates the human decision-making process as captured in our collected data. As we increase the number of hierarchical levels ($K>3$), we continue to observe the cognitive hierarchy visualizations among different levels, though there is a slight increase in the mean prediction error.\n\n\n**Q5: Could include the visuals in meshes (with ball movement if possible).**\n\nA5: Thanks for your suggestion!\n\n  - **Body Mesh**: As discussed in General Response Q2, we use 3D poses in the experiments. While the 3D poses can certainly be fitted to a parametric body mesh (such as SMPL, as shown in Figure 1) for a more intuitive visualization, this process may introduce additional errors. To avoid these potential complications, we chose to present the 3D skeletons as they are -- the direct output of the model, free from potential distortions introduced by fitting. This approach makes it easier to identify potential issues. However, in response to your suggestion, we will include the mesh fitting and visualization scripts in our code release.\n\n  - **Ball movement**: We implemented a 3D ball trajectory estimator using an object detector and multi-view triangulation. However, we chose not to include the ball information in model training for two reasons: 1) to ensure a fair comparison with previous methodologies, and 2) due to the fact that ball moves significantly faster than humans, which sometimes result in inaccurate estimations caused by motion blur and detection failures. We are committed to improving our ball detection algorithms, and in response to your suggestion, we will include the 3D ball trajectory estimator and visualization tools in our code release."}, {"review_id": "GAu3jCLK2b", "reviewer": "Reviewer_8iac", "summary": "The paper has two major contributions to the field of multi-person motion detection:\n\n1) A new open source dataset that is sufficiently large scale as compared to those already available. But more importantly is dense in terms of strategic interactions and more diversity of action distributions for reinforcement learning algorithms. This is very relevant to the field.\n\n2) To demonstrate the efficacy of the datasets, they also propose a new initiation learning framework combining behavior cloning as well as  GAIL.", "questions": "Would it be possible to include the relevant recent work in the field of multi-agent human trajectory prediction?", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1) The paper is very well written with clear figures to make the concepts easy to understand.\n2) The dataset is a welcome addition to the space of close space multi agent interaction modeling.\n3) The experimental section is very detailed with an ablative section to help understand the implications of each of the contributions of the MARL framework.", "weaknesses": "The paper suffers from two major issues. The authors ignore a large body of work that are not directly related to the close space interactions but highly relevant to the field of multi-agent trajectory prediction. This includes both datasets as well as methods available to get state of the art results in the domain.\n\n1) Lack of survey of related work with respect to the dataset: Many work in the autonomous driving domain feature highly complex social scenarios with many agents interacting with one other. The datasets include industrial benchmarks like Argoverse, nuScenes, Waymo Open Motion Dataset [1] amongst others. While these datasets including not just pedestrian trajectories but also other road users like vehicles and cyclists, a comparison must be made about how the complex multi-agent crowd level interactions are different and similar to the close space interaction modeling that is being studied in this paper. The crowds of pedestrian in these scene perform complex social maneuvers of working cooperatively ( cross walks) and destructively (jaywalking) as examples and how these strategies are relevant to this field is an important comparison. Given the existence of these other much larger dataset in terms of number of subjects available, it will be important to understand what benefits the new dataset adds to the field.\n\n2) Lack of survey of related work with respect to the methods: On the datasets mentioned above a large body of work exists involving both simple deep learning transformer and non transformer based Bayesian methods as well as reinforcement learning frameworks [4] (as an example) that the MARL framework needs to be compared against. \n\n\n[1] Waymo Open Motion Dataset\n\n[2] Symphony: Learning Realistic and Diverse Agents for Autonomous Driving Simulation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s constructive and positive feedback. We are happy that the you find our paper 'well-written', our proposed dataset 'a welcome addition', and our experiments 'very detailed'. In the following, we aim to address your questions and concerns:\n\n**Q1: Lack of survey of related work with respect to the dataset.**\n\nA1: Thank you very much for the suggestion! We provide a discussion on the multi-agent trajectory prediction datasets in General Response Q1. We will cite and discuss these works in Section 2.1 in the next revision following your suggestion.\n\n**Q2: Lack of survey of related work with respect to the methods.**\n\nA2: Thank you very much for the suggestion! We referred to the relevant recent work in the field of multi-agent human trajectory prediction in Line 20-26. We will cite and discuss more research works in this area in Section 2.2 in the next revision following your suggestion. By the way, is part of the review references ([4]) missing?"}, {"review_id": "diN3ogp4iu", "reviewer": "Reviewer_Nb5i", "summary": "The paper addresses the problem of social motion forecasting utilizing a multi-agent reinforcement learning and combines behavioral cloning and generative adversarial imitation learning. Social and strategic interactions are modeled in a “cognitive hierarchy framework”. The paper further introduces a new large-scale 3D multi-human motion dataset based on no-dribble-3-on-2 basketball.", "questions": "I wonder if it would be possible to evaluate the combined motion of all persons, e.g. not just of the per-person poses but also the global configuration of the poses is sensible or not. Imagine, for example, one person throwing the ball to their left-hand person but the right-hand person generates the \"receive\" motion - which would be unlikely. \n\nIt is unclear which version of PoseTrack the authors address in Table 1: PoseTrack17 [23] contains 556 sequences (and not 13) and PoseTrack18 [5] contains 1138 sequences. Could the authors elaborate if this version is a subset of PoseTrack?\n\nI would assume this is an encoding error but in the supplementary video every 4 to 5th frame “freezes” making it a bit more difficult to check the results frame by frame in video (QuickTime Player, OSX).\n\nWhat was the motivation of color choice for Visual representation of the Cognitive Hierarchy Visualization: I would suggest to use warm colors for the offense and cold colors for the defense as especially purple and rose are too close to each other.", "limitations": "The authors address some limitations, notably the assumption that a certain cognitive hierarchy is present in the data.\nIt would be interesting to see how far into the future the model can predict without producing failing poses or freezing up.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper is, in general, well-written and the method well-motivated. The novel dataset is highly relevant in the under-explored area of multiple human motion forecasting.", "weaknesses": "In general, 1s of forecast for such complex social interactions is too short, as players mostly just continue their motion, e.g. swing of an arm. In basketball there is the 5s rule, I would thus suggest to at least forecast 5s of future motion to get a good understanding of the interactions and of the game. Instead of MPJPE metrics for long-term motion forecasting could be employed, e.g. [b,c].\n\nThe method results in the supplementary video exhibit a lot of foot sliding, suggesting a generalization problem for walking motions. I would assume that this is due to using 3D velocities as actions, which are translation invariant but crucially not rotation invariant. The choice of a rotation-variant actions is surprising as the introduced dataset (almost) plays out on a circle, with 3 players on the circle and 2 players within. Did the authors try to alleviate the issue by augmenting the data?\n\nThe authors say that \"Human motions exhibit greater dynamism in terms of pose diversity and movement speed, making motion prediction more challenging than in previous datasets” (LL42-44) but then say that players are not allowed to “dribble” (L111 & L115) - does that not significantly reduce the range of motion, i.e. there is little global translation? \n\nThe authors should elaborate how they obtained body joint angles (L144), Figure 2. In Figure 1 it seems that the authors extract SMPL parameters as pose representation but it is never specified - however, extracting angular representations from 3D joint locations is non-trivial and the authors need to explain how those are produced. The lack of this information makes Figure 2 unconvincing: assuming SMPL was used, the skeletal structure is quite different from the parametric model used in CMU [1], we thus would expect significant differences in angular distributions. For example, hinge joints in CMU are zero everywhere except at the hinge dimension - is this the case in the authors representation as well? \n\nThe paper should replicate the MRT [59] experiments - at least for the available datasets CMU Mocap and MuPoTS-3D.\n\nHRI [32] originally was trained on local (normalized) motion and not on global coordinates: in this work, are the global coordinates passed to HRI for training and evaluation? How would HRI perform with normalization such as subtracting the root location at the last input frame? Such normalization can be undone trivially and should greatly improve the (short-term) motion prediction results. The evaluation as is is unfair for HRI if global coordinates are passed.\n\n\n**Minor issues**\n\nLL54-55: While CHT [9] is cited and put into context (LL98-105) a short overview would be helpful, as it is not clear from the paper what the authors mean by “k represents the depth of strategic thought” (L55).\n\nAn interesting dataset for evaluating the approach would be the Haggling dataset [a] where persons play a triadic haggling game, an adversarial game between two players to sell a product to a third person (buyer). This work should be at least cited and compared against in Table 1.\n\nIn the context of 3D human motion forecasting planning or “intention” has been addressed as intermediate signal in previous works [c,d] and should be cited.\n\n[a] Joo, Hanbyul, et al. \"Towards social artificial intelligence: Nonverbal social signal prediction in a triadic interaction.\" CVPR 2019.\n\n[b] Gopalakrishnan, Anand, et al. \"A neural temporal model for human motion prediction.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n\n[c] Tanke, Julian, Chintan Zaveri, and Juergen Gall. \"Intention-based long-term human motion anticipation.\" 3DV 2021.\n\n[d] Diller, Christian, Thomas Funkhouser, and Angela Dai. \"Forecasting characteristic 3D poses of human actions.\" Proceedings of the CVPR 2022.\n\n**Rebuttal**\nI appreciate the effort made by the authors to answer my questions.\n\nHowever, I still believe that 1s is too short for what the authors claim their dataset is capable of demonstrating: intricate \"strategic\" social interplay.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s constructive and detailed feedback. We are grateful that the you find our paper 'well-written', our proposed 'well-motivated', and our dataset 'highly relevant in the under-explored area'. In the following, we aim to address your concerns:\n\n**Q1: 1s is too short.** \n\nA1: We appreciate the reviewer's insightful suggestions. We understand the importance of longer-term predictions in understanding complex social interactions and the dynamics of the game. However, we would like to provide the following points to clarify our choice of a 1-second forecast horizon:\n\n- Basketball is characterized by its swift pace, whereby a significant amount of action can occur within a 1-second timeframe. In full-court matches, possession often concludes within few seconds. Our research scenario is even more intense and compact compared to conventional basketball games, and it operates at a faster pace.\n\n- Our video demonstration reveals that the 1-second future doesn't merely continue past motions. Instead, it involves defensive switches and sudden changes in moving directions. As a matter of fact, our level-1 policy is primarily motion continuation, but our policy network learns complex strategic behaviors at higher levels through the proposed cognitive hierarchy design.\n\n- We opted not to use a longer prediction horizon primarily because future motions heavily depend on the uncertain outcomes of passes. For instance, if player A is passing to player B, the subsequent motions will be greatly influenced by whether the ball is successfully received by B or stolen midway. Predicting such outcomes based on current information poses a considerable challenge for any model.\n\nWe plan to extend our datasets to include regular basketball games with more information and explore longer prediction lengths for future research.\n\n\n**Q2: Foot sliding.**\n\nA2: Foot sliding might be partly due to our model not explicitly regularizing foot contact velocities as previous works have done [R17, R18, R19]. Our research primarily targets predicting the strategic action decisions rather than the quality of the motion itself. Hence, we did not initially design our model to address these detailed aspects of the motion quality. Based on the reviewer's suggestion, we plan to explore training techniques such as foot contact loss and data augmentation to enhance the visual quality of the motion predictions. We believe these techniques could potentially alleviate the foot sliding issue.\n\n\n**Q3: Little global translation?**\n\nA3: The no-dribbling rule in this practice enforces the players to make more strategic decisions. While dribbling is prohibited for the offensive players, it doesn't necessarily result in a significant decrease in the overall range of motion. It's important to note that defensive players are not under the same restriction. They are required to continually move back and forth to maintain adequate defense coverage. This requirement leads to substantial global translation, as evidenced by the video we have provided. Therefore, while the scope of certain specific motions is reduced, the overall diversity and dynamism of human motions are preserved, and the challenge of predicting those motions remains high. The large root error (especially for the 'frozen' baseline) in Table 2 of the submission also demonstrate this idea.\n\n**Q4: Body joint angles.**\n\nA4: As discussed in General Response Q2, our dataset use 3D pose represetation. Previous work MRT [59] converts the CMU-Mocap dataset to a format of 15-joint 3D poses. We strictly followed its definition and perform data analysis on the aligned skeletal structure. We directly computed the angle of intersection formed by the hinged limbs since we do not have the limb twists of the 3D pose representations. \n\n**Q5: Replicate the MRT experiments.**\n\nA5: Following your suggestion, we train and test our method on the CMU-Mocap (UMPM) dataset [40]. Please refer to Table 1 of the rebuttal PDF for the results. Notably, our method exhibits particular strength in predicting long-term root trajectories and global motions.\n\n**Q6: Details on HRI.**\n\nA6: When training and evaluating HRI, we pass in normalized motions following the original settings. Therefore, it achieves respectable short-term motion prediction accuracy in Table 2, as supposed by the reviewer.\n\n**Q7: CHT overview.**\n\nA7: We provide an intuitive explanation on CHT and level-k reasoning in Line 51-55 and Line 195-197. We will add a short overview to Section 2.3 following your suggestion. \n\n**Q8: Haggling dataset.**\n\nA8: Please refer to General Response Q1. We will cite and compare with it following your suggestion.\n\n**Q9: Intention as intermediate signal.**\n\nA9: Thank you for the information, we will add them to Section 2.\n\n**Q10: Evaluating the combined motion.**\n\nA10: The GAIL global discriminator $D$ is developed on a similar idea, which penalizes global state-action pairs that are unlikely to occur. For a quantitative evaluation of the combined motion of all persons, incorporating a user study could be beneficial. We appreciate this suggestion and plan to include such a study in our future work.\n\n**Q11: PoseTrack version.**\n\nA11: For PoseTrack, we compare with the version provided by SoMoF [21], which may cause confusion in number of sequences given different definitions. We apologize for the confusion. We will change the dataset name to 'PoseTrack (SoMoF)' and revise the sequence numbers in next version.\n\n**Q12: Video encoding error.**\n\nA12: Thank you for bringing this issue to our attention. We have thoroughly reviewed the supplementary video and were unable to replicate the issue. Could you please provide us with your OS and QuickTime Player version? We will ensure that our video encoding is widely compatible in the future release.\n\n**Q13: Color choice.**\n\nA13: Thanks for your suggestion! We will change the visualization colors accordingly."}, {"review_id": "JFM7XRIN9T", "reviewer": "Reviewer_yiis", "summary": "This paper proposes a novel approach to address the social motion prediction problem by introducing a new large-scale multi-person 3D motion dataset featuring intense and strategic interactions among participants. The authors formulate the problem using a multi-agent reinforcement learning perspective and incorporate behavioral cloning and generative adversarial imitation learning to boost learning efficiency and generalization. They also take into account the cognitive aspects of the human social action planning process and develop a cognitive hierarchy framework to predict strategic human social interactions. The proposed approach outperforms state-of-the-art methods in challenging long-term social motion predictions. \n", "questions": "**According to review guidelines, 'if the contribution is a dataset or model, what steps did you take to make your results reproducible or verifiable?' The authors did not claim it in the main paper or abstract (It is important. Do NOT claim it in the appendix). Will authors only make a subset of data public or a full set?**\n\n+ Any ablation of $\\lambda$s?\n+ Multi-person motion capture datasets can be used not only for prediction but also for motion generation. Recent research (HumanMAC: Masked Motion Completion for Human Motion Prediction) use generative frameworks for motion prediction, and it will be great to discuss motion generation.\n+ w/o GAIL and w/o mid GAIL ablation shows that the design works little for global pose and local pose. I would like to discuss this with the authors. \n+ \"This dataset surpasses the existing ones in scale, diversity, dynamics, and interaction, thus posing new challenges to the social motion prediction problem. \" Little experiments to verify how the (1)scale, (2)diversity, and (3)interaction benefit the task?\n+ Description of annotation, SMPL or SMPL-X? How many joints? \n+ For motivation, why do authors build a dataset with intense, strategic interactions? Why are intense, strategic interactions essential for the task and the community? Intense, strategic interactions property is a characteristic of your dataset, but a lack of motivation. It will be great to show how these properties benefit the task in experiments.\n\n**Although I have many concerns about the dataset, the author's method of technological innovation is still interesting. I provide a borderline score here. If the author could discuss and address my concerns, I would improve my score.**\n", "limitations": "Limitations are discussed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The main strengths of this paper include:\n+ Introducing and analyzing the limitations of existing datasets for multi-person motion prediction tasks.\n+ Proposing the use of GAIL to improve the generalization ability of the learned policies.\n+ Provide a demo video. ", "weaknesses": "**Good motivation and interesting methods, but not sufficient efforts to support the claim.** I will state weaknesses as follows:\n\n\n+ Distribution comparison in Fig. 2 shows the comparison with CMU-Mocap, which is an old dataset. For claiming the dataset novelty, I suggest authors compare your dataset with more datasets here. Only several sub-figures of several joints are not sufficient to reflect the distribution of the full dataset. \n+ For dataset comparison, [1] has 2.2 hours of data and [2] is 6.5 hours. These two datasets should be compared. \n+ **Over-claim: I think a dataset of less than an hour cannot claim to be a large-scale dataset.**\n+ In lines 29-32, why [56, 1, 25] cannot be used for interactive tasks? CMU-Mocap can be used to perform motion prediction and generation tasks. \n+ Authors mentioned NTU-RGBD in Section 2.1 but did not compare with it in Table 1. \n+ Writing:\n  + orphan row (lines 233, 251, Table 3).\n  + Line 236: Table 4 -> Figure 4\n\n\n\n\n[1]: Umpm benchmark: A multi-person dataset with synchronized video and motion capture data for evaluation of articulated human motion and interaction.\n\n[2]: InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's constructive and positive feedback. We are grateful that you are interested in our 'method of technological innovation' with 'good motivation'. In the following, we seek to address your concerns:\n\n**Q1: Compare with more datasets and joints.**\n\nA1: We primarily compare with CMU-Mocap [1] as it is the main dataset used by previous works in this area. We will add more dataset comparisons with other datasets and joints following your suggestion.\n\n**Q2: UMPM and InterHuman dataset.** \n\nA2: Please refer to General Response Q1 for comparisons. We will cite and compare with the datasets following your suggestion.\n\n**Q3: Overclaim data size.**\n\nA3: Thanks for your valuable feedback on our dataset size claim. Nonetheles, it is important to note that the dataset is larger than any previously used in the field of multi-human motion prediction. Our dataset provides 3D skeleton ground truths with intense and strategic human interactions from professional atheletes. From such a perspective, we have greatly enlarged the available dataset in this research field for academic uses. We understand how the term \"large-scale\" might be misconstrued given the temporal length of the dataset. Therefore, in our revised manuscript, we will carefully modify the wording to more accurately represent the size and value of our dataset.\n\n\n**Q4：Why cannot be used for interactive tasks.**\n\nA4: While it's true that these datasets can be used for motion prediction and generation tasks, their utility for tasks involving interaction - a key focus of our research - is limited. This is due to the relatively low level of interactiveness, as we mentioned in Line 30-32. For instance, the individuals in these datasets are often recorded while moving casually or interacting randomly with others, rather than participating in strategic or purposeful social interactions.\n\n**Q5：NTU-RGB+D.**\n\nA5: NTU-RGB+D contains some action classes with two interacting humans. We provide a comparison in Table 3 of the rebuttal PDF. We will add the comparisons to Table 1 in the revision.\n\n**Q6:  Dataser release.**\n\nA6: The entire dataset, as well as our code and models, will be publicly released.\n\n**Q7: Ablation of $\\lambda$s.**\n\nA7: We provide an ablation study on $\\lambda$ in Table 2 of the rebuttal PDF. \n\n**Q8: Discuss motion generation.**\nA8: Thanks for your suggestion, and our dataset can also be potentially used for motion generation. We will cite and discuss HumanMAC and other related works on motion generation in _Introduction_ and _Related Works_ sections of our revised manuscript.\n\n**Q9: GAIL effects.**\n\nA9: Thanks for your insightful comment. In our manuscript, we have briefly addressed this observation in Line 275-276. Training without GAIL tends to increase the errors in long-term trajectory predictions, while it marginally reduces the errors in local pose prediction. Interestingly, we notice that models trained without GAIL tend to generate motions with smaller magnitudes, often resembling a \"freezing\" effect. This results in visually less appealing and less realistic animations. Moreover, an important aspect to note is the impact of GAIL on the emergence of cognitive hierarchies within the model. In our experiments, the absence of GAIL leads to a lack of discernible cognitive hierarchy.\n\n**Q10: Verify how the (1)scale, (2)diversity, and (3)interaction benefit the task?**\n\nA10: Thank you for bringing up the important point about the benefits of the scale, diversity, and interaction of our dataset for the social motion prediction task. We are pleased to clarify these aspects:\n\n- Scale: Larger datasets are generally considered beneficial for research tasks as they provide a more extensive base for training models, improving their ability to generalize to unseen data. The advantageous size of our model makes it a valuable resource of the task for the research community.\n  \n- Diversity: As shown in Figure 4 and discussed in Line 235-242, our dataset exhibits a high degree of diversity. The results demonstrate that the proposed dataset serves as a more convincing benchmark for social motion prediction and introduces more challenges to the field.\n\n- Interaction: As evidenced by Table 2 and Line 245-249, while the HRI [32] model performs exceptionally well on short-term motion prediction, the global context becomes crucial for long-term motion prediction. Our dataset, with its highly interactive nature, offers strong cues for motion prediction with a global context. These interactions are vital to understanding and predicting complex social behaviors.\n\nWe appreciate your feedback, and will revise our manuscript to articulate these points more clearly.\n\n\n**Q11: Annotation.**\n\nA11: Please refer to General Response Q2. We will make it more clear in the revision.\n\n**Q12: Motivation for the dataset.**\n\nA12: Thank you for your insightful question.\n\n- Our primary motivation in focusing on intense, strategic interactions stems from a desire to more accurately capture the complexity inherent in real-world human interactions. These interactions often involve strategic decision-making processes and can be social, competitive, or cooperative in nature. Most existing datasets do not adequately capture these complexities, which is a gap that our dataset aims to fill.\n- By incorporating such interactions, we enable the model to learn beyond simple human motion patterns. Our dataset encourages a deeper understanding of human actions, including aspects such as intentions and the decision-making process. Ultimately, our goal is to develop models that not only understand human behaviors but can also interact meaningfully with humans.\n- As for how these properties benefit our task, we refer you to discussions in A10 (also Table 4, Figure 2, Line 235-249).\n\nWe will revise our presentation to better highlight how these properties of our dataset contribute to the task and benefit the broader research community.\n"}]

Author rebuttal:
# General Response

We sincerely thank all reviewers for their meticulous reviews and constructive remarks. It is heartening to note that all reviewers have acknowledged the motivation behind our work, as well as its main contributions — especially the proposed method. We will incorporate discussion of relevant works and additional experiments in the ﬁnal version accordingly. In the following, we will first address some shared issues on the proposed dataset, then respond to each reviewer's comments respectively.

**Q1: Dataset comparisons**

A1: We discuss the existing multi-person motion datasets in Section 2.1 and provide a comparison with some of them (Table 1 of the submited manuscript). The table contains comparisons with _'existing multi-person motion datasets employed by previous works on the multi-person motion prediction task'_ (table caption). In other words, we primarily compare with the utilized datasets within this domain as referenced in previous works [2, 3, 58, 59, 19, 54, 40, 63]. We recognize that there are other multi-person datasets that are relevant to our work, as recommended by the reviewers. Consequently, we present a brief discussion on the differences between our work and each of these datasets below. **These works will also be cited and discussed in Section 2.1 and Table 1 of the revised manuscript following your suggestions.**



- **Multi-person Sports Datasets (Reviewer X2WW).** There exists a variety of multi-person sports datasets for group behavior understanding, e.g. NBA [R3], MultiSports [R4], Volleyball [R5], NCAA [R6]. However, these datasets are primarily designed for action recognition from RGB videos and lack pose annotations or 3D information. Felsen _et al._ [R7] collect a water polo dataset and a basketball dataset for group trajectory prediction, but they only contain trajectory-level data rather than human poses. The KTH Multiview Football dataset [R8] consists of two subsets (2D and 3D), with the 3D subset containing merely 800 time frames (32 seconds) for two isolated players. Therefore, it might be more suitable for testing or few-shot adaptation.
- **Multi-agent trajectory prediction datasets in autonomous driving (Reviewer 8iac).** As suggested by the reviewer, there are a large body of work on multi-agent trajectory prediction that is _'not directly related to'_ our task, but _'highly relevant'_. We briefly discussed this research area in Line 20-26. Given the diffrent task objectives, the datasets also exhibit different characteristics. The trajectory datasets [R9, R10, R11, R12] possess advanteges such as larger scale and diverse classes of agents. Nevertheless, these datasets only contain coarse-grained interactions at the trajectory level (mainly collision avoidance) and fail to capture the rich and fine-grained human actions. Furthermore, our dataset features intense and strategic interactions, such as frequent sudden direction changes and deception maneuvers, which are largely absent from existing datasets. Thus, our dataset can serve as a valuable supplement to the existing ones, posing additional challenges to this research field.

   
- **Haggling dataset [R13] (Reviewer Nb5i)**. The Haggling dataset shares relevance with our work as that it also invloves social interactions and body gesture prediction. The main difference is that they focus on a triadic conversational scenario, therefore the interactions are relatively simpler, and the body motions are more static.

- **UMPM benchmark [R1] (Reviewer yiis).** The UMPM dataset is a multi-person mocap dataset. As per the paper, 2/9 of the scenarios contain interaction between the subjects and is suitable for our task: _(6) a conversation with natural gestures, and (7) the subjects throw or pass a ball to each other while walking around_. Compared to it, our dataset is advantageous in terms of a larger number of subjects and more strategic interactions. We also conduct experiments on an extended version of UMPM following [40] and the results are reported in Table 1 of the rebuttal PDF.

- **InterGen [R14] (Reviewer yiis).** This concurrent work introduces a multimodal dataset called InterHuman for text-to-multi-human motion generation. Consequently, their primary objective is to cover a diverse range of interactions and annotate with extensive textual descriptions. In contrast, the focus of our dataset is intense and strategic human interactions for social motion prediction. We believe that both our dataset and InterHuman offer valuable additions to the research community, albeit with different task focuses.

**Q2: Data format**

A2: Our work primarily focuses on 3D body keypoint sequences following previous studies [59, 19, 54, 57, 40, 32], as discussed in supplementary material Line 26-28. The 3D skeletons are also the standard product of markerless multi-view motion capture methods [64, R15, R16] which are more suitable for our data collection scenario than marker-based ones. We use $J=15$ skeleton joints with the same difinition with previous works [40, 59]. 

---

## Response to Ethics Reviews

We thank the Ethics Reviewers for taking the time to review our paper and detailed suggestions.

1. The athletes were indeed informed that their identities would be kept private. We obtained written consent from each participant, clearly outlining the data collection process and intended uses, ensuring transparency. We will include the consent forms given to participants in the Appendix in the revisions.

2. While gait has been acknowledged as a behavioral biometric in recent studies, gait recognition primarily relies on video or sensor data. It is crucial to clarify that our released dataset does not contain video or sensor statistics, mitigating the risks associated with biometric identification. To our best knowledge, there is no viable solution to recognize person identities from the data we provided given no extra information.


Meta-review: The core contribution of this paper is the introduction of a novel dataset focused on 3D poses in the context of sports. This dataset is particularly intriguing as it delves into the prediction of social fine-grained motions. Given the rising demand and interest in such specific and nuanced data, the dataset holds significant value for the community.

The availability of this dataset can potentially pave the way for advancements in the domain, aiding researchers in understanding and predicting intricate motion dynamics in sports scenarios. 

We ask the authors to carefully  integrate reviewers' comments and the ethics discussion in the final version.
Decision: Accept (poster)