SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we evaluate the proposed dataset and method, comparing with the prior works. We conduct experiments with the following baseline methods including a naive baseline and multiple state-of-the-art approaches: 1) Frozen is a naive baseline that simply replicates the last frame of the input motion. 2) Social-STGCNN [42] is a multi-agent trajectory prediction method using spatio-temporal graph convolutional networks. For trajectory prediction methods, we train and test using the root (global) trajectories and replicate the last frame for the root-relative (local) pose. 3) SocialVAE [72] is a multi-agent trajectory prediction method using timewise variational autoencoder. 4) History Repeats Itself (HRI) [40] is a single-person motion prediction method that allows absolute coordinate inputs. 5) Multi-Range Transformers (MRT) [68] predicts multi-person motion utilizing a local-range encoder and a global-range encoder. 6) Social Motion Transformer (SoMoFormer) [63] employs the joints of all people as queries and predicts their future motion in parallel.
Figure 4: Comparison of baseline methods on CMU-Mocap [1] and our dataset. We compute the 1s mean local pose prediction errors. The baseline methods are trained and tested on two datasets separately.
Table 3: Performance comparison with the baseline methods. We compute the mean prediction errors of global pose, local pose, and root, respectively.
milliseconds	Global	Local	Root
400	600	800	1000	400	600	800	1000	400	600	800	1000
Frozen	119.2	165.7	207.3	244.5	62.8	82.8	98.9	112.1	99.8	139.3	175.0	207.6
Social-STGCNN [42]	188.2	246.4	298.8	346.5	62.8	82.8	98.9	112.1	173.1	225.6	273.7	317.9
SocialVAE [72]	84.0	127.2	171.7	215.1	62.8	82.8	98.9	112.1	52.6	89.7	130.3	170.8
HRI [40]	50.0	106.8	145.5	189.2	39.5	62.9	78.4	94.1	40.1	88.0	119.6	157.7
MRT [68]	66.9	103.2	140.2	176.4	49.4	68.8	85.4	99.5	50.6	79.0	109.3	140.1
SoMoFormer [63]	53.2	88.8	124.9	160.0	42.3	62.6	79.7	93.9	42.5	70.7	100.8	131.2
Ours	54.6	86.2	119.3	152.5	43.7	60.8	74.6	86.6	41.7	66.9	94.8	124.0
5.2.1 Quantitative results
We first benchmark three baseline methods to assess the motion prediction difficulty of our dataset in comparison to the commonly-used existing dataset [1]. Figure 4 reveals that for the CMU-Mocap dataset [1], merely repeating the last-frame pose (Frozen) leads to significantly lower errors than the model predictions. This finding indicates that the motions in this dataset are relatively "static", making it less challenging and informative for the social motion prediction task. In contrast, the dynamic nature of our dataset gives rise to considerably higher prediction errors of the naive Frozen baseline. The results demonstrate that the proposed dataset serves as a more convincing benchmark for social motion prediction and introduces more challenges to the field.
Furthermore, we conduct a quantitative evaluation of the proposed framework's performance and compare it with state-of-the-art approaches. As illustrated in Table 3, our method achieves competitive results with SOTA approaches. It is worth noting that the single-person method HRI [40] demonstrates respectable short-term motion prediction accuracy; however, its performance over longer horizons is constrained by the absence of global context, particularly in root trajectory prediction. In contrast, our approach significantly surpasses SOTA methods in long-term social motion prediction, highlighting its effectiveness in modeling long-range human interactions. In the subsequent section, we will demonstrate the critical role that social context plays in predicting long-term human motion via examples.
5.2.2 Qualitative results
In order to better understand the performance improvement of our method, we visualize its motion prediction results and compare with the baseline methods qualitatively. Figure 5 illustrates an representative result in which the input motion shows that a defensive player (highlighted) is switching the target of defense. Our approach correctly predicts its future direction and orientations as the offensive player at the other side will likely be covered by the teammate. In contrast, other approaches fail to reason this, predicting that the player would continue to move along the input trajectory or simply stop at the middle.
In Figure 6, three players to be noticed are highlighted in rose, purple and blue (we also call them this way below). Additionally, we visualize the actions produced by the intermediate policy networks to
Figure 5: Qualitative comparison with existing methods. Left two columns are past motion and right three columns are future motion. We highlight the prominent player with bright colors. Real motion are shown in green, and model predictions are shown in purple.
Figure 6: Actions produced by policy networks at different levels. Left two columns are past motions and right three columns are future motions.
interpret the decision-making process. Shown on the top row, player purple at level 1 policy network simply follows the past trend to receive the ball. As displayed in the second row, defender rose at level 2 gains knowledge from the first level that player purple is receiving the ball, and implements a steal attempt. Nevertheless, such aggressive actions could create defensive gaps. Defender blue at level 3 senses such gaps caused by defender rose from level 2, therefore executes cooperative defense upwards to fill the gap.
Finally, we conduct ablation studies to investigate the influence of each component. In Table 4 (a)-(e), we change the policy network depth  $K$ .  $K = 0$  corresponds to producing the final action directly without any recursive reasoning. We find that introducing the recursive reasoning process  $(K > 0)$  clearly lowers the prediction errors, proving the effectiveness of our framework design based on cognitive hierarchy. From (f) to (h), we keep  $K = 3$  and study the influence of training regularizations. (f) removes the GAIL regularization for intermediate levels  $1 \dots K - 1$ , and (g)
Table 4: Ablation study of cognition levels and architecture designs.
milliseconds	Global	Local	Root
400	600	800	1000	400	600	800	1000	400	600	800	1000
(a) K = 0	60.1	94.3	129.2	163.7	47.4	66.2	82.0	95.3	45.7	72.7	101.8	131.8
(b) K = 1	55.2	88.0	121.9	155.5	44.3	62.1	77.0	89.7	42.3	68.0	96.3	125.4
(c) K = 2	54.8	87.2	120.7	154.2	43.5	60.6	75.0	87.3	41.7	67.6	95.9	125.3
(d) K = 4	54.5	86.3	119.7	153.5	43.6	60.5	74.7	86.6	42.0	67.3	95.5	125.2
(e) K = 3, full	54.6	86.2	119.3	152.5	43.7	60.8	74.6	86.6	41.7	66.9	94.8	124.0
(f) w/o mid GAIL	54.9	87.5	121.4	155.4	43.7	60.4	74.5	86.4	41.8	67.8	96.6	126.9
(g) w/o GAIL	54.6	87.1	121.1	155.1	43.4	60.4	74.7	86.8	41.9	68.1	97.0	127.0
(h) w/o weight sharing	57.1	91.0	125.8	160.2	44.4	62.2	77.1	89.7	43.6	70.3	99.5	129.5
Table 5: Quantitative comparison of actions produced by policy networks at different levels.
milliseconds	Global	Local	Root
400	600	800	1000	400	600	800	1000	400	600	800	1000
Level-1	76.8	122.4	170.3	217.7	50.1	69.0	85.2	99.1	61.4	100.4	143.2	186.9
Level-2	61.0	99.9	144.6	192.5	47.1	66.0	82.5	96.6	46.5	78.0	116.2	158.8
Level-3	54.6	86.2	119.3	152.5	43.7	60.8	74.6	86.6	41.7	66.9	94.8	124.0
completely removes GAIL. Training without GAIL increases the long-term trajectory prediction errors, while slightly lowers the local pose prediction error. We also observe that models without GAIL tend to produce motions that appear less natural (e.g. limb twisting), leading to inferior visual effects. In (h), we use different policy network instances for levels  $1 \ldots K$ , and it turns out that sharing the policy networks leads to better generalization.
In addition to the qualitative visualization of actions generated by policy networks at different reasoning levels, we also provide a quantitative comparison of their distance from ground-truth. Table 5 demonstrates that policy networks at different levels learn different actions, and policy networks at higher reasoning levels produce actions with less prediction errors. We hypothesize that, through the proposed recursive reasoning process, policy networks at different levels display a range of cognitive types, with behavior spanning from relatively naive to substantially rational. Higher-level policies more closely approximate an equilibrium strategy, which is usually in alignment with expert demonstrations. We provide several example motion sequences in the video demo to further illustrate this observation.
In this work, we propose the first large-scale multi-person 3D motion dataset featuring strategic human social interactions. This dataset surpasses the existing ones in scale, diversity, dynamics, and interaction, thus posing new challenges to the social motion prediction problem. We reformulate the problem using a MARL perspective. Building on this, we propose a framework that effectively combines behavioral cloning (BC), generative adversarial imitation learning (GAIL), and cognitive hierarchy. Our approach demonstrates strong generalization capabilities and improved interpretability for modeling strategic human social interactions.
Limitations: Our approach is based on the cognitive hierarchy assumption and may not hold true for all type of social interactions. This work do not explicitly model social roles, which can be explored as future work.
Acknowledgment This work was partially supported by National Key R&D Program of China (2022ZD0114900). We thank Xuesong Fan and Rujie Wu for their generous support during the data collection process.
[74] Hang Ye, Wentao Zhu, Chunyu Wang, Rujie Wu, and Yizhou Wang. Faster voxelpose: Real-time 3d human pose estimation by orthographic projection. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VI, pages 142-159. Springer, 2022.
[75] Cunjun Yu, Xiao Ma, Jiawei Ren, Haiyu Zhao, and Shuai Yi. Spatio-temporal graph transformer networks for pedestrian trajectory prediction. In European Conference on Computer Vision, pages 507-523. Springer, 2020.
[76] Wentao Zhu, Xiaoxuan Ma, Dongwoo Ro, Hai Ci, Jinlu Zhang, Jiaxin Shi, Feng Gao, Qi Tian, and Yizhou Wang. Human motion generation: A survey. arXiv preprint arXiv:2307.10894, 2023.
A.1.1 Data Collection
We record basketball drill videos from 11 camera views at a frame rate of 25 FPS. Multi-person 3D poses are estimated using Faster-VoxelPose [74]. To obtain 2D heatmaps, we first resize the image to  $960 \times 512$  pixels and employ the ResNet [24] backbone. The 3D space size is set to  $16m \times 16m \times 3m$ , with the granularity of human detection networks at  $80 \times 80 \times 20$ . To estimate the joint locations of each subject, we set the human bounding box size to  $2m \times 2m \times 2.4m$  and use a voxel size of  $64 \times 64 \times 64$ . We further apply JDETracker [69] with a tracking frame rate of 25FPS and a threshold of  $0.2m$ . Next, we conduct an automatic failure detection process. We first discard motion clips with incorrect person counts and then filter outliers with root position changes exceeding a  $0.4m$  threshold compared to neighboring frames. Subsequently, we adopt the one-euro filter [11] for temporal smoothing. We set different parameters for different body joints because they exhibit different moving patterns. For upper limb joints such as wrists and elbows, we set  $f_{\mathrm{cutoff}} = 0.0287$  and  $\beta = 0.5$ ; for lower body joints including ankles and knees, we set  $f_{\mathrm{cutoff}} = 0.0274$ ,  $\beta = 1$ ; and for the rest of the body, we set  $f_{\mathrm{cutoff}} = 1$ ,  $\beta = 1$ . Finally, we perform a manual check to ensure the correctness of the data sequences.
A.1.2 Data Analysis
Figure 7: Clip length distribution comparison between our dataset and CMU-Mocap [1].
We visualize the clip lengths of our dataset and compare them with CMU-Mocap [1]. Figure 7 shows that our dataset features a longer overall duration and a larger number of long sequences.
Figure 8: Distribution comparison between our dataset and CMU-Mocap [1]. (a): We plot the joint angle distribution over the entire dataset. (b): We calculate the joint angle standard deviation for each motion sequence, then plot the std distribution.
Furthermore, we assess pose diversity and motion intensity utilizing joint angle distributions. As Figure 8(a) shows, our dataset encompasses a broader range of plausible body joint angles, indicating a
superior pose diversity. We then investigate motion intensity using intra-motion variation. Figure 8(b) illustrates that motion sequences in our dataset have higher joint angle variances, indicating more dynamic motions.
We represent human motion with 3D skeleton sequences following previous works [68, 48, 55]. Specifically, each  $x_{t}^{p}\in \mathbb{R}^{J\times 3}$  represents a 3D human pose for person  $p$  at timestep  $t$ , containing the Cartesian coordinates of  $J$  skeleton joints.
We apply Discrete Cosine Transform (DCT) [4] to the input motions and Inverse Discrete Cosine Transform (IDCT) for the decoded actions following [40, 68]. We employ a local-range Transformer encoder and a global-range Transformer encoder [68] as the state encoders. We utilize Transformer decoders as the policy networks. We concatenate the local state features  $s_l^p$  with global state features  $s_g$  (for level-0) or joint agent actions at the previous level  $\mathbf{a}_{(k - 1)}$  (for level-  $k$ ,  $k \geq 1$ ). For GAIL training, we implement a discriminator  $D$  that takes a pair of state  $s$  and joint actions  $\mathbf{a}$  as input and aims to classify whether the actions are from the policy network or expert demonstrations. The discriminator consists of a global-range Transformer encoder, a temporal average pooling layer, and an MLP with one hidden layer. We implement the proposed framework with PyTorch [47] using a Linux machine with 1 NVIDIA V100 GPU. We set  $\lambda = 0.002$ , batch size 32, learning rate 0.0001, and train for 60 epochs using Adam [33] optimizer.
Table 6: Ablation study of  $\lambda$  .
milliseconds	Global	Local	Root
400	600	800	1000	400	600	800	1000	400	600	800	1000
λ = 0.01	61.3	94.1	127.5	160.2	50.3	69.3	84.9	97.4	45.6	71.1	99.2	128.0
λ = 0.005	54.8	86.9	120.5	154.6	43.8	61.1	74.9	87.4	41.8	67.4	97.6	126.2
λ = 0.002	54.6	86.2	119.3	152.5	43.7	60.8	74.6	86.6	41.7	66.9	94.8	124.0
λ = 0.001	54.3	86.1	119.5	153.5	43.1	60.0	74.2	86.1	41.4	66.8	95.2	125.3
λ = 0.0002	54.2	86.5	120.2	154.1	43.0	59.9	74.2	86.2	41.6	67.6	96.2	126.1
We provide an additional ablation study on loss weight  $\lambda$  in Table 6.
A.4 Discussions
Broader Impacts Our approach does not rely on explicit biometric markers like unique fingerprints, retina scans, or body markers. The skeletal structure representation utilized in our dataset is based on generalized joint positions and movements. Our approach should only be employed with legally obtained human motion data. Meanwhile, our approach may contribute to more interpretable decision-making processes and holds potential for applications in sports tactical analysis and education.
Assets To collect our dataset, we acquire written consent from the participating athletes and coaches. We plan to release the processed 3D human motion sequences under a license agreement that permits their use for non-commercial scientific research purposes.
Human Subjects We inform the participating athletes about the data collection process and its intended uses, ensuring that we obtain their written consent. The instructions are provided by their coach, and the collected drills are part of their regular training routines. All participants receive compensation in accordance with their standard practices. To ensure privacy, we have implemented rigorous measures. Our data release strategy does not involve disclosing raw video data that may contain identifiable information, such as facial features. Instead, we will exclusively release processed 3D human motion sequences derived from the video data, guaranteeing the anonymity of the athletes.