Multi-Person 3D Motion Prediction with Multi-Range Transformers
Authors: Jiashun Wang
Huazhe Xu
Medhini Narasimhan
Xiaolong Wang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Motion Prediction
Multi-Person Interaction
Transformer

Abstract: We propose a novel framework for multi-person 3D motion trajectory prediction. Our key observation is that a human's action and behaviors may highly depend on the other persons around. Thus, instead of predicting each human pose trajectory in isolation, we introduce a Multi-Range Transformers model which contains of a local-range encoder for individual motion and a global-range encoder for social interactions. The Transformer decoder then performs prediction for each person by taking a corresponding pose as a query which attends to both local and global-range encoder features. Our model not only outperforms state-of-the-art methods on long-term 3D motion prediction, but also generates diverse social interactions. More interestingly, our model can even predict 15-person motion simultaneously by automatically dividing the persons into different interaction groups.  Project page with code is available at https://jiashunwang.github.io/MRT/.

Multi-Person 3D Motion Prediction with Multi-Range Transformers
Jiashun Wang'
Huazhe Xu²
Medhini Narasimhan Xiaolong Wang'
UC San Diego
UC Berkeley
ji#0770ucsd. edu
(huazhe_xu_ _x, Binazzee_xumeedhnnee medhini) -berkeleebberreyee edu xis0120eng ucsd. edu
Abstract
We propose a novel framework for multi- pertiipeesonn 3D motion trajectory prediction. Our key observation is that a human's 's action and behaviors may highly depend on the other persons around. Thus, instead of predicting each human pose trajectory in isolation, we introduce Mulri-Range Transformers model which contains of a local-range encoder for individual motion and a global-range encoder for social interactions. The Transformer decoder then performs prediction for each person by taking a corresponding pose as a query which attends both local and global-range encoder features. Our model not only outperforms long-term 3D motion prediction, but also generates diverse social interactions. model can even predict 15-person motion simultancously
-ffe-of-thheart methods on
More interestingly, by automatically dividing the persons into different interaction groups. Project page with code is available at bettps:/jjaahnueee
gi thub. 1o/MRT/.
Introduction
Given a few time steps of human motion, able forecast how the person will continue to move and imagine the complex dynamics of their motion in the future. The ability to perform such predictions allows human motion is an essential component for many real world computer vision applications such as surveillance systems, and collision avoidance for robotics and autonomous vehicles. The research on 3D human motion prediction has caught a lot of attention in recent years [44, 43], where deep models are designed to take few steps of 3D motion trajectory inputs and predict long-term future 3D motion trajectory as the outputs.
react and plan own behaviors. Similarly, a predictive model for
While encouraging results have been shown in previous work, most of the research focus on single human 3D motion prediction. Our key observation is that, how a human acts and behaves may highly depend on the people around. Especially during interactions with multiple agents, agent will need to predict the other agents' intentions, and then respond accordingly [54]. Thus instead predicting each human motion in isolation, we propose to build a model predict multi-person 3D motion and interactions. Such model needs the following properties: (i) understand each agent's 's own motion in previous time steps to obtain smooth and natural future motion; (ii) within a crowd of agents, understand which agents are interacting with each other and learn to predict based on the social interactions; (iii) the time scale for prediction needs to be long-term.
In this paper, we introduce Multi-Range Transformers for multi-person 3D motion trajectory pre- diction. The Transformer
[63] has shown to be very effective in modeling long-term relations in language modeling [16] and recently in visual recognition [17]. Inspired by these encouraging results, propose to explore Transformer models for predicting long-term human motion (3 seconds into the future). Our Multi-Range Transformers contain a local-range Transformer encoder for each individual person trajectory, global-range Transformer encoder for modeling social interactions, and Transformer decoder for predicting each person's 's future motion trajectory in 3D.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Figure 1:
Our motion prediction results. Left: The last time step of the input sequence. Right: The predicted diverse and continuous motion with multi-person social interactions. We use different color indicate different persons and the darker color for the further in future for predictions.
Specifically, given the human pose joints (with 3D locations in the world coordinate) I-second time steps inputs, the local-range Transformer encoder processes each person's 's trajectory separately and focuses on the local motion for smooth and natural prediction. The global-range Transformer 3D pose joints across different persons and different time steps,
encoder performs self-attention and it automatically learns which persons that one person should be attending to model their social interactions. Our Transformer decoder will then take single human 3D pose in one time step as the query input and encoder features as the key and value inputs to compute attention for prediction. We perform prediction for different persons by using different query pose inputs. By using only one time step person pose as the query for the decoder instead of a sequence of motion steps, we create bottleneck force the Transformer to exploit the relations between different time steps and persons in the encoders, instead of just repeating the existing motion alone [43].
We perform experiments multiple datasets including CMU-Mocap [1], MuPoTS-3D [48], motion prediction in 3D (with 2 ~ 3 persons). Our method achieves significant improvement over tate-offtthe-ar approaches for long-term predictions and the gain enlarges as we increase the future prediction time steps from second to 3 seconds. Qualitatively, visualize that method can predict interesting behaviors and interactions between different persons while previous approaches will repeat the same poses as goes to further steps in the future. More interestingly, extend the task perform prediction with 9 15 persons by mixing the CMU-Mocap [1] and the Panoptic [24] datasets, We visualize part of the prediction results with multi-person interactions in Fig. 1. We show that Multi-Range Transformers can not only perform predictions with a crowd of persons, but also learn to automatically group the persons into different social interaction clusters using the attention mechanism.
3DPW [64] for
2 Related Work
3D Motion Prediction. Predicting future human pose in 3D has been widely studied with Recurrent
Neural Networks (RNNs) [15, 31, 41, 46, 51, 20, 22, 26, 72, 21]. For example, Fragkiadaki er al. [19] propose Mencoder-Reecuren--Deee Decoder (ERD) model which incorporates nonlinear encoder and decoder networks before and after recurrent layers. Besides using RNNs, temporal convolution networks have also show promising results on modeling long-term motion [37, 28, 10, 10, 44, 43, 5]. For example, Li et al. [37] use a convolutional long-term encoder to encode the given history motion into hidden variable and then use decoder to predict the future sequence. While these approaches show encouraging results, most of these studies fix the pose center and ignore the global body trajectory. Instead of solving two problems separately, recent works start looking into jointly predict human pose and the trajectory in the world coordinate [70, 73, 71, 65, 11]. For example, Cao er al. [11] introduce to predict human motion under the constraint of 3D scene context. Our work also predicts the human motion with both 3D poses and the trajectory movements at the same time. Going beyond single human prediction, predict multi-human motion and interaction.
Social interaction with multiple persons. Multi-person trajectory prediction has been long standing problem in decades [25, 47, 67, 52, 75, 7, 6, 23, 18, 49, 36, 42, 56, 8, 34, 61, 69, 62].

---

For example, Alahi er al. [7] present LSTM [27] model which jointly reasons across multiple individuals scene. Gupta al. [23] propose to predict socially plausible futures with Generative Adversarial Networks. However, most of these approaches focus on the global movement of the humans. To model more fine- one-grained human-hume human interactions, recent research have proposed to predict multi-person poses and trajectories at the same time [58, 57, 2J. For example, Shu et al. [58] propose a MCMC based algorithm automatically discovers semantically meaningful interactive social affordance from RGB-D videos. Adeli et al. [2] introduce to combine context constraints in multi- person motion prediction. Inspired by these works, we propose a novel Mante-Range Transformers model which scales up the long-term) prediction with even more than 10 persons.
Transformers. Transformer is first introduced by Vaswani et al. [63] and has been widely applied in language processing [16, 55, 9] and computer vision [66, 29, 50, 14, 35, 17, 12, 13, 60, 68, 59, 45, 39, 53]. For example, Dosovitskiy el al. [17] introduce that using Transformer alone can lead to competitive performance in image classification. Beyond recognition tasks, Transformers have also been applied in modeling the human motion [11, 40, 38, 5, 43). Aksan el al. [5] use space-time self-attention mechanism with skeleton joints synthesize long-term motion. Mao al. [43] propose to extract motion attention to capture the similarity between the current motion and the historical motion. Beyond modeling single human motion, we introduce multi-range encoders to combine local motion trajectory and global interactions, and a decoder to predict multi-person motion.
3 Method
3.1
Representation
Given a scene with N persons and their corresponding history motion, our goal is predict their future 3D motion. Specifically, given X = [a -.,,%] representing the history motion of person where n = 1, ...; N, and l: is the time step. We aim predict the future motion X K where T represents the end of the sequence. We use vector 2t E R3J containing the Cartesian coordinates of the J skeleton joints to represent the pose of the person at time step k. In contrast to most previous motion prediction works which center the pose (joint positions) at the origin, we instead use the absolute joint positions in the world coordinate. In our method, 2r contains both the trajectory and the pose information. For simplicity, we omit subseript n when n only represents an arbitrary person, c.g., taking 2T Ti:k-
3.2 Network Architecture
We propose a Multi-Range Transformers architecture, which allows each person to query other persons' and their own history of motion for generating socially and physically plausible future motion in 3D. The network architecture is shown in Fig. 2. The proposed architecture is composed of a motion predictor P and motion discriminator D. In the predictor P, two Transformer-base encoders encode the individual (local) and global motion separately and one Transformer-basee decoder decodes smooth and natural motion sequence.
To ensure the smoothness of the motion, the model requires dense sampling of the input sequence. We apply our local-range encoder to each individual's 's motion to achieve this. For modeling the interaction of all the persons in the whole scene, we apply our global-range encoder to their motions, which performs relatively sparse sampling of the sequences. And this only need to be calculated once for the whole scene.
The motion discriminator D is Transformer-basee classifier to determine whether the generated motion is natural. We apply Discrete Cosine Transform (DCT) [4, 44] to encode the inputs for the encoders and the Inverse Discrete Cosine Transform (IDCT) for the decoder outputs. We introduce the architectures for each component as following.
3.2.1 Local-range Transformer Encoder
When predicting one person's motion, we first use our Local-range Transformer encoder to process this person's 's history motion. We use offset Ar; Ti+1 x; between two time steps to represent the motion. We apply DCT and a linear layer to Ar and then add the sinusoidal positional embedding [63] to get the local motion embedding for k time steps 41.6 [l...., 4e). We concatenate
3

---

Transformer Decoder
Local-range Transformer Eicoder
Transforner
Decoder
Forward
Temporal positional
Lx
encodiigg
Global Globul-range Transforner Encoder
Spatial positional encoding
Motion
Discriminator
Attention
Encoder
Feature
Query
Figure 2: Network architecture. Individual input motion is sent to the Local-range Transformer Encoder and all the person's 's motions are sent to the Global-range Transformer Encoder. & represents concatenate and @ represents add. The encoded motion features together with the query person skeleton for the Transformer Decoder. The output the future motion prediction results. On the right, show the architecture of the Transformer decoder. The encoder architecture is similar with the decoder except that the query, key and value are from the same input.
used as the key and value
them as a set tokens ELoc = 1l,,...444 4| stack alternating layers in the local-range encoder and introduce the technique use in each layer.
4I and feed them to the Transformer encoder. There are
Firstly, a Multi-Head Attention is used for extracting the motion information,
MultiHead Q, K, V) [head]; i..; head,, W
@KKKiT V
where head; softmax(
Vdk
OO is a projection parameter matrix, dx is the dimension of the key and is the number of the heads use. We use self-attention and get the query Qioe. key Kiocs and value Vioe from Ei for each head,,
Qioc Eroo Eioe w.Q..) Ki
Erse
w-(Ka) loc
Viac Ejoc W.(.
(2)
where (((2.)) (=(K,)) (V3)) loe loe loc are projection parameter matrices. loc represents the local-range We then employ residual connection and the layer normalization techniques
architecture. We
further apply feed forward layer, again followed by a residual connection and a layer normalization following [63]. The whole process forms layer of local-range Transformer encoder. We stack L such Transformer encoders to update the local motion embedding and obtain the local motion feature e1.b [e....., Ex] as the output, with e; represents the feature for time step
3.2.2 Global-range Transformer Encoder
In the global-range Transformer encoder, we aim to encode all the N people's motion in the scene. In method, this only needs be calculated one time and then can be concatenated with any person's local motion feature to predict the correspond person's 's future motion. We first apply a linear layer each person's motion =T.s and plus the sinusoidal positional embedding get the global motion embedding 1:N lol .... D .... ... .... N for N persons in le time steps. We L layers of Transformers to encode the global motion embedding. We apply the Multi-head Attention mechanism similar Eq. the global embedding which calculated as,
Yyros Egioc (QQ))) glob
(
ytoo Eglos (K,
glob Vylos Eglob WVV.0) glob
(3)
where Egiob = 161 .... .... ..... NTT and w-(Q.)) w-(K.i) and w-(V,a) projection parameter matrices. glob represents the global-range. Then feed them to the normalization and reed-forward layers same as local-range Transformer encoder. After applying such L Transformer encoders to the global embedding, output and get the global motion feature flsN We concatenate the local 61 and global fl:N features together as H [e1, .... Ck. ft. .... ft.
get the output o!:N We add spatial positional encoding to the
JiN. ..... JNI. and feed them the decoder.

---

3.2.3 Positional Encoding
Temporal positional encoding. We apply the sinusoidal positional encoding [63] to the inputs in both local and global-range encoder. This technique is routine in many Transformer-base based methods for injecting information about the relative absolute temporal position to the models.
Spatial positional encoding. We propose spatial positional encoding (SPE) technique the outputs of the global-range Transformer encoder. Before forwarding to the Transformer decoder, we want provide the spatial distance between the query token I& and the tokens of every time step of each person 1:N Intuitively, the location information helps clustering different persons in different social interactions groups, especially in scene with crowd of persons. We calculate SPE as,
Titk
SPE(2 mb) = exp( 3J ||=," - I& l12)
(4)
where n = I,..., N,..,,,, = 1,.., and x 2g E RV. SPE(-) will explicitly calculate the spatial distance between two persons. the SPE (a:] 3.. will then add of (the output of the global-range encoder) respectively in order to get the global feature f1:
3.2.4 Transformer Decoder
We send the local-global motion feature H together with static human pose 2g at time step into the decoder. We use the similar Multi-head attention mechanism the Transformer encoders. But differently, we take the single pose as the query and use the feature from the encoders get keys and values. We also apply L layers of Transformer decoder. Specifically, we apply a linear layer to Ty and then get q in order to get the query Qáee, we get the key Kéec and value Vdee both from local-global motion feature H as,
Vaoc q" (' K (K.) Vine (W(V (V.)
dec
where ((Q..) w-(k.)) and (((V.)) are projection parameter matrices. At the end of the decoder, apply two fully connected layers followed by Inverse Discrete Cosine Transform (IDCT) [4, 44] and output an offset motion sequence [ArR, .... A which can easily lead to the future 3D motion trajectory 1.T The model outputs a sequence of motion directly instead pose each time and this design can prevent generating freezing motion [40]. Note also add residual connections and layer normalization between layers.
3.2.5 Motion Discriminator
The design of such encoder-decoder architecture helps predict the future motion. To ensure natural and continuous long-term motion, we use discriminator D to adversially train the Predictor P. The output motion 1 .. is given as input the Transformer encoder with the same architecture of the local-range encoder and we further use another two fully connected layers to predict values (1.0) representing that & are real fake poses. We use the ground-truth future poses to provide as the positive examples. We train the predictor P and discriminator 'D jointly.
3.3 Training and Inference
We train our predictor P with both the reconstruction loss and the adversarial loss,
Lp = AveaL + Ladu
(6)
where Xade and Aree constant coefficients balance the training loss. We calculate the Lree and Lody as follows,
Lyec = T- Ac, - Arel? T k t=k
(7)
Lods T (D( - 112
We train our discriminator D following [33] with loss LD,
1
Lp T-k D( 11 T- (D()++a:)) 1
(8)
5

---

where 16+1;T is from the predicted motion and Uk+1-T is from the real motion. We train the discriminator that classifies the real ones as 1, where RT represents all the poses are natural.
We propose an efficient strategy which is progressively increasing the input sequence length during training and inference. Since Transformer-based encoders are used, there is no limit to the length of the input sequence. During training, we provide Fi:k to predict 15+1:26 (2t as the query), and we provide 21:24 to predict 1:34 (2x as the query) and so on. We sample different lengths of sequence in this way input during training. During inference, we predict the future motion in an auto rugoesegrees way. Given Ti:k. our model predicts 11-+1:2A. Then, given the input and our prediction results, we use them as inputs for our model again to predict 1:3+ and so on. The advantage of such design that when predicting longer motions, we still maintain the early motions as inputs to the model, instead of using a fixed length to predict each of the future time steps [44, 43], which may cause the loss of early interactive information. Through the experiment, we find this strategy could largely reduce the error accumulation.
4 Experiments
4.1 Datasets
We perform our experiments on multiple datasets. CMU-Mocap [1] contains high quality motion sequences of ~ 2 persons in each scene. Panoptic [24], MuPoTSS3D [48] and 3DPW [64] datasets are collected using cameras with pose estimation and optimization. Consequently, the estimated joint positions in the latter datasets contain more noise and are less smooth in comparison to CMU-Mocap. For multi-person data, there are about 3 ~ persons in each scene in Panoptic, 2 ~ 3 persons in MuPoTS- 3D each scene and 2 persons in 3DPW. It is worth noting that in the Panoptic dataset, most of the scenes are people standing and chatting. with small movement. The skeleton joint positions in 3DPW are obtained from moving camera so that there is more unnatural foot skating.
Dataset settings. We design two different settings to evaluate our method. The first setting consists of a small number of people (2 ~ 3). We use CMU-Mocap as our training data. CMU-Mocap contains a large number of scenes with a single person moving and small number of scenes with two persons interacting and moving. We sample from these two parts and mix them together as our training data. We make all the CMU-Mocap data consists of persons in each scene. We sample test set from CMU-Mocap in similar way. We evaluate the generalization ability of our model by testing on MuPoTS-33 3D and the 3DPW dataset with the model trained on the entire CMU-Mocap dataset.
The second setting consists of scenes with more people. For the training data, we sample motions from CMU-Mocap and Panoptic and then mix them. We integrate CMU-Mocap and Panoptic scenes to build single scene with more people. For the test data, we sample one version from both CMU- Mocap and Panoptic, namely Mix1. And we sample one version from CMU-Mocap. MuPoTS-3D and 3DPW, namely Mix2. There are 9 ~ 15 persons in each scene in Mix and 11 persons in Mix2. The positive motion data for discriminator is sampled from CMU-Mocap's s single person motion.
4.2 Implementation Details
In experiments, we give 1 second history motion (k= 15 time steps) input and recursively
predict the future 3 seconds (45 time steps) as Sec. 3.3 described. We use L = 3 alternating layers with 8 heads in each Transformer. We use Adam [32] as the optimizer for our networks. During training, we set 3 x 10 as the learning rate for predictor P and 5 x 10-4 as the learning rate for discriminator D. We set Xrea = and Xndo 5 x 10-4, For experiments with 2 ~ 3 persons, we set a batch size of 32 and for scene with more people, we set a batch size of 8.
4.3 Metrics and Methods for Comparisons
Metrics. We first use Mean Per Joint Position Error (MPJPE) [30] without aligning as the metric to compare the multi-person motion prediction results in 1, 2 and 3 seconds. Without aligning, MPJPE will reflect the error caused by both trajectory and pose. We also compare the root error and pose error (MPJPE with aligning) separately. The root error is the I.2 root joint position error. Further, we compare the distribution of the movement between the start and end of the outputs on different datasets. Specifically, we measure this movement by calculating the mean L2 distance between the

---

CMU-Mocap (3 persons)
MuPoTS-3D (2 persons)
3DW
Mix1
Mix2
method
persons)
15 persons)
(11 persons)
1s
s
1s
3s
1 s
2s
3s
1s
2s
3s
LTD [44
1.37 2.19
3.26
1.19
1.81
2.34
4.67
7.10
8.71
2.10
3.19
4.15
1.72
2.58
3.45
HRI [43]
1.49 2.60 3.07
0.94
1.68
2.29
4.07
6.32
8.01
1.80
3.14
4.21
1.60
2.71
3.67
PocialPool [2]
1.15 2.71
3.90
0.92
1.67
2.51
4.17
7.17
9.27
1.85
3.39
4.84
1.72
3.06
4.26
Ours w/o Local
1,42 2.20
2.99
1.28
2.10
2.78
3.96
5.94
7.75
2.09
3.34
4.34
1.85
2.92
3.83
Ours w/o Global
0.99 1.71
2.50
0.92
1.67
2.50
4.17
6.85
8.91
1.77
3.10
4.19
1.42
2.29
3.06
Ours w/o D
1.13 1.84
2.57
0.92
1.62
2.26
4.17
6.41
8.09
1.75
3.00
4.00
1.34
2.19
2.95
Ours w/o SPE
1.05
1.68
2.37
0.92
1.51
2.23
3.92
6.18
7.79
1.75
3.09
4.13
1.31
2.15
2.92
Ours
0.96
1.57
2.18
0.89
1.59
2.22
3.87
6.12
7.83
1.73
2.99
3.97
1.29
2.09
2.82
Table 1: MPJPE on different datasets. We compare our method with the previous SOTA methods and ablative baselines on predicting 1, 2 and 3 seconds motion. Best results are shown in boldface.
CMU-Mocap
MuPoTS-3D
3DPW
method
Root
Pose
Root
Pose
Root
Pose
Is
2s
3s
1s
2s
38
1s
2s
3s
1s
2s
38
1s
2s
3s
Is
2s
3s
LTD]44]
0.97 1.73
2.62
0.98
1.21
1.37
0.89
1.39
1.91
0.88
1.14
1.31
4.28
6.79
8.41
1.54
1.76
1.98
HRI [43]
0.96 2.06
3.11
1.05
1.37
1.58
0.66
1.30
2.16
0.73
1.07
1.30
3.67
6.42
8.64
1.43 1.75
1.94
SocialPool [2]
0.96 2.01 2.96
1.03
1.41
1.71
0.96
1.38
2.21
0.72
1.08
1.30
3.76
6,86 9.07
1.60
1.95 2.15
Ours
0.60 1.12 1.71 0.79 1.05 1.22
0.67
1.25 1.86
0.69
0.99
1.19
3.42
5.69 7.30
1.52 1.75 1.93
Table 2:
Root and pose error on different datasets. We compare our method with the previous SOTA methods on predicting 1, 2 and 3 seconds motion. Best results are shown in boldface.
start and end skeleton joint positions. This surrogate metric is designed measure whether method could predict a plausible movement, instead of standing and repeating. We also perform a user study using Amazon Mechanical Turk (AMT) to compare the realness of the predicted motion. We let the user to score each motion prediction from 1 to 5, with a higher score implying the video looks more natural inspired by [65, 74].
We argue that for multi-person motion prediction, joint positions are more relevant compared to angle- based representations, since most of the data suitable for this task is obtained by cameras and pose estimation, containing only the position representations. Another consideration for comparison method selection is many related studies only model the pose fixing at the origin, while we need to select methods that could predict absolute motion. We select two competitive state-of-the-art person motion prediction methods: LTD [44] is graph-based method and HRI [43] is an attention-based method. Both of them allow absolute coordinate inputs, which fits our task and settings well. Most relevant to our work is SocialPool [2], method which uses GRU [15] to model the motion sequences and proposes to use social pool to model the interaction. We remove the image input of it as well as the image feature, keeping the sequence to sequence and social pool structures. We use the same data to train these methods.
Methods for comparisons.
4.4 Evaluation Results
4.4.1
Quantitative Results
We report the MPJPE in 0.1 meters of 1 second, 2 seconds and 3 seconds predicted motion on CMU-Mocap, MuPoTS-3D and 3DPW, Mix and Mix2 respectively in Tab. 1. In both cases with a small number and a large number of people, our method achieves state-of-the-art performance for different prediction time lengths. We achieve up to 20% improvement when compared to the previous single-person person-basedddedde methods [44, 43] and achieve up to 30% improvement compared to the muultipeerson-eee method [2]. In SocialPool [2], the same global feature added to all the persons which interferes with the model's prediction for each individual, especially when there are large number of people. Because of this, SocialPool's 's performance even not as good as the previous single-persor person- based method. However, in our design the model can use the features corresponding to one person to query the global motion feature which automatically allows to use the motion information belonging to other persons. Therefore, our method can achieve good results on scenes with any number of people. We also compare the root error and pose error respectively in Tab. 2. The results are in 0.1 meters. Generally, method can predict the trajectory of the root and the
7

---

Dittr bution the movement CMU-Mocco
Distribution of the movement MuPoTS-3C
Distrioution of the movemment 3DPW
CIT
Ours
Ours
1.2
Our
Soria/Porl
SocialPeol
SocialPool
3-
HRI
HRI
1.3
HRI
LTD
LTD
0.3
2-
3-
0.5
0.4
1
0-
0.0
1
0.0
0.5
1.0
2.0
2.5
Figure 3: Distribution of the movement between the start and end of the predictions on different datasets. X-axis shows the moving distance. Ours is the most similar to the ground truth while the others intend to predict smaller movement. The model is only trained on CMU-Mocap.
method
CMU-Mocap
MuPoTS-3D
3DPW
Mix1
Mix2
LTD [44]
3.61
3.6610.93
3.75
3.79
3.75+0.90
HRT [43]
3.36+-0.96
3.59+1.25
3.76 0.72
3.67+00.9
3.714-0.90
SocialPool [2]
3.49++0.87
1.66
0.66
3.62++0.92
1.49
Ours
3.62+
3.68+0098
3.78+0082
3.74+0.83
3.77+-0.82
GT
3.7810.76
3.85
3.77=0.81
3.77-10.87
3.8811079
Table User study. We perform a user study using the AMT. We provide the average of the human evaluated score w.r.t. the average + the standard deviations. Best results are shown in boldface.
poses more accurately, especially for the root prediction. We will also introduce in the following experiments that previous methods often intend to predict shorter movement.
We show the distribution of the moving distance between the start and the end of the predictions in Fig. 3. It shows that other methods intend to predict motion with less movement, with much higher value in y-axis close to 0 in x-axis. We observe the results of other methods sometimes just stay in the same spatial location while the ground truth is moving with large distance. On the contrary, the distribution of our result is very similar to the ground truth on CMU-Mocap and even on completely unseen MuPoTS-30 and 3DPW datasets. The reason is that our Transformer encoders can take any time length motion as input for modeling the long-term motion. And using the decoder to output a Ar motion is also beneficial to model the position movement. For user study, report the average and the standard error of the score in Tab. 3. A higher average score means users think that the results more "natural looking" and a lower standard error indicates that the scores are more stable. Our results get better reviews consistently across all datasets.
4.4.2 Qualitative Results
We provide qualitative comparisons in Fig. 4. Our predictions are more natural and smooth while being close to the real record. It can be seen that RNN-based method (SocialPool [2]) will quickly produce freezing motion, which is consistent with the claims in [5, 40]. Another finding is that when predicting the absolute skeleton joint positions, decoding based on an input seed sequence (HRI [43]) or adding the input sequential residual (LTD [44]) to the output, will make the predicted motion have hysteresis and repeat the history. For example, in a forward motion, the prediction may jump back into temporally unreasonable position and then continue move forward. We argue the positions of the past sequences have a negative impact on the model's 's prediction using the previously proposed design. This also consistent with the conclusion we get in Fig. 3. However, our method, using a static pose as query and predicting A sequence, could solve this problem effectively.
4.4.3 Ablation Study
If
2f
4f
half
all
We perform ablation study different modules of our network. We provide the MPJPE re- sults on different datasets in Tab. 1. We move the local-range encoder, global-range encoder, discriminator and the spatial positional encoding respectively. After removing each module, the overall performance of the model has declined.
second
0.96
0.96
0.96
1,03
1.03
2 seconds
1.57
1.58
1.57
1.71
1.73
3 seconds
2.18
2.22
2.26
2.44
2.50
Table 4: MPJPE with different decoder query input length on CMU-Mocap. "f" denotes frame. "half" denotes half of the sequence and "all" denotes the whole sequence.
8

---

$
HRI
$
3R
SocialPool
$
8
Ours
Ingut
Octput
Figure 4: Qualitative comparison with other methods. Left two columns input and right three columns are outputs. Our result is the closest to the real record and the others fail to predict a walking motion and predict less accurate interaction motion. The input data is from CMU-Mocap.
We find local-range encoder could largely reduce the prediction error. We also prove the effectiveness of the global-range encoder and an adversial train- ing would improve the prediction performance. is worth noting that our spatial positional encoding performs better scene with more people, which is in line with our expectations.
1-layer
3-layer
6-layer
9-layer
sccond
1.02
0.96
1.07
2.69
2 scconds
1.71
1.57
1.79
4.71
3 seconds
2.47
2.18
2.58
6.83
Table MPJPE with different number of Transformer layers on CMU-Mocap.
second seconds 3 seconds
We report the MPJPE with different decoder query input length on CMU-Mocap in Tab. 4. We success- fully prove that using only single pose the query for the decoder instead of a sequence of motion could create bottleneck force the Transformer to learn to predict the future instead of repeating the existing motion. We further explore the suitable number of the Transformer layers. We conduct experiments us- ing 1, 3. and layers respectively on CMU-Mocap. The results in Tab. 5 show that --layer-Transforre suitable for both shorter and longer prediction. Fur-
fixed length 0.96
1.91
2.91
0.96
1.57
2.18
Table 6: MPJPE on CMU-Mocap of our method and fixed length input.
All DCT
No DCT
Ours
second
0.99
1.44
0.96
seconds
1.65
2.43
1.57
seconds
2.37
3.47
2.18
Table 7: MPJPE CMU-Mocap. Best re-
sults are shown in boldface.
thermore, prove the effectiveness of our strategy described in 3.3, conduct experiment using a fixed length (one second motion) as input to the encoders and we report the quantitative comparisons using MPJPE on CMU-Mocap in Tab. 6. It can be seen strategy is helpful in reducing the accumulation error, especially for long-term prediction.
We conduct ablation study on the role of DCT [4, 44]. In our method, we apply DCT to the input of local-range Transformer encoder and apply IDCT to the output of the decoder. Because DCT can provide more compact representation, which nicely captures the smoothness of human motion, particularly terms of 3D coordinates [44]. We train a model which we apply all the input with DCT and apply IDCT to the output of the decoder, namely "All DCT And further train a model without any DCT or IDCT, namely "No DCT We show the MPJPE on Mocap [1] in Tab. 7. We prove our design is more suitable for this task. Though DCT is quite helpful when predicting the human motion, from the experiment we find that for modeling the human interaction, position-bass bassed representation is more suitable.
4.5 Analysis of Attention
In this section, we discuss the attention results each person the decoder. In our model, the person in the last frame will query his/her past motion well the history motion of everyone. We give

---

Input
Query Person
Attention Score
Key, Valae from Local-nage
Res, Velae loo Ghibalaage
High
Pascel
Persat
Pasce
Figure 5: We show the input sequence, query person and the corresponding attention scores in the decoder. The dotted red box shows that the two interacting people have more similar attention score distribution. Darker pose color represents the further in future.
visualization of the attention score history motion of three persons in the scene. On the right, show different person's 's query and how the attention score looks like in the decoder. We separately normalize the scores for keys and values from the wocal-ragee encoder and the global-range encoder. Since the query q is only vector, so is the attention score. We concatenate the attention score vectors from each head together and draw them column represents different persons at different time steps. Keys and values are from [e1,... 6x] and fh. ff.... f2l respectively.
first layer of the decoder Fig. 5. On the we show
single frame 2ę to query,
matrix. Each row represents head for person and each
Fig. 5 shows that different query person could automatically weight different people at different time. Furthermore, we found that people in different interaction groups in scene will have more similar attention score distributions when querying the outputs from the global-range encoder. This shows that our method has the ability to automatically divide the persons into different interaction groups in the decoder without any label of this. On the other hand, querying the outputs from the local-range encoder usually have relatively fine-grained results. Specifically, the color changes more frequently. When querying outputs from the global range encoder, the color is relatively consistent one head. This is consistent with statements about the dense sampling and sparse sampling of the two encoders in Sec. 3.2.
Conclusion
In this paper, we propose a novel framework to predict multi-person trajectory motion. We design Multi-Range Transformers architecture, which encodes both individual motion and social interaction and then outputs a natural long-term motion with a correspond pose as the query in the decoder. Compared with previous methods, model can predict more accurate and natural multi-person 3D motion.
Broader Impact. The original intention of our research is to protect people's safety in surveillance systems, and collision avoidance for roboties and autonomous vehicles. However, we remain concerned about the invasion of people's privacy through the human motion and behavior. Since do not use the image of person as any input, it will not be easy to obtain the identity information of specific person and we pleased to this. But we are still concerned about whether it is possible to identify a person based solely on the person's 's skeletons and motions.
6 Acknowledgments and Funding Transparency Statement
This work was supported, in part, by grants from DARPA LwLL, NSF CCF-2112665 (TILOS), NSF 1730158 CI-New: Cognitive Hardware and Software Ecosystem Community Infrastructure (CHASE-CI), NSF ACI-1541349 CC*DNI Pacific Research Platform, and gifts from Qualcomm, TuSimple and Picsart.
10

---

References
[1] Cmu graphics lab motion capture database. http://mocap.. cmu. edu/. 2, 6,9
[2] Vida Adeli, Ehsan Adeli, lan Reid, Juan Carlos Niebles, and Hamid Rezatofighi. Socially and contextually aware human motion and pose forecasting. IEEE Robotics and Automation Letters, ((4):6033-604) 2020. 7,
[3] Vida Adeli, Mahsa Ehsanpour, Ian Reid, Juan Carlos Niebles, Silvio Savarese, Ehsan Adeli, and Hamid Rezatofighi. Tripod: Human trajectory and pose dynamics forecasting in the wild. arXiv preprint arXiv:2104.0400 2021. 3
[4] Nasir Ahmed, T_ Natarajan, and Kamisetty R Rao. Discrete cosine transform. JEEE transactions on Computers, 100(1):90-9)) 93, 1974. 3,
[5] Emre Aksan, Peng Cao, Manuel Kaufmann, and Otmar Hilliges. Attention, please: A spatio-temporal transformer for 3d human motion prediction. arXiv preprint wXXiv2204.00099) 2020. 2,
[6] Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social Istm: Human trajectory prediction crowded spaces. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 961-971, 2016. 2
[7] Alexandre Alahi, Vignesh Ramanathan, and Li Fei-Fei. Socially aware large-scale crowd forecasting, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2203-2210, 2014.
2,
[8] Javad Amirian, Jean- Bernard Hayet, and Julien Pettéé Social ways: Learning multi- modal distributions of pedestrian trajectories with gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0- 0, 2019. 2
[9] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint a005v:2005. /4165, 2020. 3
[10] Judith Butepage, Michael J Black, Danica Kragic, and Hedvig Kjellstrom. Deep representation learning for human motion prediction and classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4158-6166, 2017. 2
[11] Zhe Cao, Hang Gao, Karttikeya Mangalam, Qi-Zhi Cai, Minh Vo, and Jitendra Malik. Long-term human motion prediction with scene context. In European Conference on Computer Vision, pages 387-404. Springer, 2020. 2,
[12] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference Computer Vision, pages 213 229. Springer, 2020.
[13] Mark Chen. Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pages 1691-1703. PMLR, 2020.
[14] Rewon Child, Scott Gray, Alee Radford, and llya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509 2019.
[15] Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv: /409. /259, 2014. 2,
[16] Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint
2018. 1,
[17] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1192 2020. 1,3
[18] Tharindu Fernando, Simon Denman, Sridha Sridharan, and Clinton Fookes. Soft+ hardwired attention: An Istm framework for human trajectory prediction and abnormal event detection. Neural networks, 108:466-478, 2018.
[19] Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik. Recurrent network models for human dynamics. In Proceedings of the IEEE International Conference on Computer Vision, pages 4346-4354 2015.
11

---

[20] Partha Ghosh. Jie Song, Emre Aksan, and Otmar Hilliges. Learning human motion models for long-term predictions. In 2017 International Conference 3D Vision (3DV), pages 458-466. IEEE, 2017.
[21] Anand Gopalakrishnan, Ankur Mali, Dan Kifer, Lee Giles, and Alexander G Ororbia, A neural temporal model for human motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 121 12116-12125, 2019.
[22] Liang-Yan Yan Gui, Yu- Yu-Xiong Wang, Xiaodan Liang, and José MF Moura. Adversarial geometry awwae human motion prediction. In Proceedings of the European Conference on Computer Vision (ECCV), pages 786-803, 2018.
[23] Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks, In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 2255-2264, 2018. 2,3
[24] Lei Tan Lin Gui Bart Nabbe lain Matthews Takeo Kanade Shohei Nobuhara Hanbyul Joo, Hao Liu and Yaser Sheikh. Panoptic studio: A massively multiview system for social motion capture. In ICCV, 2015. 2,
[25] Dirk Helbing and Peter Molnar. Social force model for pedestrian dynamics. Physical review E, (55)4222
1995. 2
[26] Alejandro Hernandez, Jurgen Gall, and Francese Moreno-Noguer. Human motion prediction via spatio- temporal inpainting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 71 7134-7143, 2019. 2
[27] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 183):1735- 1997. 3
[28] Daniel Holden, Jun Saito, Taku Komura, and Thomas Joyce. Learning motion manifolds with convolutional
autoencoders. In SIGGRAPH Asia 2015 Technical Briefs, pages 1-4. 2015. 2
[29] Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, and Yichen Wei. Relation networks for object detection.
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3588-3599
2018. 3
[30] Catalin lonescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions on pattern analysis and machine intelligence, 36(7):1325-133) 2013.
[31] Ashesh Jain, Amir R Zamir, Silvio Savarese, and Ashutosh Saxena. Structural Deep learning on seatio-temporal graphs. In Proceedings of the ieee conference on computer vision and pattern recognition. pages 133085317. 2016. 2
[32] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6940 44860990.. 2014.
[33] Muhammed Kocabas, Nikos Athanasiou, and Michael J. Black. Vibe: Video inferenee for human body pose and shape estimation. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
[34] Vineet Kosaraju, Amir Sadeghian. Roberto Martin-Martía, lan Reid, S Hamid Rezatofighi. and Silvio Savarese. Social-bigat: Multimodal trajectory forecasting using bicycle-gan gan and graph attention networks. arXiv preprint 1xXiv:9907.03395 2019. 2
[35] Juho Lee, Yoonho Lee, Jungtack Kim, Adam Kosiorek, Seungjin Choi, and Yee whye Teh. Set transformer: A framework for attention based cntuutaationiivvriinnt neural networks. In International Conference on Machine Learning, pages 3744-3753. PMLR, 2019. 3
[36] Namhoon Lee, Wongun Choi, Paul Vernaza, Christopher B Choy, Philip HS Torr, and Manmohan Chan- draker. Desire: Distant future prediction in dynamic scenes with interacting agents. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 336-345, 2017. 2
[37] Chen Li, Zhen Zhang. Wee Sun Lee, and Gim Hee Lee. Convolutional sequence to sequence model for human dynamics. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages
2018. 2
[38] Jiaman Li, Yihang Yin, Hang Chu, Yi Zhou, Tingwu Wang. Sanja Fidler, and Hao Li. Learning to generate diverse dance motions with transformer. arXiv preprint 2rXiv:2008.08177, 2020.
12

---

[39] Ke Li. Shijie Wang, Xiang Zhang, Yifan Xu. Weijian Xu, and Zhuowen Tu. Pose recognition with cascade transformers. arXiv preprint arXiv:2104.066 2021.
[40] Ruilong Li, Shan Yang. David A. Ross, and Angjoo Kanazawa. Learn to dance with aist++; Music conditioned 3d dance generation, 2021. 3, 5,8
[41] Zimo Li, Yi Zhou, Shuangju Xiao, Chong He, Zeng Huang, and Hao Li. Auto-condiiioned recurrent networks for extended complex human motion synthesis. arXiv preprint arXiv: 170.00533, 2017. 2
[42] Karttikeya Mangalam, Harshayu Girase, Shreyas Agarwal, Kuan-Hui Lee, Ehsan Adeli, Jitendra Malik, and Adrien Gaidon. It is not the journey but the destination: Endpoint conditioned trajectory prediction. In European Conference on Computer Vision, pages 336-776. Springer, 2020. 2
[43] Wei Mao, Miaomiao Liu, and Mathieu Salzmann. History repeats itself: Human motion prediction via motion attention. In European Conference on Computer Vision, pages 489. Springer. 2020. 1, 2, 3, 6, 7, 8
[44] Wei Mao, Miaomiao Liu, Mathieu Salzmann, and Hongdong Li. Learning trajectory dependencies for human motion prediction. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 9489 9497, 2019. 5, 8,8,9
[45] Weian Mao, Yongtao Ge, Chunhua Shen, Zhi Tian, Xinlong Wang, and Zhibin Wang- Tipose: Direct human pose estimation with transformers. arXiv preprint arXiv:2103. 153200 2021. 3
[46] Julieta Martinez, Michael J Black, and Javier Romero. On human motion prediction using recurrent neural networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 2891-2900, 2017. 2
[47] Ramin Mehran, Alexis Oyama, and Muharak Shah. Abnormal crowd hchavior detection using social force model. 2009 IEFE Conference on Computer Vision and Pattern Recognition, pages 935-942, IEEF, 2009. 2
[48] Dushyant Mchta, Oleksandr Sotnychenko, Franziska Mueller, Weipeng Xu, Srinath Sridhar, Gerard Pons- Moll, and Christian Theobalt. Single-shot multi-person 3d pose estimation from monocular rgb. In 3D Vision (3DV), 2018 Sixth International Conference on. IEEE, sep 2018. 2,
[49] Ahduallah Mohamed, Kun Qian, Mohamed Fllhoseiny, and Christian Claudel. Social-stgenm: A social spatio-temporal graph convolutional neural network for human trajectory prediction. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14424-14432, 2020. 2
[50] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran, Image transformer. In International Conference Machine Learning, pages 4055-4064 PMLR, 2018. 3
[51] Dario Pavllo, David Grangier, and Michael Auli. Quatemet: A quaternion based recurrent model for human motion. arXiv preprint arXiv: 1805.804855555 2018.
[52] Stefano Pellegrini, Andreas Ess, Konrad Schindler, and Lue Van Gool. You'll never walk alone: Modeling social behavior for multi-targe tracking. In 2009 IEEE 12th International Conference on Computer Vision, pages 261-268. IEEE, 2009. 2
[53] Mathis Petrovich, Michael J Black, and Gül Varol. Action-conditioned 3d human motion synthesis with transformer vae. arXiv preprint arXiv:2104.05670. 2021. 3
[54] David Premack and Guy Woodruff. Does the chimpanzee have a theory of mind? Behavioral and brain sciences, 1(4):515-526, 1978.
[55] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 18))99 2019. 3
[56] Amir Sadeghian, Vineet Kosaraju, Ali Sadeghian, Noriaki Hirose, Hamid Rezatofighi, and Silvio Savarese. Sophie: An attentive gan for predicting paths compliant social and physical constraints. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 1349-1358, 2019. 2
[57] Tianmin Shu, Xiaofeng Gao, Michael S Ryoo, and Song- Chhn Zhu. Learning social affordance grammar from videos: Transferring human interactions to human robot interactions. In 2017 IEEE international conference robotics and automation (ICRA), pages 1669-1676 IEEE, 2017.
13

---

[58] Tianmin Shu, Michael S Ryoo, and Song-Chun Zhu. Learning social affordance for human-robot interaction. arXiv preprint woXiv:1004.0362, 2016. 3
[59] Lucas Stoffi, Maxime Vidal, and Alexander Mathis. End-to-end trainable multi-instance pose estimation with transformers. arXiv preprint arXiv:2103. 12111, 2021. 3
[60] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7464-7473, 2019.
[61] Jianhua Sun, Qinhong Jiang, and Cewu Lu. Recursive social behavior graph for trajectory prediction. In Proceedings of the IEEE/CVF Conference un Computer Vision and Puttern Recognition, pages 660-669, 2020. 2
[62] Jianhua Sun, Yuxuan Li, Hao-Shu Fang, and Cewu Lu, Three steps to multimodal trajectory prediction: Modality clustering, classification and synthesis. arXiv preprint arXiv:2103.07855 2021.
[63] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv: 2017. 1, 3,3,,, 5,
[64] Timo von Mareard, Roberto Henschel, Michael Black, Bodo Rosenhahn, and Gerard Pons-Moll, Recover- ing accurate 3d human pose in the wild using imus and a moving camera. In European Conference on Computer Vision (ECCV), sep 2018. 2,
[65] Jiashun Wang, Huazhe Xu, Jingwei Xu, Sifei Liu, and Xiaolong Wang. Synthesizing long-term 3d human motion and interaction in 3d scenes. arXiv preprint arXiv:2012.0555 2020. 2,7
[66] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 7794-7803, 2018.
[67] Kota Yamaguchi, Alexander C Berg, Luis E Ortiz, and Tamara L Berg. Who are you with and where are you going? In CVPR 2011, pages 1345-1352. IEEE, 2011. 2
[68] Sen Yang, Zhibin Quan, Mu Nie, and Wankou Yang. Transpose: Towards explainable human pose estimation by transformer. arXiv preprint 2xX1v:2012.14214, 2020.
[69] Cunjun Yu, Xiao Ma, Jiawei Ren, Haiyu Zhao, and Shuai Yi. Spatio-temporal graph transformer networks for pedestrian trajectory prediction. In European Conference Computer Vision, pages 507-523. Springer, 2020. 2
[70] Ye Yuan and Kris Kitani. Diverse trajectory forecasting with determinantal point processes. arXiv preprint arXiv:I907.0443 94770497777 2019.
[71] Ye Yuan, Shih-En Wei, Tomas Simon, Kris Kitani, and Jason Saragih. Simpoe: Simulated character control for human pose estimation. arXiv preprint (tti::21440000 2021.
[72] Jason Y Zhang, Panna Felsen, Angjoo Kanazawa, and Jitendra Malik. Predicting 3d human dynamics from video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7114-7123, 2019. 2
[73] Yan Zhang, Michael Black, and Siyu Tang. We are more than our joints: Predicting how 3d bodies move. arXiv preprint arXiv:2012.00619, 2020. 2
[74] Yan Zhang. Mohamed Hassan, Heiko Neumann, Michael J. Black, and Siyu Tang. Generating people in scenes without people. In Computer Vision and Pattern Recognition (CVPR), June 2020.
[75] Bolei Zhou, Xiaogang Wang, and Xiaoou Tang. Understanding collective crowd hehaviors: Learning a mixture model of dynamic pedestrian-agent In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 2871-2878 IEEE, 2012. 2
14


Reviews:
[{"review_id": "Xm5GogtGnZ", "reviewer": "Reviewer_yyef", "summary": "They propose a novel framework for multi-person 3D motion trajectory prediction. The multi-Range Transformer model contains a local-range encoder for individual motion and a global-range encoder for social interactions. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: Based on Transformer, this paper proposes Multi-Range Transformer model multi-person 3D motion trajectory prediction. This is a novel task. The proposed method also has some specially designed architecture for this task.  \n\nQuality: Multi-Range Transformer achieves a significant improvement over state-of-the-art approaches for long-term predictions, and the gain enlarges as they increase the future prediction time steps from 1 second to 3 seconds.  \n\nClarity: The method is clearly explained.  \n\nSignificance: Most of the previous work focus on single human 3D motion prediction. Multi-person motion prediction is a novel task. This paper also observes how a human acts and behaves highly depend on the people around. \n\n \n\nTypos: \n\nLine 23: “acts and behaves ma highly” -> “acts and behaves may highly” ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. What about the long-term (longer than 3 seconds into the future) motion prediction results? \n2. Previously, from my experience, position encoding is not critical for the motion prediction task. Is the designed Temporal position encoding & Spatial position encoding critical? Could you perform an ablation study to see the effect of  \n1) With Temporal position encoding & Spatial position encoding \n2) Only Temporal position encoding \n3) Only Spatial position encoding ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "RFMUFXrojso", "reviewer": "Reviewer_3D5S", "summary": "This work focuses on the 3D human motion prediction in scenes containing multiple agents. While prior work focused on single human 3D motion forecasting, this work focuses on jointly predicting the motion of all agents in a scene. To do this, this work proposes a novel Transformer-based architecture (named Multi-Range Transformer) which allows the future prediction of each agent to be based on the encoding of the entire past scene. The architecture consists of a local and global transformer encoder to encode the agent in question and all other agents' past, respectively. A Transformer decoder is employed to predict the future motion of the agent question. A discriminator  Transformer is used to assess the \"realness\" of a generated trajectory with the ground-truth future poses acting as positive examples. The approach is evaluated on multiple datasets quantitatively comparing to prior approaches and qualitatively using AMT.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- This work proposes a novel transformer-based architecture to perform 3D motion prediction of multi-agent scenes.\n- The related works section addresses prior relevant work in this field adequately.\n- Extensive evaluation is performed on multiple datasets using quantitative as well as qualitative approaches, showing superior results to baseline approaches.\n- The attention diagrams (6, 3 in suppl.) provide the intuition behind why using attention for multi-agent scenes is beneficial.\n\nWeaknesses:\n- It is unclear if the strength of the method is really due to the use of the transformer for encoding and decoding or due to predicting $\\Delta x$ at every timestep. It would have been a stronger comparison if, for example, SocialPool was predicting the same $Delta x$ output as the Multi-Range Transformer. This is something the authors note could be contributing to the better performance (in paragraph starting at line 262).\n- The user study seems to indicate a marginal qualitative improvement over prior methods, and it is unclear if it is statistically significant given the standard deviation. Some details in regards to the number of turkers employed would be beneficial for the reviewers.\n- A limitation of this proposed approach is that it does not address the potential for multiple futures scene rollouts. For example, if one of two interacting agent moves in one way, the other agent can react to it in multiple ways. I do not see a way for the model to encompass these possibilities.\n- The motivation behind using the adversarial loss in addition to the reconstruction loss is lacking. What is the purpose of having a discriminator here? What would happen if we removed the reconstruction loss and only trained with the adversarial loss?\n\nClarifications:\n- If I understand correctly, the output of the SPE() function is a scalar representing the distance between two agents. How exactly are these scalar values added to the output of the global-range transformer output? element-wise?\n- Is moving distance (Fig 4) how far the model predicts an agent moves in the scene? Perhaps the name should be changed to \"distance traversed\".\n- The placement of Tables 1-3 and Figure 3 makes it a little straining to look at. It would be good to reorganize the figures should this work be accepted.\n\nThere was some grammar issues throughout the manuscript that should be addressed in the final version should this work be accepted. Here are some examples:\nline 23: \"behaves ma highly depend\".\nline 167: \" architecture outputs sequence directly prevents generating...\"\nline 185-186: \"different lengths of sequence in this way input during training.\"\nline 257: \"even not as good\".\nline 263: \"other methods intend to predict...\" --> \"other methods tend to predict...\"\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This work addresses concerns over the societal impact of surveillance and privacy invasion. However, it does not address the limitations of the approach. The conclusion should be lengthened to discuss what could be done to improve the current approach. This can include extending the approach to multi-modal predictions where the model predicts multiple future scenes that are all realistic.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "NzB27zg1HH9", "reviewer": "Reviewer_jyew", "summary": "This paper tackles the problem of multi-person 3D motion trajectory prediction, and proposes a method that focuses on modelling the social interactions between people. In particular, the proposed method consists of a motion generator that generates future motion trajectory given past trajectory, and a motion discriminator that classifies natural and unreal trajectories. Both modules are trained jointly in an adversarial way. To model social interactions, the motion generator consists of a Transformer based encoder, which has both a local branch that encodes single-person motion, and a global branch that encodes multi-person interactions; and a Transformer based decoder that predicts the future trajectory. Expeirments on multiple datasets show improvements over previous methods both quantitatively and qualitatively (via human study).", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall the paper is easy to read, technically sound, and shows good results.\nMy main concern with the paper is about the experimental evaluation. In particular, the paper only compares with two previous state-of-the-art methods on the same task, and one method on modelling social interaction (which is not originally designed in the same domain). This makes the experimental results not convincing enough. While I appreciate the authors provide the user study to quantify the qualitative results, I would suggest adding more methods (both for the task, and for interaction modelling) in comparison to make the experimental results more rigorous and thorough.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Needs more discussions on the limitations of the work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "0yyV0fJSom", "reviewer": "Reviewer_7KAw", "summary": "This paper proposes a method to predict the motion of multiple people into the future for 3 seconds. For each person, the predictions are made based on the entire history of that person and the history of all other people present in the scene as well.\nTo do so, a Transformer-based architecture is used. The last known frame of each person serves as a query and the keys and values are computed from features that were extracted from both the person-specific history and the histories of the remaining people. Those features themselves were computed via a Transformer-based encoder. The method operates on 3D joint positions which are pre-processed with the Discrete Cosine Transform (DCT).\nThe method is evaluated on multi-person sequences that are native to some datasets (CMU, 3DPW, Panoptic). Because the availability of such sequences is limited, other sequences are mixed to create synthetic scenes with up to 15 people. The mean per-joint positional error (MPJPE) is used as a metric for all prediction horizons and a user study is provided.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**\nThe task of predicting human motion depending on how other people move in the scene is not new, but relatively little work exists compared to single-person forecasting. The proposed method uses well-known building blocks (DCT, Transformer, adversarial training) which are combined reasonably for the task at hand. The paper also tackles the problem of estimating global root translation, which is typically neglected in other motion prediction papers. Related work could be discussed more in-depth. Specifically, I find long citation lists like on Ln. 75 or 86 not very useful.\n\n**Quality**\nThe description of the network architecture is mostly fine. My only question is if the architecture can support a varying number of people at training and test time. Based on the description of the method I do not think it is possible as the $E_{glob}$ stacks the features of all $N$ people into one vector (Ln. 137, 140). However, the evaluation of MuPoTS-3D in Tab. 2 (2 ~ 3 persons) seems to suggest otherwise. Does passing N or (N-1) subjects change only the input “sequence” length? Could the authors clarify this?\n\nIn the following I have a number of concerns regarding the evaluation and the data:\n- As Fragkiadaki et al. [19] have noted, comparing motions for long time horizons (i.e. > 1 second) with the MPJPE is problematic due to the stochastic nature of human motion. This paper evaluates the MPJPE at prediction horizons of 1, 2, and 3 seconds. I find these numbers to be of limited value because a baseline might have a low MPJPE at 3 seconds, but that does not necessarily mean it performs badly. In my opinion, it would make more sense to report MPJPE for shorter time horizons up to 1 second, and adopt distribution-based metrics for longer time horizons, e.g. like Aksan et al. [5]. The authors do include a distribution-based metric in Fig. 4, but this only shows how far a joint traveled between the beginning and end of the sequence, which carries little information about the validity of the actual poses.\n- The MPJPE is computed on poses that contain global root translation (Ln. 226-227). This makes it impossible to know how much the global root misalignment affects the error and how much pose errors are contributing. Intuitively, predicting the global root position is difficult, so I assume its contribution to the error metric is substantial. In any case, the root misalignment and pose error should be reported separately. This is furthermore important because the two baselines [43, 44] do not consider global root translation, and hence comparing them on a metric that includes global root translation could be deemed unfair.\n- While it is appreciated that a user study on the quality of the generated motion is provided (Tab. 4), I am not sure about the significance of the results. All the reported averages are well within one standard deviation of the ground-truth average. How should we interpret the values? It is also not mentioned how big the sample size is. \n- I do not find the visual quality of the results to be very compelling. There seem to be a lot of foot skating artifacts. In the example where two people are dancing (example 3 starting at 03:28), there seem to be drastic variations of the bone length. There is no discussion about these shortcomings in the paper.\n- The paper mixes several motion capture datasets together (CMU, 3DPW, MuPoTS-3D, Panoptic). These datasets all have different skeletal representations. In the supplementary, it is explained how those representations are unified. As far as I can tell, the authors just pick a subset of joints that is common to all 4 datasets. In my opinion, this is insufficient. Although all 4 datasets are likely to have - for example - a “neck” joint, it is not guaranteed that they define the neck joint exactly the same way. E.g., in one dataset it might be closer to the head than in the other. To align the data properly, one would have to define a reference skeleton (e.g. the CMU skeleton) and then register all other datasets to that reference skeleton. Another possibility would be to use a large-scale dataset that is already unified, i.e. AMASS. This is already used by related work [5].\n- It is not clear how different sequences are merged to create multi-person sequences. While I appreciate the effort to create synthetic multi-person sequences it should be discussed how they are merged exactly, and what the advantages/downsides of the specific merging strategy are. The paper mentions that attention weights can be used to define interaction groups. This is potentially an interesting mechanism, but it highly depends on how the sequences were stitched together. As it stands now, a simple nearest neighbor query might do a good job already.\n- Ablation studies to show the impact of the global range encoder, the discriminator, the spatial positional encoding, and the length of the decoder query are provided. The differences are sometimes marginal. E.g. in Table 2 on MuPoTS-3D for 1 second, the SocialPool baseline [2] and all ablation experiments score exactly the same. In Table 5 (ablation on decoder query length) feeding 1, 2, or 4 frames performs exactly the same, and so does feeding half the sequence or the entire sequence. Furthermore, the following two claims are not well supported:\n  - Ln. 45-48 “we create a bottleneck to force the Transformer to exploit the relations between different time steps and persons in the encoders, instead of just repeating the existing motion alone”. It is not clear from the paper that leaving out the bottleneck leads to the repetition of existing motion.\n  - Ln. 170-172 “To ensure a natural and continuous long-term motion, we use a discriminator D to adversarially train the Predictor P”. It is not clear from the paper that adding adversarial training helps in making natural and continuous predictions (the MPJPE does not allow us to make a statement about whether motions are “natural and continuous”).\n- Lastly, the paper does not really adequately address its limitations. A weakness that should be discussed is the error accumulation problem. Because each prediction is now based on as many (potentially erroneous) predictions as we have people in the scene, the error accumulation problem could be amplified compared to the single-person prediction case.\n\n**Clarity**\nThe overall organization of the paper is adequate. It does however contain typos and formulations that hinder the readability on many occasions. I think a general proofreading pass could address those.\n\n**Significance**\nBecause of the above-mentioned flaws, I do not think this paper advances the state-of-the-art in a demonstrable way. Although considering societal contexts in motion prediction can be interesting to other researchers, I am not sure the setting this paper constructed is ideal to do so.\n\n**Summary**\nI generally find motion prediction conditioned on other people’s movements an interesting topic. I also think that the Transformer framework is a good fit for this setting, which I’m considering as the strength of the paper. However, the various weaknesses in evaluation and data preprocessing - outlined in more detail under “Quality” in the “Main Review” section - clearly outweigh the strengths of this paper. This paper thus seems premature and requires a major revision.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The broader impact statement is very short and vague. It could be discussed in more detail what “the invasion of people’s privacy through the human motion and behavior” exactly means.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The final scores for this paper are just below the usual threshold for acceptance. 
While this paper explores an interesting approach to modelling multiple 3D agent motions, the experimental work was assessed by half of the reviewers as needing more work in a number of ways to fully evaluate the method. The number of users used for the human evaluation was seen as being too small and the specifics of the metrics used to evaluate the method were put into question with respect to evaluating motions of longer duration. With another revision and re-submission I am confident the authors will have a successful submission, but at this time the AC recommends rejection.

Decision: Accept (Poster)