Multi-Person 3D Motion Prediction with Multi-Range Transformers
Authors: Jiashun Wang
Huazhe Xu
Medhini Narasimhan
Xiaolong Wang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Motion Prediction
Multi-Person Interaction
Transformer

Abstract: We propose a novel framework for multi-person 3D motion trajectory prediction. Our key observation is that a human's action and behaviors may highly depend on the other persons around. Thus, instead of predicting each human pose trajectory in isolation, we introduce a Multi-Range Transformers model which contains of a local-range encoder for individual motion and a global-range encoder for social interactions. The Transformer decoder then performs prediction for each person by taking a corresponding pose as a query which attends to both local and global-range encoder features. Our model not only outperforms state-of-the-art methods on long-term 3D motion prediction, but also generates diverse social interactions. More interestingly, our model can even predict 15-person motion simultaneously by automatically dividing the persons into different interaction groups.  Project page with code is available at https://jiashunwang.github.io/MRT/.

Multi-Person 3D Motion Prediction with Multi-Range Transformers
Jiashun Wang'
Huazhe Xu²
Medhini Narasimhan Xiaolong Wang'
UC San Diego
UC Berkeley
ji#0770ucsd. edu
(huazhe_xu_ _x, Binazzee_xumeedhnnee medhini)
@berkeley edu xis0120eng ucsd. edu
Abstract
We propose a novel framework for multi- pertiipeeoonn 3D motion trajectory prediction. Our key observation is that a human's 's action and behaviors may highly depend on the other persons around. Thus, instead of predicting each human pose trajectory in isolation, we introduce Mulri-Range Transformers model which contains of a local-range encoder for individual motion and a global-range encoder for social interactions. The Transformer decoder then performs prediction for each person by taking a corresponding pose as a query which attends both local and global-range encoder features. Our model not only outperforms long-term 3D motion prediction, but also generates diverse social interactions. model can even predict 15-person motion simultancously
-ffe-of-thheart methods on
More interestingly, by automatically dividing the persons into different interaction groups. Project page with code is available at ptttp::/jjashunme
gi thub. 1o/MRT/.
Introduction
Given a few time steps of human motion, able forecast how the person will continue to move and imagine the complex dynamics of their motion in the future. The ability to perform such predictions allows human motion is an essential component for many real world computer vision applications such as surveillance systems, and collision avoidance for robotics and autonomous vehicles. The research on 3D human motion prediction has caught a lot of attention in recent years [44, 43], where deep models are designed to take few steps of 3D motion trajectory inputs and predict long-term future 3D motion trajectory as the outputs.
react and plan own behaviors. Similarly, a predictive model for
While encouraging results have been shown in previous work, most of the research focus on single human 3D motion prediction. Our key observation is that, how a human acts and behaves may highly depend on the people around. Especially during interactions with multiple agents, agent will need to predict the other agents' intentions, and then respond accordingly [54]. Thus instead predicting each human motion in isolation, we propose to build a model predict multi-person 3D motion and interactions. Such model needs the following properties: (i) understand each agent's 's own motion in previous time steps to obtain smooth and natural future motion; (ii) within a crowd of agents, understand which agents are interacting with each other and learn to predict based on the social interactions; (iii) the time scale for prediction needs to be long-term.
In this paper, we introduce Multi-Range Transformers for multi-person 3D motion trajectory pre- diction. The Transformer
[63] has shown to be very effective in modeling long-term relations in language modeling [16] and recently in visual recognition [17]. Inspired by these encouraging results, propose to explore Transformer models for predicting long-term human motion (3 seconds into the future). Our Multi-Range Transformers contain a local-range Transformer encoder for each individual person trajectory, global-range Transformer encoder for modeling social interactions, and Transformer decoder for predicting each person's 's future motion trajectory in 3D.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Figure 1:
Our motion prediction results. Left: The last time step of the input sequence. Right: The predicted diverse and continuous motion with multi-person social interactions. We use different color indicate different persons and the darker color for the further in future for predictions.
Specifically, given the human pose joints (with 3D locations in the world coordinate) I-second time steps inputs, the local-range Transformer encoder processes each person's 's trajectory separately and focuses on the local motion for smooth and natural prediction. The global-range Transformer 3D pose joints across different persons and different time steps,
encoder performs self-attention and it automatically learns which persons that one person should be attending to model their social interactions. Our Transformer decoder will then take single human 3D pose in one time step as the query input and encoder features as the key and value inputs to compute attention for prediction. We perform prediction for different persons by using different query pose inputs. By using only one time step person pose as the query for the decoder instead of a sequence of motion steps, we create bottleneck force the Transformer to exploit the relations between different time steps and persons in the encoders, instead of just repeating the existing motion alone [43].
We perform experiments multiple datasets including CMU-Mocap [1], MuPoTS-3D [48], motion prediction in 3D (with 2 ~ 3 persons). Our method achieves significant improvement over kate-of-the-an approaches for long-term predictions and the gain enlarges as we increase the future prediction time steps from second to 3 seconds. Qualitatively, visualize that method can predict interesting behaviors and interactions between different persons while previous approaches will repeat the same poses as goes to further steps in the future. More interestingly, extend the task perform prediction with 9 15 persons by mixing the CMU-Mocap [1] and the Panoptic [24] datasets, We visualize part of the prediction results with multi-person interactions in Fig. 1. We show that Multi-Range Transformers can not only perform predictions with a crowd of persons, but also learn to automatically group the persons into different social interaction clusters using the attention mechanism.
3DPW [64] for
2 Related Work
3D Motion Prediction. Predicting future human pose in 3D has been widely studied with Recurrent
Neural Networks (RNNs) [15, 31, 41, 46, 51, 20, 22, 26, 72, 21]. For example, Fragkiadaki er al. [19] propose Mencoder-Reecuren--Deee Decoder (ERD) model which incorporates nonlinear encoder and decoder networks before and after recurrent layers. Besides using RNNs, temporal convolution networks have also show promising results on modeling long-term motion [37, 28, 10, 10, 44, 43, 5]. For example, Li et al. [37] use a convolutional long-term encoder to encode the given history motion into hidden variable and then use decoder to predict the future sequence. While these approaches show encouraging results, most of these studies fix the pose center and ignore the global body trajectory. Instead of solving two problems separately, recent works start looking into jointly predict human pose and the trajectory in the world coordinate [70, 73, 71, 65, 11]. For example, Cao er al. [11] introduce to predict human motion under the constraint of 3D scene context. Our work also predicts the human motion with both 3D poses and the trajectory movements at the same time. Going beyond single human prediction, predict multi-human motion and interaction.
Social interaction with multiple persons. Multi-person trajectory prediction has been long standing problem in decades [25, 47, 67, 52, 75, 7, 6, 23, 18, 49, 36, 42, 56, 8, 34, 61, 69, 62].

---

For example, Alahi er al. [7] present LSTM [27] model which jointly reasons across multiple individuals scene. Gupta al. [23] propose to predict socially plausible futures with Generative Adversarial Networks. However, most of these approaches focus on the global movement of the humans. To model more fine- one-grained human-hume human interactions, recent research have proposed to predict multi-person poses and trajectories at the same time [58, 57, 2J. For example, Shu et al. [58] propose a MCMC based algorithm automatically discovers semantically meaningful interactive social affordance from RGB-D videos. Adeli et al. [2] introduce to combine context constraints in multi- person motion prediction. Inspired by these works, we propose a novel Mante-Range Transformers model which scales up the long-term) prediction with even more than 10 persons.
Transformers. Transformer is first introduced by Vaswani et al. [63] and has been widely applied in language processing [16, 55, 9] and computer vision [66, 29, 50, 14, 35, 17, 12, 13, 60, 68, 59, 45, 39, 53]. For example, Dosovitskiy el al. [17] introduce that using Transformer alone can lead to competitive performance in image classification. Beyond recognition tasks, Transformers have also been applied in modeling the human motion [11, 40, 38, 5, 43). Aksan el al. [5] use space-time self-attention mechanism with skeleton joints synthesize long-term motion. Mao al. [43] propose to extract motion attention to capture the similarity between the current motion and the historical motion. Beyond modeling single human motion, we introduce multi-range encoders to combine local motion trajectory and global interactions, and a decoder to predict multi-person motion.
3 Method
3.1
Representation
Given a scene with N persons and their corresponding history motion, our goal is predict their future 3D motion. Specifically, given X = [a -.,,%] representing the history motion of person where n = 1, ...; N, and l: is the time step. We aim predict the future motion X K where T represents the end of the sequence. We use vector 2t E R3J containing the Cartesian coordinates of the J skeleton joints to represent the pose of the person at time step k. In contrast to most previous motion prediction works which center the pose (joint positions) at the origin, we instead use the absolute joint positions in the world coordinate. In our method, 2r contains both the trajectory and the pose information. For simplicity, we omit subseript n when n only represents an arbitrary person, c.g., taking 2T Ti:k-
3.2 Network Architecture
We propose a Multi-Range Transformers architecture, which allows each person to query other persons' and their own history of motion for generating socially and physically plausible future motion in 3D. The network architecture is shown in Fig. 2. The proposed architecture is composed of a motion predictor P and motion discriminator D. In the predictor P, two Transformer-base encoders encode the individual (local) and global motion separately and one Transformer-basee decoder decodes smooth and natural motion sequence.
To ensure the smoothness of the motion, the model requires dense sampling of the input sequence. We apply our local-range encoder to each individual's 's motion to achieve this. For modeling the interaction of all the persons in the whole scene, we apply our global-range encoder to their motions, which performs relatively sparse sampling of the sequences. And this only need to be calculated once for the whole scene.
The motion discriminator D is Transformer-basee classifier to determine whether the generated motion is natural. We apply Discrete Cosine Transform (DCT) [4, 44] to encode the inputs for the encoders and the Inverse Discrete Cosine Transform (IDCT) for the decoder outputs. We introduce the architectures for each component as following.
3.2.1 Local-range Transformer Encoder
When predicting one person's motion, we first use our Local-range Transformer encoder to process this person's 's history motion. We use offset Ar; Ti+1 x; between two time steps to represent the motion. We apply DCT and a linear layer to Ar and then add the sinusoidal positional embedding [63] to get the local motion embedding for k time steps 41.6 [l..... 4e). We concatenate
3

---

Transformer Decoder
Local-range Transformer Eicoder
Transforner
Decoder
Forward
Temporal positional
Lx
encodiigg
Global- Globul-range Transforner Encoder
Spatial positional encoding
Motion
Discriminator
Attention
Encoder
Feature
Query
Figure 2: Network architecture. Individual input motion is sent to the Local-range Transformer Encoder and all the person's 's motions are sent to the Global-range Transformer Encoder. & represents concatenate and @ represents add. The encoded motion features together with the query person skeleton for the Transformer Decoder. The output the future motion prediction results. On the right, show the architecture of the Transformer decoder. The encoder architecture is similar with the decoder except that the query, key and value are from the same input.
used as the key and value
them as a set tokens ELoc = 1l,,...444 4| stack alternating layers in the local-range encoder and introduce the technique use in each layer.
4I and feed them to the Transformer encoder. There are
Firstly, a Multi-Head Attention is used for extracting the motion information,
MultiHead Q, K, V) [head]; i..; head,, W
@KKKiT V
where head; softmax(
Vdk
OO is a projection parameter matrix, dx is the dimension of the key and is the number of the heads use. We use self-attention and get the query Qioe. key Kiocs and value Vioe from Ei for each head,,
Qioc Eroo Eioe w.Q..) Ki
Erse
w-(Ka) loc
Viac Ejoo W.(.
(2)
where (((2.)) (=(K,)) (V3)) loe loe loc are projection parameter matrices. loc represents the local-range. We then employ residual connection and the layer normalization techniques
architecture. We
further apply feed forward layer, again followed by a residual connection and a layer normalization following [63]. The whole process forms layer of local-range Transformer encoder. We stack L such Transformer encoders to update the local motion embedding and obtain the local motion feature e1.b [e....., Ex] as the output, with e; represents the feature for time step
3.2.2 Global-range Transformer Encoder
In the global-range Transformer encoder, we aim to encode all the N people's motion in the scene. In method, this only needs be calculated one time and then can be concatenated with any person's local motion feature to predict the correspond person's 's future motion. We first apply a linear layer each person's motion =T.s and plus the sinusoidal positional embedding get the global motion embedding 1:N lol .... D .... .... .... N for N persons in le time steps. We L layers of Transformers to encode the global motion embedding. We apply the Multi-head Attention mechanism similar Eq. the global embedding which calculated as,
Yyros Egioc (QQ))) glob
ytoo Eglos (K,,))
glob Vylos Eglob WVV.0) glob
(3)
where Egiob = 161 .... .... ..... NTT and w-(Q.)) w-(K.i) and W-(V,)) projection parameter matrices. glob represents the global-range. Then feed them to the normalization and reed-forward layers same as local-range Transformer encoder. After applying such L Transformer encoders to the global embedding, output and get the global motion feature flsN We concatenate the local 61 and global fl:N features together as H [e1, .... Ck. ft. .... ft.
get the output o!:N We add spatial positional encoding to the
JiN. ..... JNI. and feed them the decoder.

---

3.2.3 Positional Encoding
Temporal positional encoding. We apply the sinusoidal positional encoding [63] to the inputs in both local and global-range encoder. This technique is routine in many Transformer-base based methods for injecting information about the relative absolute temporal position to the models.
Spatial positional encoding. We propose spatial positional encoding (SPE) technique the outputs of the global-range Transformer encoder. Before forwarding to the Transformer decoder, we want provide the spatial distance between the query token I& and the tokens of every time step of each person 1:N Intuitively, the location information helps clustering different persons in different social interactions groups, especially in scene with crowd of persons. We calculate SPE as,
Titk
SPE(2 mb) = exp( 3J ||=," - I& l12)
(4)
where n = I,..., N,..,,,, = L,.., and x 2g E RV. SPE(-) will explicitly calculate the spatial distance between two persons. the SPE (a:] 3.. will then add of (the output of the global-range encoder) respectively in order to get the global feature f1:
3.2.4 Transformer Decoder
We send the local-global motion feature H together with static human pose 2g at time step into the decoder. We use the similar Multi-head attention mechanism the Transformer encoders. But differently, we take the single pose as the query and use the feature from the encoders get keys and values. We also apply L layers of Transformer decoder. Specifically, we apply a linear layer to Ty and then get q in order to get the query Qáee, we get the key Kéec and value Vdee both from local-global motion feature H as,
Vaoc q" (' K (K.) Vine (W(V (V.)
dec
where ((Q..) w-(k.)) and (((V.)) are projection parameter matrices. At the end of the decoder, apply two fully connected layers followed by Inverse Discrete Cosine Transform (IDCT) [4, 44] and output an offset motion sequence [ArR, .... A which can easily lead to the future 3D motion trajectory 1.T The model outputs a sequence of motion directly instead pose each time and this design can prevent generating freezing motion [40]. Note also add residual connections and layer normalization between layers.
3.2.5 Motion Discriminator
The design of such encoder-decoder architecture helps predict the future motion. To ensure natural and continuous long-term motion, we use discriminator D to adversially train the Predictor P. The output motion 1 .. is given as input the Transformer encoder with the same architecture of the local-range encoder and we further use another two fully connected layers to predict values (1.0) representing that & are real fake poses. We use the ground-truth future poses to provide as the positive examples. We train the predictor P and discriminator 'D jointly.
3.3 Training and Inference
We train our predictor P with both the reconstruction loss and the adversarial loss,
Lp = AveaL + Ladu
(6)
where Xade and Aree constant coefficients balance the training loss. We calculate the Lree and Lody as follows,
Lyec = T- Ac, - Arel? T k t=k
(7)
Lods T (D( - 112
We train our discriminator D following [33] with loss LD,
1
Lp T-k D( 11 T- (D()++a:)) 1
(8)
5

---

where Æ6+1;T is from the predicted motion and Uk+1-T is from the real motion. We train the discriminator that classifies the real ones as 1, where RT represents all the poses are natural.
We propose an efficient strategy which is progressively increasing the input sequence length during training and inference. Since Transformer-based encoders are used, there is no limit to the length of the input sequence. During training, we provide Fi:k to predict 15+1:26 (2t as the query), and we provide 21:24 to predict 1:34 (2x as the query) and so on. We sample different lengths of sequence in this way input during training. During inference, we predict the future motion in an auto- rugoesegree way. Given Ti:k. our model predicts 11-+1:2A. Then, given the input and our prediction results, we use them as inputs for our model again to predict 1:3+ and so on. The advantage of such design that when predicting longer motions, we still maintain the early motions as inputs to the model, instead of using a fixed length to predict each of the future time steps [44, 43], which may cause the loss of early interactive information. Through the experiment, we find this strategy could largely reduce the error accumulation.
4 Experiments
4.1 Datasets
We perform our experiments on multiple datasets. CMU-Mocap [1] contains high quality motion sequences of ~ 2 persons in each scene. Panoptic [24], MuPoTSS3D [48] and 3DPW [64] datasets are collected using cameras with pose estimation and optimization. Consequently, the estimated joint positions in the latter datasets contain more noise and are less smooth in comparison to CMU-Mocap. For multi-person data, there are about 3 ~ persons in each scene in Panoptic, 2 ~ 3 persons in MuPoTS- 3D each scene and 2 persons in 3DPW. It is worth noting that in the Panoptic dataset, most of the scenes are people standing and chatting. with small movement. The skeleton joint positions in 3DPW are obtained from moving camera so that there is more unnatural foot skating.
Dataset settings. We design two different settings to evaluate our method. The first setting consists of a small number of people (2 ~ 3). We use CMU-Mocap as our training data. CMU-Mocap contains a large number of scenes with a single person moving and small number of scenes with two persons interacting and moving. We sample from these two parts and mix them together as our training data. We make all the CMU-Mocap data consists of persons in each scene. We sample test set from CMU-Mocap in similar way. We evaluate the generalization ability of our model by testing on MuPoTS-33 3D and the 3DPW dataset with the model trained on the entire CMU-Mocap dataset.
The second setting consists of scenes with more people. For the training data, we sample motions from CMU-Mocap and Panoptic and then mix them. We integrate CMU-Mocap and Panoptic scenes to build single scene with more people. For the test data, we sample one version from both CMU- Mocap and Panoptic, namely Mix1. And we sample one version from CMU-Mocap. MuPoTS-3D and 3DPW, namely Mix2. There are 9 ~ 15 persons in each scene in Mix and 11 persons in Mix2. The positive motion data for discriminator is sampled from CMU-Mocap's s single person motion.
4.2 Implementation Details
In experiments, we give 1 second history motion (k= 15 time steps) input and recursively
predict the future 3 seconds (45 time steps) as Sec. 3.3 described. We use L = 3 alternating layers with 8 heads in each Transformer. We use Adam [32] as the optimizer for our networks. During training, we set 3 x 10 as the learning rate for predictor P and 5 x 10-4 as the learning rate for discriminator D. We set Xrea = and Xndo 5 x 10-4, For experiments with 2 ~ 3 persons, we set a batch size of 32 and for scene with more people, we set a batch size of 8.
4.3 Metrics and Methods for Comparisons
Metrics. We first use Mean Per Joint Position Error (MPJPE) [30] without aligning as the metric to compare the multi-person motion prediction results in 1, 2 and 3 seconds. Without aligning, MPJPE will reflect the error caused by both trajectory and pose. We also compare the root error and pose error (MPJPE with aligning) separately. The root error is the I.2 root joint position error. Further, we compare the distribution of the movement between the start and end of the outputs on different datasets. Specifically, we measure this movement by calculating the mean L2 distance between the

---

CMU-Mocap (3 persons)
MuPoTS-3D (2 persons)
3DW
Mix1
Mix2
method
persons)
15 persons)
(11 persons)
1s
s
1s
3s
1 s
2s
3s
1s
2s
3s
LTD [44
1.37 2.19
3.26
1.19
1.81
2.34
4.67
7.10
8.71
2.10
3.19
4.15
1.72
2.58
3.45
HRI [43]
1.49 2.60 3.07
0.94
1.68
2.29
4.07
6.32
8.01
1.80
3.14
4.21
1.60
2.71
3.67
PocialPool [2]
1.15 2.71
3.90
0.92
1.67
2.51
4.17
7.17
9.27
1.85
3.39
4.84
1.72
3.06
4.26
Ours w/o Local
1,42 2.20
2.99
1.28
2.10
2.78
3.96
5.94
7.75
2.09
3.34
4.34
1.85
2.92
3.83
Ours w/o Global
0.99 1.71
2.50
0.92
1.67
2.50
4.17
6.85
8.91
1.77
3.10
4.19
1.42
2.29
3.06
Ours w/o D
1.13 1.84
2.57
0.92
1.62
2.26
4.17
6.41
8.09
1.75
3.00
4.00
1.34
2.19
2.95
Ours w/o SPE
1.05
1.68
2.37
0.92
1.51
2.23
3.92
6.18
7.79
1.75
3.09
4.13
1.31
2.15
2.92
Ours
0.96
1.57
2.18
0.89
1.59
2.22
3.87
6.12
7.83
1.73
2.99
3.97
1.29
2.09
2.82
Table 1: MPJPE on different datasets. We compare our method with the previous SOTA methods and ablative baselines on predicting 1, 2 and 3 seconds motion. Best results are shown in boldface.
CMU-Mocap
MuPoTS-3D
3DPW
method
Root
Pose
Root
Pose
Root
Pose
Is
2s
3s
1s
2s
38
1s
2s
3s
1s
2s
38
1s
2s
3s
Is
2s
3s
LTD]44]
0.97 1.73
2.62
0.98
1.21
1.37
0.89
1.39
1.91
0.88
1.14
1.31
4.28
6.79
8.41
1.54
1.76
1.98
HRI [43]
0.96 2.06
3.11
1.05
1.37
1.58
0.66
1.30
2.16
0.73
1.07
1.30
3.67
6.42
8.64
1.43 1.75
1.94
SocialPool [2]
0.96 2.01 2.96
1.03
1.41
1.71
0.96
1.38
2.21
0.72
1.08
1.30
3.76
6,86 9.07
1.60
1.95 2.15
Ours
0.60 1.12 1.71 0.79 1.05 1.22
0.67
1.25 1.86
0.69
0.99
1.19
3.42
5.69 7.30
1.52 1.75 1.93
Table 2:
Root and pose error on different datasets. We compare our method with the previous SOTA methods on predicting 1, 2 and 3 seconds motion. Best results are shown in boldface.
start and end skeleton joint positions. This surrogate metric is designed measure whether method could predict a plausible movement, instead of standing and repeating. We also perform a user study using Amazon Mechanical Turk (AMT) to compare the realness of the predicted motion. We let the user to score each motion prediction from 1 to 5, with a higher score implying the video looks more natural inspired by [65, 74].
We argue that for multi-person motion prediction, joint positions are more relevant compared to angle- based representations, since most of the data suitable for this task is obtained by cameras and pose estimation, containing only the position representations. Another consideration for comparison method selection is many related studies only model the pose fixing at the origin, while we need to select methods that could predict absolute motion. We select two competitive state-of-the-art person motion prediction methods: LTD [44] is graph-based method and HRI [43] is an attention-based method. Both of them allow absolute coordinate inputs, which fits our task and settings well. Most relevant to our work is SocialPool [2], method which uses GRU [15] to model the motion sequences and proposes to use social pool to model the interaction. We remove the image input of it as well as the image feature, keeping the sequence to sequence and social pool structures. We use the same data to train these methods.
Methods for comparisons.
4.4 Evaluation Results
4.4.1 Quantitative Results
We report the MPJPE in 0.1 meters of 1 second, 2 seconds and 3 seconds predicted motion on CMU-Mocap, MuPoTS-3D and 3DPW, Mix and Mix2 respectively in Tab. 1. In both cases with a small number and a large number of people, our method achieves state-of-the-art performance for different prediction time lengths. We achieve up to 20% improvement when compared to the previous single-person person-basodddedde methods [44, 43] and achieve up to 30% improvement compared to the muultipeerson-eee method [2]. In SocialPool [2], the same global feature added to all the persons which interferes with the model's prediction for each individual, especially when there are large number of people. Because of this, SocialPool's 's performance even not as good as the previous single-persor person- based method. However, in our design the model can use the features corresponding to one person to query the global motion feature which automatically allows to use the motion information belonging to other persons. Therefore, our method can achieve good results on scenes with any number of people. We also compare the root error and pose error respectively in Tab. 2. The results are in 0.1 meters. Generally, method can predict the trajectory of the root and the
7

---

Dittr bution the movement CMU-Mocco
Distribution of the movement MuPoTS-3C
Distrioution of the movemment 3DPW
CIT
Ours
Ours
1.2
Our
Soria/Porl
SocialPeol
SocialPool
3-
HRI
HRI
1.3
HRI
LTD
LTD
0.3
2-
3-
0.5
0.4
1
0-
0.0
1
0.0
0.5
1.0
2.0
2.5
Figure 3: Distribution of the movement between the start and end of the predictions on different datasets. X-axis shows the moving distance. Ours is the most similar to the ground truth while the others intend to predict smaller movement. The model is only trained on CMU-Mocap.
method
CMU-Mocap
MuPoTS-3D
3DPW
Mix1
Mix2
LTD [44]
3.61
3.6610.93
3.75
3.79
3.75+0.90
HRT [43]
3.36+-0.96
3.59+1.25
3.76 0.72
3.67+00.9
3.714-0.90
SocialPool [2]
3.49++0.87
1.66
0.66
3.62++0.92
1.49
Ours
3.62+
3.68+0098
3.78+0082
3.74+0.83
3.77+-0.82
GT
3.7810.76
3.85
3.77=0.81
3.77-10.87
3.8811079
Table User study. We perform a user study using the AMT. We provide the average of the human evaluated score w.r.t. the average + the standard deviations. Best results are shown in boldface.
poses more accurately, especially for the root prediction. We will also introduce in the following experiments that previous methods often intend to predict shorter movement.
We show the distribution of the moving distance between the start and the end of the predictions in Fig. 3. It shows that other methods intend to predict motion with less movement, with much higher value in y-axis close to 0 in x-axis. We observe the results of other methods sometimes just stay in the same spatial location while the ground truth is moving with large distance. On the contrary, the distribution of our result is very similar to the ground truth on CMU-Mocap and even on completely unseen MuPoTS-30 and 3DPW datasets. The reason is that our Transformer encoders can take any time length motion as input for modeling the long-term motion. And using the decoder to output a Ar motion is also beneficial to model the position movement. For user study, report the average and the standard error of the score in Tab. 3. A higher average score means users think that the results more "natural looking" and a lower standard error indicates that the scores are more stable. Our results get better reviews consistently across all datasets.
4.4.2 Qualitative Results
We provide qualitative comparisons in Fig. 4. Our predictions are more natural and smooth while being close to the real record. It can be seen that RNN-based method (SocialPool [2]) will quickly produce freezing motion, which is consistent with the claims in [5, 40]. Another finding is that when predicting the absolute skeleton joint positions, decoding based on an input seed sequence (HRI [43]) or adding the input sequential residual (LTD [44]) to the output, will make the predicted motion have hysteresis and repeat the history. For example, in a forward motion, the prediction may jump back into temporally unreasonable position and then continue move forward. We argue the positions of the past sequences have a negative impact on the model's 's prediction using the previously proposed design. This also consistent with the conclusion we get in Fig. 3. However, our method, using a static pose as query and predicting A sequence, could solve this problem effectively.
4.4.3 Ablation Study
If
2f
4f
half
all
We perform ablation study different modules of our network. We provide the MPJPE re- sults on different datasets in Tab. 1. We move the local-range encoder, global-range encoder, discriminator and the spatial positional encoding respectively. After removing each module, the overall performance of the model has declined.
second
0.96
0.96
0.96
1,03
1.03
2 seconds
1.57
1.58
1.57
1.71
1.73
3 seconds
2.18
2.22
2.26
2.44
2.50
Table 4: MPJPE with different decoder query input length on CMU-Mocap. "f" denotes frame. "half" denotes half of the sequence and "all" denotes the whole sequence.
8

---

$
HRI
$
3R
SocialPool
$
8
Ours
Ingut
Octput
Figure 4: Qualitative comparison with other methods. Left two columns input and right three columns are outputs. Our result is the closest to the real record and the others fail to predict a walking motion and predict less accurate interaction motion. The input data is from CMU-Mocap.
We find local-range encoder could largely reduce the prediction error. We also prove the effectiveness of the global-range encoder and an adversial train- ing would improve the prediction performance. is worth noting that our spatial positional encoding performs better scene with more people, which is in line with our expectations.
1-layer
3-layer
6-layer
9-layer
sccond
1.02
0.96
1.07
2.69
2 scconds
1.71
1.57
1.79
4.71
3 seconds
2.47
2.18
2.58
6.83
Table MPJPE with different number of Transformer layers on CMU-Mocap.
second seconds 3 seconds
We report the MPJPE with different decoder query input length on CMU-Mocap in Tab. 4. We success- fully prove that using only single pose the query for the decoder instead of a sequence of motion could create bottleneck force the Transformer to learn to predict the future instead of repeating the existing motion. We further explore the suitable number of the Transformer layers. We conduct experiments us- ing 1, 3. and layers respectively on CMU-Mocap. The results in Tab. 5 show that --layer-Transforre suitable for both shorter and longer prediction. Fur-
fixed length 0.96
1.91
2.91
0.96
1.57
2.18
Table 6: MPJPE on CMU-Mocap of our method and fixed length input.
All DCT
No DCT
Ours
second
0.99
1.44
0.96
seconds
1.65
2.43
1.57
seconds
2.37
3.47
2.18
Table 7: MPJPE CMU-Mocap. Best re-
sults are shown in boldface.
thermore, prove the effectiveness of our strategy described in 3.3, conduct experiment using a fixed length (one second motion) as input to the encoders and we report the quantitative comparisons using MPJPE on CMU-Mocap in Tab. 6. It can be seen strategy is helpful in reducing the accumulation error, especially for long-term prediction.
We conduct ablation study on the role of DCT [4, 44]. In our method, we apply DCT to the input of local-range Transformer encoder and apply IDCT to the output of the decoder. Because DCT can provide more compact representation, which nicely captures the smoothness of human motion, particularly terms of 3D coordinates [44]. We train a model which we apply all the input with DCT and apply IDCT to the output of the decoder, namely "All DCT And further train a model without any DCT or IDCT, namely "No DCT We show the MPJPE on Mocap [1] in Tab. 7. We prove our design is more suitable for this task. Though DCT is quite helpful when predicting the human motion, from the experiment we find that for modeling the human interaction, position-bass bassed representation is more suitable.
4.5 Analysis of Attention
In this section, we discuss the attention results each person the decoder. In our model, the person in the last frame will query his/her past motion well the history motion of everyone. We give

---

Input
Query Person
Attention Score
Key, Value from Local-nage
Res, Velae loo Ghibalaage
High
Pascel
Persat
Pasce
Figure 5: We show the input sequence, query person and the corresponding attention scores in the decoder. The dotted red box shows that the two interacting people have more similar attention score distribution. Darker pose color represents the further in future.
visualization of the attention score history motion of three persons in the scene. On the right, show different person's 's query and how the attention score looks like in the decoder. We separately normalize the scores for keys and values from the wocal-ragee encoder and the global-range encoder. Since the query q is only vector, so is the attention score. We concatenate the attention score vectors from each head together and draw them column represents different persons at different time steps. Keys and values are from [e1,... 6x] and fh. ff.... f2l respectively.
first layer of the decoder Fig. 5. On the we show
single frame 2ę to query,
matrix. Each row represents head for person and each
Fig. 5 shows that different query person could automatically weight different people at different time. Furthermore, we found that people in different interaction groups in scene will have more similar attention score distributions when querying the outputs from the global-range encoder. This shows that our method has the ability to automatically divide the persons into different interaction groups in the decoder without any label of this. On the other hand, querying the outputs from the local-range encoder usually have relatively fine-grained results. Specifically, the color changes more frequently. When querying outputs from the global range encoder, the color is relatively consistent one head. This is consistent with statements about the dense sampling and sparse sampling of the two encoders in Sec. 3.2.
Conclusion
In this paper, we propose a novel framework to predict multi-person trajectory motion. We design Multi-Range Transformers architecture, which encodes both individual motion and social interaction and then outputs a natural long-term motion with a correspond pose as the query in the decoder. Compared with previous methods, model can predict more accurate and natural multi-person 3D motion.
Broader Impact. The original intention of our research is to protect people's safety in surveillance systems, and collision avoidance for roboties and autonomous vehicles. However, we remain concerned about the invasion of people's privacy through the human motion and behavior. Since do not use the image of person as any input, it will not be easy to obtain the identity information of specific person and we pleased to this. But we are still concerned about whether it is possible to identify a person based solely on the person's 's skeletons and motions.
6 Acknowledgments and Funding Transparency Statement
This work was supported, in part, by grants from DARPA LwLL, NSF CCF-211266 (TILOS), NSF 1730158 CI-New: Cognitive Hardware and Software Ecosystem Community Infrastructure (CHASE-CI), NSF ACI-1541349 CC*DNI Pacific Research Platform, and gifts from Qualcomm, TuSimple and Picsart.
10

---

References
[1] Cmu graphics lab motion capture database. http://mocap.. cmu. edu/. 2, 6,9
[2] Vida Adeli, Ehsan Adeli, lan Reid, Juan Carlos Niebles, and Hamid Rezatofighi. Socially and contextually aware human motion and pose forecasting. IEEE Robotics and Automation Letters, ((4):6033-604) 2020. 7,
[3] Vida Adeli, Mahsa Ehsanpour, Ian Reid, Juan Carlos Niebles, Silvio Savarese, Ehsan Adeli, and Hamid Rezatofighi. Tripod: Human trajectory and pose dynamics forecasting in the wild. arXiv preprint arXiv:2104.0400 2021. 3
[4] Nasir Ahmed, T_ Natarajan, and Kamisetty R Rao. Discrete cosine transform. JEEE transactions on Computers, 100(1):90-99) 93, 1974. 3,
[5] Emre Aksan, Peng Cao, Manuel Kaufmann, and Otmar Hilliges. Attention, please: A spatio-temporal transformer for 3d human motion prediction. arXiv preprint wXXiv2204.00099) 2020. 2,
[6] Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social Istm: Human trajectory prediction crowded spaces. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 961-971, 2016. 2
[7] Alexandre Alahi, Vignesh Ramanathan, and Li Fei-Fei. Socially aware large-scale crowd forecasting, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2203-2210, 2014.
2,
[8] Javad Amirian, Jean- Bernard Hayet, and Julien Pettéé Social ways: Learning multi- modal distributions of pedestrian trajectories with gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0- 0, 2019. 2
[9] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint a005v:2005. /4165, 2020. 3
[10] Judith Butepage, Michael J Black, Danica Kragic, and Hedvig Kjellstrom. Deep representation learning for human motion prediction and classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4158-6166, 2017. 2
[11] Zhe Cao, Hang Gao, Karttikeya Mangalam, Qi-Zhi Cai, Minh Vo, and Jitendra Malik. Long-term human motion prediction with scene context. In European Conference on Computer Vision, pages 387-404. Springer, 2020. 2,
[12] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference Computer Vision, pages 213 229. Springer, 2020.
[13] Mark Chen. Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pages 1691-1703. PMLR, 2020.
[14] Rewon Child, Scott Gray, Alee Radford, and llya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509 2019.
[15] Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv: /409. /259, 2014. 2,
[16] Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint
2018. 1,
[17] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1192 2020. 1,3
[18] Tharindu Fernando, Simon Denman, Sridha Sridharan, and Clinton Fookes. Soft+ hardwired attention: An Istm framework for human trajectory prediction and abnormal event detection. Neural networks, 108:466-478, 2018.
[19] Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik. Recurrent network models for human dynamics. In Proceedings of the IEEE International Conference on Computer Vision, pages 4346-4354 2015.
11

---

[20] Partha Ghosh. Jie Song, Emre Aksan, and Otmar Hilliges. Learning human motion models for long-term predictions. In 2017 International Conference 3D Vision (3DV), pages 458-466. IEEE, 2017.
[21] Anand Gopalakrishnan, Ankur Mali, Dan Kifer, Lee Giles, and Alexander G Ororbia, A neural temporal model for human motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 121 12116-12125, 2019.
[22] Liang-Yan Yan Gui, Yu- Yu-Xiong Wang, Xiaodan Liang, and José MF Moura. Adversarial geometry awwae human motion prediction. In Proceedings of the European Conference on Computer Vision (ECCV), pages 786-803, 2018.
[23] Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks, In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 2255-2264, 2018. 2,3
[24] Lei Tan Lin Gui Bart Nabbe lain Matthews Takeo Kanade Shohei Nobuhara Hanbyul Joo, Hao Liu and Yaser Sheikh. Panoptic studio: A massively multiview system for social motion capture. In ICCV, 2015. 2,
[25] Dirk Helbing and Peter Molnar. Social force model for pedestrian dynamics. Physical review E, (55)4222
1995. 2
[26] Alejandro Hernandez, Jurgen Gall, and Francese Moreno-Noguer. Human motion prediction via spatio- temporal inpainting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 71 7134-7143, 2019. 2
[27] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 183):1735- 1997. 3
[28] Daniel Holden, Jun Saito, Taku Komura, and Thomas Joyce. Learning motion manifolds with convolutional
autoencoders. In SIGGRAPH Asia 2015 Technical Briefs, pages 1-4. 2015. 2
[29] Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, and Yichen Wei. Relation networks for object detection.
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3588-3599
2018. 3
[30] Catalin lonescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions on pattern analysis and machine intelligence, 36(7):1325-133) 2013.
[31] Ashesh Jain, Amir R Zamir, Silvio Savarese, and Ashutosh Saxena. Structural Deep learning on seatio-temporal graphs. In Proceedings of the ieee conference on computer vision and pattern recognition. pages 133085317. 2016. 2
[32] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6940 44860990..00.. 2014.
[33] Muhammed Kocabas, Nikos Athanasiou, and Michael J. Black. Vibe: Video inferenee for human body pose and shape estimation. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
[34] Vineet Kosaraju, Amir Sadeghian. Roberto Martin-Martía, lan Reid, S Hamid Rezatofighi. and Silvio Savarese. Social-bigat: Multimodal trajectory forecasting using bicycle-gan gan and graph attention networks. arXiv preprint 1xXiv:9907.03395 2019. 2
[35] Juho Lee, Yoonho Lee, Jungtack Kim, Adam Kosiorek, Seungjin Choi, and Yee whye Teh. Set transformer: A framework for attention based intuutaationiivvriinn neural networks. In International Conference on Machine Learning, pages 3744-3753. PMLR, 2019. 3
[36] Namhoon Lee, Wongun Choi, Paul Vernaza, Christopher B Choy, Philip HS Torr, and Manmohan Chan- draker. Desire: Distant future prediction in dynamic scenes with interacting agents. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 336-345, 2017. 2
[37] Chen Li, Zhen Zhang. Wee Sun Lee, and Gim Hee Lee. Convolutional sequence to sequence model for human dynamics. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages
2018. 2
[38] Jiaman Li, Yihang Yin, Hang Chu, Yi Zhou, Tingwu Wang. Sanja Fidler, and Hao Li. Learning to generate diverse dance motions with transformer. arXiv preprint 2rXiv:2008.08177, 2020.
12

---

[39] Ke Li. Shijie Wang, Xiang Zhang, Yifan Xu. Weijian Xu, and Zhuowen Tu. Pose recognition with cascade transformers. arXiv preprint arXiv:2104.066 2021.
[40] Ruilong Li, Shan Yang. David A. Ross, and Angjoo Kanazawa. Learn to dance with aist++; Music conditioned 3d dance generation, 2021. 3, 5,8
[41] Zimo Li, Yi Zhou, Shuangju Xiao, Chong He, Zeng Huang, and Hao Li. Auto-condiiioned recurrent networks for extended complex human motion synthesis. arXiv preprint arXiv: 170.00533, 2017. 2
[42] Karttikeya Mangalam, Harshayu Girase, Shreyas Agarwal, Kuan-Hui Lee, Ehsan Adeli, Jitendra Malik, and Adrien Gaidon. It is not the journey but the destination: Endpoint conditioned trajectory prediction. In European Conference on Computer Vision, pages 336-776. Springer, 2020. 2
[43] Wei Mao, Miaomiao Liu, and Mathieu Salzmann. History repeats itself: Human motion prediction via motion attention. In European Conference on Computer Vision, pages 489- Springer. 2020. 1, 2, 3, 6, 7, 8
[44] Wei Mao, Miaomiao Liu, Mathieu Salzmann, and Hongdong Li. Learning trajectory dependencies for human motion prediction. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 9489 9497, 2019. 5, 8,8,9
[45] Weian Mao, Yongtao Ge, Chunhua Shen, Zhi Tian, Xinlong Wang, and Zhibin Wang- Tipose: Direct human pose estimation with transformers. arXiv preprint arXiv:2103. 153200 2021. 3
[46] Julieta Martinez, Michael J Black, and Javier Romero. On human motion prediction using recurrent neural networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 2891-2900, 2017. 2
[47] Ramin Mehran, Alexis Oyama, and Muharak Shah. Abnormal crowd hchavior detection using social force model. 2009 IEFE Conference on Computer Vision and Pattern Recognition, pages 935-942, IEEF, 2009. 2
[48] Dushyant Mchta, Oleksandr Sotnychenko, Franziska Mueller, Weipeng Xu, Srinath Sridhar, Gerard Pons- Moll, and Christian Theobalt. Single-shot multi-person 3d pose estimation from monocular rgb. In 3D Vision (3DV), 2018 Sixth International Conference on. IEEE, sep 2018. 2,
[49] Ahduallah Mohamed, Kun Qian, Mohamed Fllhoseiny, and Christian Claudel. Social-stgenm: A social spatio-temporal graph convolutional neural network for human trajectory prediction. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14424-14432, 2020. 2
[50] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran, Image transformer. In International Conference Machine Learning, pages 4055-4064 PMLR, 2018. 3
[51] Dario Pavllo, David Grangier, and Michael Auli. Quatemet: A quaternion based recurrent model for human motion. arXiv preprint arXiv: 1805.804855555 2018.
[52] Stefano Pellegrini, Andreas Ess, Konrad Schindler, and Lue Van Gool. You'll never walk alone: Modeling social behavior for multi-targe tracking. In 2009 IEEE 12th International Conference on Computer Vision, pages 261-268. IEEE, 2009. 2
[53] Mathis Petrovich, Michael J Black, and Gül Varol. Action-conditioned 3d human motion synthesis with transformer vae. arXiv preprint arXiv:2104.05670. 2021. 3
[54] David Premack and Guy Woodruff. Does the chimpanzee have a theory of mind? Behavioral and brain sciences, 1(4):515-526, 1978.
[55] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 18))99 2019. 3
[56] Amir Sadeghian, Vineet Kosaraju, Ali Sadeghian, Noriaki Hirose, Hamid Rezatofighi, and Silvio Savarese. Sophie: An attentive gan for predicting paths compliant social and physical constraints. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 1349-1358, 2019. 2
[57] Tianmin Shu, Xiaofeng Gao, Michael S Ryoo, and Song- Chhn Zhu. Learning social affordance grammar from videos: Transferring human interactions to human robot interactions. In 2017 IEEE international conference robotics and automation (ICRA), pages 1669-1676 IEEE, 2017.
13

---

[58] Tianmin Shu, Michael S Ryoo, and Song-Chun Zhu. Learning social affordance for human-robot interaction. arXiv preprint woXiv:1004.0362, 2016. 3
[59] Lucas Stoffi, Maxime Vidal, and Alexander Mathis. End-to-end trainable multi-instance pose estimation with transformers. arXiv preprint arXiv:2103. 12111, 2021. 3
[60] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7464-7473, 2019.
[61] Jianhua Sun, Qinhong Jiang, and Cewu Lu. Recursive social behavior graph for trajectory prediction. In Proceedings of the IEEE/CVF Conference un Computer Vision and Puttern Recognition. pages 660-669, 2020. 2
[62] Jianhua Sun, Yuxuan Li, Hao-Shu Fang, and Cewu Lu, Three steps to multimodal trajectory prediction: Modality clustering, classification and synthesis. arXiv preprint arXiv:2103.07855 2021.
[63] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv: 2017. 1, 3,3,,, 5,
[64] Timo von Mareard, Roberto Henschel, Michael Black, Bodo Rosenhahn, and Gerard Pons-Moll, Recover- ing accurate 3d human pose in the wild using imus and a moving camera. In European Conference on Computer Vision (ECCV), sep 2018. 2,
[65] Jiashun Wang, Huazhe Xu, Jingwei Xu, Sifei Liu, and Xiaolong Wang. Synthesizing long-term 3d human motion and interaction in 3d scenes. arXiv preprint arXiv:2012.0553 2020. 2,7
[66] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 7794-7803, 2018.
[67] Kota Yamaguchi, Alexander C Berg, Luis E Ortiz, and Tamara L Berg. Who are you with and where are you going? In CVPR 2011, pages 1345-1352. IEEE, 2011. 2
[68] Sen Yang, Zhibin Quan, Mu Nie, and Wankou Yang. Transpose: Towards explainable human pose estimation by transformer. arXiv preprint 2xX1v:2012.14214, 2020.
[69] Cunjun Yu, Xiao Ma, Jiawei Ren, Haiyu Zhao, and Shuai Yi. Spatio-temporal graph transformer networks for pedestrian trajectory prediction. In European Conference Computer Vision, pages 507-523. Springer, 2020. 2
[70] Ye Yuan and Kris Kitani. Diverse trajectory forecasting with determinantal point processes. arXiv preprint arXiv:I907.0443 94970497777 2019.
[71] Ye Yuan, Shih-En Wei, Tomas Simon, Kris Kitani, and Jason Saragih. Simpoe: Simulated character control for human pose estimation. arXiv preprint (tti::21440000 2021.
[72] Jason Y Zhang, Panna Felsen, Angjoo Kanazawa, and Jitendra Malik. Predicting 3d human dynamics from video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7114-7123, 2019. 2
[73] Yan Zhang, Michael Black, and Siyu Tang. We are more than our joints: Predicting how 3d bodies move. arXiv preprint arXiv:2012.00619, 2020. 2
[74] Yan Zhang. Mohamed Hassan, Heiko Neumann, Michael J. Black, and Siyu Tang. Generating people in scenes without people. In Computer Vision and Pattern Recognition (CVPR), June 2020.
[75] Bolei Zhou, Xiaogang Wang, and Xiaoou Tang. Understanding collective crowd hehaviors: Learning a mixture model of dynamic pedestrian-agent In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 2871-2878 IEEE, 2012. 2
14


Reviews:
[{"review_id": "rvJyRnVzFdA", "reviewer": "Reviewer_uWwa", "summary": "This paper proposes a transformer-based model to predict future 3D poses of a group of people, given 3D poses from previous observations. The authors explicitly model local interactions (ie dependencies between poses at previous time steps for a given person) and global interactions (dependencies between different people in the scene), and obtain state-of-the-art results on 3 different datasets. The transformer model is an encoder-decoder type structure, where the present and past poses are encoded, and the query to the decoder is the pose at the current frame. The model then predicts 3D poses for T future frames.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method proposed by the authors, using an encoder-decoder structure of transformers (similar to that of the original transformer for machine translation), is a good way of modelling temporal dependencies in 3D pose of a single person, and also interactions with other people in the scene. The transformer-based architecture can also naturally handle a variable sequence length at the input.\n\nThe authors use two encoders, one to model local interactions, and another to model global interactions between all people in the scene. The \"Global range transformer encoder\" has N times more input tokens than the local transformer, where N is the number of people. Therefore, since the \"Global transformer\" is a superset of the \"Local transformer\", is the local transformer strictly necessary? And would any potential benefits of having a \"Local + Global\" transformer combination disappear as the training dataset size increases? Results from other large-scale transformer papers [17] suggest that additional inductive biases in the network design becomes less important with larger dataset and model sizes. I would appreciate the author's comments on this.\n\nAn interesting choice made by the authors was to progressively increase the input sequence length to the model during inference. Ie, the input pose, and all previous predictions made in the sequence are used as the input to the encoder of the model. It would be good to ablate the effect of this design choice: What if the input sequence length is always fixed (except the beginning of the sequence)?\n\nThe authors also include some qualitative results showing that the decoder groups together groups of interacting people. However, the authors use a synthetically created dataset where they mix pose trajectories from different sources (ie CMU-Mocap and Panoptic). Therefore, it is easy to distinguish these groups based on the positional embedding alone, as the \"mixed samples\" are far apart in 3D space. These results would be more interesting to me, if the model was able to distinguish groups of people from the same original scene. Right now, it appears that the model is distinguishing people from different dataset splits instead.\n\nThe model produces good results in terms of MJPJE accuracy. However, future prediction by definition can have multiple possible outcomes, and the proposed method always predicts a single prediction, rather than diverse ones (ie [36]).\n\nIt would also be good to discuss this concurrent work in the final version (although I understand that it is too recent for the authors to compare to)\n\nM Petrovich et al. Action-Conditioned 3D Human Motion Synthesis with Transformer VAE. arxiv 2104.05670. 2021\n\nThere are also a couple of typos in the text\n\nLine 23: \"ma\" -> \"may\"\nLine 24: \"response accordingly\" --> \"respond accordingly\"\nLine 70 and 103: \"in the world coordinate\" -> \"in world coordinates\" / \"in the world coordinate frame\"\nLine 137: \"plus the sinusoidal embedding\" --> \"add the sinusoidal embedding\"\n\n\n\nOverall, I am quite borderline with this paper. It shows that a transformer-based encoder-decoder model is effective for the task of 3D future prediction. However, it does not appear that original to me, given the success of transformers in so many different domains. And the proposed model is fairly similar to the encoder-decoder model for machine translation in the original transformer paper. Although the authors perform explicit modelling of \"local\" and \"global\" interactions, I do not think they have really adapted the transformer architecture specifically to the future prediction task by for example, generating diverse outputs, or by conditioning the future prediction on some input.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Both of these are barely described in the main paper. The checklist says they are detailed in the supplementary, but I did not see it there.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "HKqUwf2vWRI", "reviewer": "Reviewer_MkB1", "summary": "This paper describes a model for multi-person motion prediction. It is based on the encode-decoder architecture. Both are constructed with self-attention transformers. The model takes in the absolution 3D coordinates of body joints from multiple persons and predict the future movement of the joints in 3D space for a timespan of 1-3seconds. \n\nAdditionally, the authors introduced several pactices in the model design including: 1) the use of the last seen frame in the observed as queries; 2) the use of DCT to encode joints; 3) The separation of a global encoder and a local encoder. The global encodes input of multiple persons while a local encoder encoder data from a single person. The model is then trained with both the reconstruction loss and a adversarial loss implemented via a Transformer based discriminator model. \n\nThe authors conducted experiments on human motion sequence datasets. Improved results over compared baselines are observed. The authors further provided analysis on the query design and number of transformer layers used. Qualitative analysis of the prediction is also provided. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Originality\n\nAs far as I know this method is new for the tasks of multiple person motion prediction. There is a related field of object trajectory prediction but in that task mostly the trajectories of centers of mass are concerned while in this work each joint's location needs to be predicted. \n\n## Quality\nThe model itself is described clearly. It achieves good improvement over the baselines. The authors have provided a few ablation studies of design choices. \n\nThere are several things that are not assessed as listed below:\n- whether the local range Transformer is necessary given that the global transformer already sees all input data. \n- the use of DCT as input encoding\n- the combined prediction of absolute joint locations vs. center of mass trajectory + relative joint locations\nSome of these items are claimed as motivation of design choices so they warrant a study. \n\nThe description of the discriminator is a little bit vague. One questions not answered is why it is not designed as a similar architecture as the prediction part. In experiments, it is only giving marginal improvements on most of the datasets (Table 1/2/3). So this seems either a missing opportunity or deadweight in the design. The authors please clarify. \n\n ## Significance\nBeing able to predict the motion of multiple person simultaneously is a nice and practical property of the proposed method. Based on this I feel it will be useful for practitioners. The idea of local vs. global in prediction is interesting but in the current form it lack thorough investigation. \n\n## Additional questions\n\nI have the following additional questions: \n\n1. The MPJPE metric has no unit. Is it in mm, cm, m, or any other metric system?\n2. Predicting future motion is by its natural non-deterministic. Here we are comparing the prediction with the \"groundtruth\", which is one possible instantiation among many possible futures. Is there any consideration in this aspect?\n ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors has addressed the societal impact of the proposed method. \n\nI didn't see much discussion about the limitation of the work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "Ga9fELvGinM", "reviewer": "Reviewer_i6Ex", "summary": "This paper proposes a Transformer-based deep learning architecture to predict multi-person 3D motion trajectories. To better model the social influence of neighbors, both a local-range (individual motion) and a global-range (social interactions) Transformer encoders are used.\nThe proposed framework is benchmarked on several 3D motion prediction datasets, namely CMU-Mocap, Panoptic, MuPoTS-3D, and 3DPW (and some combinations of these). The results show an improvement w.r.t. the select baselines of 20-30%. From a qualitative point of view, the model predictions seem smoother and more natural compared to previous efforts. Furthermore, the model is able to predict many people at the same time, and different social groups appear to cluster together when examining the social attention scores.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The submission presents a state-of-the-art model for 3-D human motion trajectory prediction. More specifically, the model uses two encoders and one decoder based on Transformers and a discriminator consisting of fully-connected layers. I think the overall quality of the paper is good and both the architecture and results are systematically presented.\n\nThe comparison with previously published results is properly done, and the proposed method is able to outperform state-of-the-art performance by a noticeable margin. The two considered experimental settings, namely generalization and fusion, allow the reader to understand the impact of considering more than 2/3 people as well as the generalization capability of the model.\n\nHowever, I have some concerns regarding several statements reported in the paper. For example:\n- the authors states in the abstract that the model 'also generates diverse social interactions'. I think this aspect should be better investigated (or explained) since it is not clear from the qualitative and quantitative results. What does 'diverse' mean in this case? 'Different' among the people in the scene?\n- Then, regarding the second experimental setting, the authors fuse several scenes from different datasets but they use absolute coordinates. How the different coordinates have been taken into account, if they are different? And, most importantly, people are involved in different actions and fusing them will\nresult in not natural interactions. Is my interpretation correct? \n- In Section 3, it is stated that 'x^k_n contains both the trajectory and the pose information'. It is not clear to me the difference since the dimension of this vector is '3J' and not '3J+1' as it should be with the addition of the trajectory information. \n- Absolute coordinates are then transformed into relative motion (x_(i+1) - x_i) in this work while, as stated, previous works use joint positions centered at the origin.  Is this not the same thing? If not, why?\n\nBesides the above concerns, I report some minor comments below:\n- Page 1, line 23: 'ma' should be 'may';\n- Page 1, line 25: 'response' should be 'respond';\n- Page 1, line 27: 'Such a model will need have the following'. This sentence should be correctly rephrased;\n- Page 4, line 130: 'update' is repeated two times;\n- Page 5, line 160: 'this single pose to as'. 'to' should be removed;\n- Page 6, line 210: ', We sample' should be ', we sample';\n- Page 7, line 272: 'cross' should be 'across';\n- It would be better, in my opinion, to report the number of observed and predicted time-steps in section 3 (Method) since it is a very important detail of the task; \n- The authors perform experiments considering 9-15 people. It would be nice to see a qualitative example of this setting in the paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I believe limitations and potential negative societal impact of this work have been adequately addressed by the authors.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "Fr3o-eX0G1o", "reviewer": "Reviewer_7u8M", "summary": "The paper proposes a learning framework based on transformers for 3D human body motion prediction considering the motion and interaction of other people in the scene. The proposed approach contains a local-range transformer encoder to encode each individual motion and a global-range transformer encoder for encoding social interactions. Then a transformer decoder is used to predict the future motion for each person. The results are evaluated on three datasets, CMU-Mocap, MuPoTS-3D, and 3DPW  and compared against few state-of-the-art methods. ", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1- Clarity: The main problem, motivation and the idea of the paper including its technical details are very clear. Though, the paper contains quite a few typos and grammatical errors/\n\n2- Contribution: the paper does not offer any novel theoretical and practical contribution. The method uses some arbitrary architecture variants of transformer encoders followed by a transformer decoder to predict the 3D human body skeleton of each person. The idea of social interaction for predicting human motion is not new, even in the context of the human skeleton body pose (e.g. 2 and 3). Transformers has been already used to incorporate human interaction for the human trajectory forecasting problem, eg Yu et al, \"Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction\", ECCV 2020. \n\n3- Experiments: The results of the proposed method have only been compared against a few out of many existing frameworks on human body skeleton prediction and only tested on few arbitrary datasets. Also, no details for the implementation (or hyper-parameters) of the competing methods are provided in the main text or supp material file. It is also not clear how these methods are compared because the frameworks [43] and [44] are designed to predict human body skeleton motion only (no trajectory motion is considered or predicted), while the evaluated datasets and framework [2] consider both tasks jointly. I couldn't find any public implementation for the framework [2]. it is very important to elaborate the implementation and hyperparameters details of competing methods to ensure a fair comparison. I could also see there exist a benchmark named SoMoF which aims to provide a benchmark for the same problem this paper targeted.  This approach could be evaluated on this benchmark instead. Human body pose and motion are not only affected by the other human body motion but also the objects and obstacles in the scene. several human body motion forecasting frameworks have been already proposed to model these physical interactions e.g. Cao et al. Long-term human motion prediction with scene context. ECCV, 2020 and the comparison with these relevant works fare missing,", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors attempted to address the limitation and potential negative social impact of their work on page 9.\n", "review_text": "", "extra_scores": {"ethical_concerns": "\nN/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "CbE654vOJ_", "reviewer": "Reviewer_BvU8", "summary": "- The paper proposes to tackle the problem of multi person 3D motion prediction. \n- Given the trajectories of multiple people in the scene, the model reasons about both long and short term context using separate transformer based encoders for both. \n- A transformer based decoder then takes a single pose at a time step for each person and uses encoded features to predict the future trajectory. \n- Experiments are performed on the CMU-Mocap, MuPoTs-3D and 3DPW dataset. and the results show improvements over state-of-the-art. \n- Qualitative visualizations show that the model can uncover behaviors, interactions and group people in crowds. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The problem is properly motivated. \n- Transformers for the problem, are natural given that it can handle variable length input and handle long range temporal dependencies. \n- The idea of combining global and local in the proposed way is interesting. Not all global context might be important for all pose sequences, the proposed approach helps tackle that. \n- The proposed approach shows considerable improvement over state-of-the-art approaches\n- The prediction visualizations when groups of people were involved were quite interesting and the accompanying video in the supplementary was helpful. \n- The experimentation is thorough and includes a user study. \n\nConcerns:\n- DCT : It is not clear to me why DCT was used for this task. The paper [44] uses DCT to encode temporal information and then works with the DCT bases for input to a graph model. In this paper, the transformer encoder already captures the temporal information. Was it merely an empirical observation (as given in the supplementary) ?\n- L136-137 : Was the position embedding added to the concatenated embedding or to the output of the linear layer ? \n- Global-range transformer : Should you have an input dimension which differentiates embeddings of the different people from each other ? Spatial position encoding could potentially be useful for this but I am curious why that encoding was added after the encoder ?\n- Will the calculation of SPE scale ? Since it requires computing distance between every pair. \n- While the user study does indicate a slight improvement compared to the others, the difference seems to be small. The GT values seem small too. Is that expected ?\n- Did the authors experiment with any data augmentations ?\n\nOther suggestions :\n- Typo : L23 : ‘ma’ \n- Rephrase : L 120-121\n- Typo : L134 : ‘contact’ -> ‘concatenate’ ?\n- Rephrase : L167-168\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- The authors have adequately discussed the limitations and societal impact of their work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "0UfY_hu98Nj", "reviewer": "Reviewer_c8qX", "summary": "--- Summary\n\nThis paper proposes a model for multi-person 3D motion prediction. Different from the previous works, the proposed model uses the Transformer architecture and is able to better model the multi-person social interactions. The model consists of a local-range encoder to encode the motion of each individual person, a global-range encoder to encode the social interactions, and a transformer decoder to generate the trajectory predictions. And it also has a discriminator module to make the generated trajectories more natural. The proposed model is trained with both the reconstruction loss and the discriminator loss. The evaluations were performed on multiple datasets including CMU-Mocap, Panoptic, MuPoTS-3D, and 3DPW, and the proposed model outperforms the baselines by 20% ~ 30%.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "--- Strengths\n\n- The evaluation is thorough. The proposed model was compared against multiple state-of-the-art baselines on multiple datasets, and it was shown to outperform the baselines by a large margin. The authors also performed a few ablation studies to show that the proposed design components are necessary. There was also a user study performed with Amazon Mechanical Turk that compared the quality of the predictions.\n\n- The paper is well-written.\n\n\n--- Issues and suggestions\n\n- It's interesting to have the user study that quantitatively compares the quality of the predictions, but all models compared (and including the ground-truth) seem to end up having very close scores, so maybe the design of this study can be improved?\n\n\n--- Other comments\n\n- Typo in Line 22: \"ma\".\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The majority of the reviewers agreed that this is a solid paper that deserves acceptance. The negative assessment by one of the reviewers was considered overly harsh by the ACs. Authors are highly encouraged to address the key comments reported by reviewers as well as to implement all the improvements (as indicated by authors in the rebuttal) in the final camera-ready version. 
Decision: Accept (Poster)