GENERATIVE MODEL-ENHANCED HUMAN MOTION PREDICTION
Authors: Anthony Bourached
Ryan-Rhys Griffiths
Robert Gray
Ashwani Jha
Parashkev Nachev
Venue:  (2021)

Abstract: The task of predicting human motion is complicated by the natural heterogeneity and compositionality of actions, necessitating robustness to distributional shifts as far as out-of-distribution (OoD). Here we formulate a new OoD benchmark based on the Human3.6M and CMU motion capture datasets, and introduce a hy- brid framework for hardening discriminative architectures to OoD failure by aug- menting them with a generative model. When applied to current state-of-the-art discriminative models, we show that the proposed approach improves OoD ro- bustness without sacrificing in-distribution performance, and can theoretically facilitate model interpretability. We suggest human motion predictors ought to be constructed with OoD challenges in mind, and provide an extensible general framework for hard- ening diverse discriminative architectures to extreme distributional shift.

Under review as a conference paper at ICLR 2021
GENERATIVE MODEL-ENHANCED HUMAN MOTION PREDICTION
Anonymous authors Paper under double-blind review
ABSTRACT
The task of predicting human motion is complicated by the natural heterogeneity and compositionality of actions, necessitating robustness to distributional shifts far as nut-of-distributio (OoD). Here formulate new OoD benchmark based on the Human3.6X and CMU motion capture datasets, and introduce a hybrid framework for hardening discriminative architectures to OoD failure by augmenting them with a generative model. When applied to current state-of-the- art discriminative models, we show that the proposed approach improves OoD robustness without sacrificing in-distribution performance, and can theoretically facilitate model interpretabilityy We suggest human motion predictors ought be constructed with OoD challenges in mind, and provide an extensible general framework for hardening diverse discriminative architectures to extreme distribu- tional shift.
INTRODUCTION
Human motion is naturally intelligible as a time-varying graph of connected joints constrained by locomotor anatomy and physiology. Its prediction allows the anticipation of actions with appli- cations across healthcare (Geertsema et al., 2018; Kakar et al., 2005). physical rehabilitation and training (Chang et al., 2012; Webster & Celik, 2014), robotics (Koppula & Saxena, 2013b;a; Gui et al., 2018b), navigation (Paden et al., 2016; Alahi et al., 2016; Bhattacharyya et al., 2018; Wang al., 2019), manufacture (Švec et al., 2014), entertainment (Shirai et al., 2007; Rofougaran al., 2018; Lau & Chan, 2008), and security (Kim & Paik, 2010; Ma et al., 2018).
The favoured approach to predicting movements over time has been purely inductive, relying on the history of specific class of movement to predict its future. For example, state space models (Koller & Friedman, 2009) enjoyed early success for simple, common or cyclic motions (Taylor et al., 2007: Sutskever et al., 2009; Lehrmann et al., 2014). The range, diversity and complexity of human motion has encouraged a shift to more expressive, deep neural network architectures (Fragkiadaki et al., 2015; Butepage al., 2017: Martinez et al., 2017; Li et al., 2018; Aksan et al., 2019; Mao et al., 2019; Li et al., 2020b; Cai et al., 2020), but still within a simple inductive framework.
This approach would be adequate were actions both sharply distinct and highly stereotyped. But their complex, compositional nature means that within one category of action the kinematics may vary substantially, while between two categories they may barely differ. Moreover, few real-world tasks restrict the plausible repertoire to a small number of classes-distinct or otherwise-that could be explicitly learnt. Rather, any action may be drawn from great diversity of bost kinematic and teleological-the that shape the characteristics of the underlying movements. This has two crucial implications. First, any modelling approach that lacks awareness of the full space of motion possibilities will be vulnerable to poor generalisation and brittle performance in the face of kinematic anomalies. Second, the very notion In-Distribution (ID) testing becomes moot, for the relations between different actions and their kinematic signatures are plausibly determinable only across the entire domain of action. A test here arguably needs to be to be considered a robust test at all.
(OoD) if
These considerations are amplified by the nature of real-world applications of kinematic modelling, such as anticipating arbitrary deviations from expected motor behaviour early enough for tomatic intervention to mitigate them. Most urgent in the domain of autonomous driving (Bhat- tacharyya et al., 2018; Wang et al., 2019), such safety concerns are of the highest importance, and

---

Under review as a conference paper at ICLR 2021
best addressed within the fundamental modelling framework. Indeed, Amodei et al. (2016) cites the ability to recognize our own ignorance as a safety mechanism that must be a core com- ponent in safe AL. Nonetheless, to our knowledge, current predictive models of human kinematics neither quantify OoD performance nor are designed with it in mind. There is therefore a need for two frameworks, applicable across the domain of action modelling: model to anomalous cases, and another for quantifying OoD performance with established bench- mark datasets. General frameworks are here desirable in preference to new models, for the field is evolving so rapidly greater impact can be achieved by introducing mechanisms that can be ap- plied to a breadth of candidate architectures, even if they are demonstrated in only a subset. Our approach here is founded on combining latent variable generative model with standard predictive model, illustrated with the current state-of-ther afrtte-artttttt discriminative architecture (Mao et al., 2019; Wei et al., 2020), a strategy that has produced state-of-the-arr in the medical imaging domain Myronenko (2018). Our aim is to achieve robust performance within a realistic, low-volume, heteroetrrogy data regime by providing general mechanism for enhancing a discriminative architecture with generative model.
for hardening a predictive
In short, contributions to the problem of achieving robustness to distributional shift in human motion prediction are as follows:
1. We provide framework to benchmark OoD performance on the most widely used open- source motion capture datasets: Human3.6M (lonescu et al., 2013), and CMU-MMcapp', and evaluate state-of-the-art models on it.
2. We present framework for hardening deep feed-forward models to OoD samples. We show that the hardened models are fast to train, and exhibit substantially improved OoD performance with minimal impact ID performance.
We begin section 2 with a brief review of human motion prediction with deep neural networks, and of OoD generalisation using generative models. section 3, we define a framework for bench- marking OoD performance using open-source multi-action datasets. We introduce in section 4 the discriminative models that we harden using generative branch to achieve a state-of-the-art (SOTA) OoD benchmark. We then turn in section 5 to the architecture of the generative model and the overall objective function. Section 6 presents our experiments and results. We conclude in section 7 with a summary of our results, current limitations, and caveats, and future directions for developing robust and reliable OoD performance and a quantifiable awareness unfamiliar behaviour.
2 RELATED WORK
Deep-network based human motion prediction. Historically, sequence-to-sen to-seuunnce prediction using Recurrent Neural Networks (RNNs) have been the de facto standard for human motion pre- diction (Fragkiadaki et al., 2015; Jain al., 2016; Martinez et al., 2017: Pavilo et al., 2018: Gui et al., 201 8a; Guo & Choi, 2019; Gopalakrishnan et al., 2019; Li et al., 2020b). Currently, the SOTA is dominated by feed forward models (Butepage et al., 2017: Li et al., 2018; Mao et al., 2019; Wei et al., 2020). These are inherently faster and casier to train than RNNs. The jury is still out, how- ever, on the optimal way handle temporality for human motion prediction. Meanwhile, recent trends have overwhelmingly shown that graph-based approaches are an effective means to encode the spatial dependencies between joints (Mao et al., 2019; Wei et al., 2020), or sets of joints (Li et al., 2020b). In this study, we consider the SOTA models that have graph-based approaches with a feed forward mechanism as presented by (Mao et al., 2019), and the subsequent extension which leverages motion attention, Wei et al. (2020). We show that these may be augmented to improve robustness to OoD samples.
Generative models for Out-of- Distribbbtttn prediction and detection. Despite the power of deep neural networks for prediction in complex domains (LeCun et al., 2015), they face several challenges that limits their suitability for safety-critical applications. Amodei et al. (2016) list ro- bustness distributional shift models, have been used extensively for detection of OoD inputs and have been shown to generalise
of the five major challenges to AI safety. Deep generative
't http://mocap.cs.cmm.dd
2

---

Under review as a conference paper at ICLR 2021
well in such scenarios (Hendrycks & Gimpel, 2016; Liang et al., 2017; Hendrycks et al., 2018). While recent work has showed some failures in simple OoD detection using density estimates from deep generative models (Nalismick al., 2018; Daxberger & Hernández- Lobatoo 2019), they remain a prime candidate for anomaly detection (Kendall & Gal, 2017; & Hernández-LLobtoo 2019).
Grathwohl et al., 2019; Daxberger
Myronenko (2018) use a Variational Autoencoder (VAE) (Kingma & Welling, 2013) to regularise an encoder-decoder architecture with the specific aim of better generalisation. By simultaneously using the encoder as the recognition model of the VAE, the model is encouraged to base its segmentations on complete picture of the data, rather than on reductive representation that more likely to be fitted to the training data. Furthermore, the original loss and the VAE's loss are combined as weighted sum such that the discriminator's objective still dominates. Further work may also reveal useful interpretability of behaviour (via visualisation of the latent space as in Bourached & Nachev (2019)), generation of novel motion (Motegi al., 2018), or reconstruction of missing joints in Chen et al. (2015).
3 QUANTIFYING OUT-OF-DISTRIBUTION PERFORMANCE OF HUMAN MOTION PREDICTORS
Even very compact representation of the human body such as OpenPose's 17 joint parameterisation Cao al. (2018) explodes to unmanageable complexity when temporal dimension is introduced of the scale and granularity necessary to distinguish between different kinds of action: typically many seconds, sampled hundredths of second. Moreover, though there are anatomical and physio- logical constraints on the space of licit joint configurations, and their trajectories, the repertoire of possibility remains vast, and the kinematic demarcations of telcologically different actions remain indistinct. Thus, no practically obtainable dataset may realistically represent the possible distance between instances. To simulate OoD data first need ID data that can be varied in its quantity and heterogeneity, closely replicating cases where particular kinematic morphology may be rare, and therefore undersampled, and cases where kinematic morphologies are both highly variable within defined class and similar across classes. Such replication needs to accentuate the challenging aspects of each scenario.
We therefore propose to evaluate OoD performance where only single action, drawn from a single action distribution, available for training and hyperparameter search, and testing carried out on the remaining classes. In appendix A, to show that the action categories we have chosen can be distinguished at the time scales on which our trajectories are encoded, we train a simple classifier and show it separates the selected ID action from the others with high accuracy (100% precision and recall for the CMU dataset). Performance over the remaining set of actions may thus be considered OoD.
4 BACKGROUND
Here we describe the current SOTA model proposed by Mao et al. (2019) (GCN), We then describe the extension by Wei et al. (2020) Cattention-GCM, which antecedes the GCN prediction model with motion attention.
4.1 PROBLEM FORMULATION
We are given a motion sequence X1:N = (X1, X2, X3, x,) consisting of N consecutive human poses, where x; E K with K the number of parameters describing each pose. The goal to predict the poses XN+1:N4T for the subsequent I' time steps.
4.2 DCT-BASED TEMPORAL ENCODING
The imput is transformed using Discrete Cosine Transformations (DCT). In this way each resulting coefficient encodes information of the entire sequence at a particular temporal frequency. Further- more, the option to remove high or low frequencies is provided. Given a joint, k the position of over N time steps is given by the trajectory vector: X ".,, where convert
3

---

Under review as a conference paper at ICLR 2021
DCT vector of the form: C₁ [C Ck.Nl where Ck,1 represents the Ith DCT coefficient. For
du E IR 11,0,--. ,0,, these coefficients may be computed
N Tk VI
Ck. = N
COS
(2n - 1)(2 - 1))
(1)
d11
2N
If frequencies cropped, the DCT is invertible via the Inverse Discrete Cosine Transform (IDCT):
2 Fki V
CK. vI+ + dig
1
cos
(2n 1)(1 1)(1-1)) 1))
22
Mao et al. use the DCT transform with graph convolutional network architecture to predict the output sequence. This achieved by having an equal length input-output sequence, where the input is the DCT transformation of Xk [. NK #R.N+11-1. N4T), here [TR,1..... NK is the observed sequence and +. N+T are replicas of æK,N (ie Ik,n 2K,N for n 2 N). The target is now simply the ground truth Xk-
4.3 GRAPH CONVOLUTIONAL NETWORK
Suppose C & (Xx(N+)) is defined on a graph with & nodes and N +' dimensions, then we define graph convolutional network respect this structure. First we define Graph Convolutional Layer (GCL) that, input, takes the activation of the previous layer (A (l-I),,, where is the current layer.
GCI(A -SAI- +b
where A C E (KX((++), and S KxK layer-specific learnable normalised graph laplacian that represents connections between joints, W E "R= xx"" are the learnable inter-layer weightings and b E | are the learnable biases where rl are the number of hidden units in layer
4.4 NETWORK STRUCTURE AND LOss
The network consists of 12 Graph Convolutional Blocks (GCBs), each containing GCLs with skip (or residual) connections, see figure 7. Additionally, there is one GCL at the beginning of the network, and one at the end. !] = 256, for each layer, 1. There one final skip connection from the DCT inputs the DCT outputs, which greatly reduces train time. The model has around 2.6M parameters. Hyperbolic tangent functions are used as the activation function. Batch normalisation applied before each activation.
The outputs are converted back to their original coordinate system using the IDCT (equation 2) to be compared the ground truth. The loss used for joint angles is the average I distance between the ground-truth joint angles, and the predicted ones. Thus, the joint angle loss is:
N+T
Ik,n FR.ml
K(N +
where & is the predicted tthh joint at timestep n and æk is the corresponding ground truth.
This is separately trained on 3D joint coordinate prediction making of the Mean Per Joint Posi- tion Error (MPJPE), as proposed in lonescu et al. (2013) and used in Mao et al. (2019); Wei et al. (2020). This is defined, for each training example,
4

---

Under review as a conference paper at ICLR 2021
GCL
CCB
Figure 1: GCN network architecture with VAE branch. Here 16 is the number of latent
variables per joint.
N+T
km = J(N+T
Pijn
-Pjn 1|
(5)
where Pj.n E R3 denotes the predicted jth joint position in frame And Pj,n is the corresponding ground truth, while is the number of joints in the skeleton.
4.5 MOTION ATTENTION EXTENSION
Wei al. (2020) extend this model by summing multiple DCT transformations from different sec- tions of the motion history with weightings learned via an attention mechanism. For this extension, the above model (the GCN) along with the anteceding motion attention is trained end-to-end. We refer to this as the attention-GCN.
OUR APPROACH
Myronenko (2018) augment an encoder-decoder discriminative model by using the encoder recognition model for a Variational Autoencoder (VAB), (Kingma Welling, 2013; Rezende al., 2014). Myronenko (2018) show this to be very effective regulariser. Here, also use VAE, for conjugacy with the discriminator, we use graph convolutional layers in the decoder. This can be compared to the Variational Graph Autoencoder (VGAE), proposed by Kipf & Welling (2016). However, Kipf & Welling's application is link prediction task in citation networks and thus is desired to model only connectivity in the latent space. Here model connectivity, position, and temporal frequency. To reflect this distinction, the layers immediately before, and after, the latent space are fully connected creating homogenous latent space.
The generative model sets a precedence for information that can be modelled causally, while leaving elements of the discriminative machinery, such as skip connections, capture correlations that remain useful for prediction but are not necessarily persuant to the objective of the generative model. In addition to performing the role of regularisation in general, we show that we gain robustness to distributional shift across similar, but different, actions that are likely to share generative properties. The architecture may be considered with the visual aid in figure
5.1
VARIATIONAL. AUTOENCODER (VAE) BRANCH AND LOSS
Here define the first GCB blocks VAE recognition model, with latent variable z R Kxn, = N(J2, o2), where Pz E Kxn, oz E R n, = 8, or 32 depending on training stability.
The KL divergence between the latent space distribution and a spherical Gaussian N I) is given by:

---

Under review as a conference paper at ICLR 2021
f2
- KL((((())))))
(ux + oz (og((oz))
(6)
The decoder part of the VAE has the same structure as the discriminative branch; GCBs. We parametrise the output neurons as E R*x(N+T), and log(a²) E R K ((N+T) We can now model the reconstruction of imputs samples of maximum likelihood of Gaussian distribution which constitutes the second term of the negative Variational Lower Bound (VLB) of the VAE:
N+T
log(oR,) (2g |Ck, pt..12))
l lo
Σ
e log(o
i=1
where CR are the DCT coefficients of the ground truth.
5.2 TRAINING
We train the entire network together with the additional of the negative VLB:
N+T K
6k,n Lk,nl
(eG fi)
(N (N+T) K
VLB
Discriminitiee loss
Here a hyperparameter of the model. The overall network is R 3.4M parameters. The number of parameters varies slightly per the number of joints, K. since this is reflected in the size of the graph in each layer (k = 48 for H3.6M, K = for CMU joint angles, and K = 75 for CMU Cartesian coordinates). Furthermore,
trained, the generative model is not required for prediction and hence for this purpose is as compact as the original models.
EXPERIMENTS
6.1 DATASETS AND EXPERIMENTAL SETUP
Human3.6M (H3.6M) The H3.6M dataset (Ionescu al., 2011; 2013), so called contains selection of 3.6 million 3D human poses and corresponding images, consists of seven actors each performing actions, such as walking, eating, discussion, sitting, and talking on the phone. Mar- tinez et al. (2017); Mao al. (2019); Li et al. (2020b) all follow the same training and evaluation procedure: training their motion prediction model on 6 (5 for train and for cross-validation) actors, for each action, and evaluate metrics the final actor, subject 5. For easy comparison to these ID baselines, we maintain the same train; cross-vats vross-validttion; and test splits. However, we use the single, most well-defined action (see appendix A), walking, for train and cross-validation, and we report test error on all the remaining actions from subject 5. In this way we conduct all parameter selection based ID performance.
CMU motion capture (CMU-mocapp The CMU dataset consists of 5 general classes of actions. Similarly to (Li et al., 2018; 2020a; Mao et al., 2019) we use 8 detailed actions from these classes: "basketball', "basketball signal', 'directing traffic' 'jumping, 'running', 'soccer', 'walking', and 'window washing'. We use two representations, 64-dimensional vector that gives exponen- tial map representation (Grassia, 1998) the joint angle, and 75-dimensional vector that gives the 3D Cartesian coordinates of 25 joints. We do not tune any hyperparameters this dataset and use only a train and test set with the same split as is common in the literature (Martinez et al., 2017; Mao al., 2019).

---

Under review as a conference paper at ICLR 2021
Model configuration We implemented the model in PyTorch (Paszke al., 2017) using the ADAM optimiser (Kingma & Ba, 2014). The learning rate was set to 0.0005 for all experiments where, unlike Mao et al. (2019); Wei et al. (2020), we did not decay the learning rate as it was hy- pothesised that the dynamic relationship between the discriminative and generative loss would make this redundant. The batch size was 16. For numerical stability, gradients were elipped to a maximum (2-norm of 1 and log(o²) and values were clamped between -20 and 3. Code for all experiments available at the following anonymized link: https//anonnymos.AAppeaa www.n.ooceneeee.eeeeee la7a2b5-dal3- 4338-80de-55 51e526913ddd
Walking (ID)
Eating (OeD)
Smoking (OoD)
Average (of
OnD)
milliseconds
80
160
320
400
80
160
320
400
80
160
320
400
80
160
320
400
GCN (OaD)
0.22
0.37
0.60
0.22
0.38
0.65
0.79
0.28
1.10
0.38
1.09
1:27
Sid Dev
0.001
0.008
0.008
0.01
0.003
0.01
0.03
0.04
0.01
0.01
0.02
0.02
0.007
0.02
0.04
0.04
(OoD)
0.23
0,37
0.59
0.64
0.21
0.37
0.59
0.72
0.28
0.54
1.01
0.99
0.38
0.68
1,07
1.21
Snd Dev
0.003
0.004
0.03
0.03
0.008
0.01
0.03
0.04
0.005
0.01
0.01
0.02
0.006
0.01
0.01
0.02
Table 1: Short-term prediction of Eucildean distance between predicted and ground truth joint angles on H3.6M. Each experiment conducted 3 times. We report the mean and standard deviation. Note that we have lower variance in results for ours. Full table in appendix, table 6.
Walking
Eating
Smoking
Discussion
Average
milliseconds
560
1000
560
1000
560
1000
560
1000
560
1000
GCN (OoD)
0.80
0.80
0.89
1.20
1.26
1.85
1.45
1.88
1.10
1.43
ours (OoD)
0.66
0.72
0.90
1.19
1.17
1.78
1.44
1.90
1.04
1.40
Table Long-tem term prediction of Eucildean distance between predicted and ground truth joint angles on H3.6M.
Baseline comparison Both Mao al. (2019) (GCN), and Wei et al. (2020) (attention- use this same Graph Convolutional Network (GCN) architecture with DCT inputs. In particular, Wei al. (2020) increase the amount of history accounted for by the GCN by adding motion attention mechanism to weight the DCT coefficients from different sections of the history prior to being inputted to the GCN. We compare against both of these baselines on OoD actions. For attention- GCN we leave the attention mechanism preceding the GCN unchanged such that the generative branch of the model is reconstructing the weighted DCT inputs to the GCN, and the whole network is end-to-end differentiable.
Hyperparameter search Since a new term has been introduced to the loss function, it was neces- sary to determine sensible weighting between the discriminative and generative models. In Myro- nenko (2018), this weighting was arbitrarily set to 0.1. It is natural that the optimum value here will relate to the other regularisation parameters in the model. Thus, we conducted random hyperparam- eter search for Parop and À in the ranges Pdrop = [0, 0.5] on a linear scale, and x = [10, 0.00001] on logarithmic scale. For fair comparison we also conducted hyperparameter search on GCN, for values of the dropout probability (parop) between 0. and 0.9. For each model, 25 experiments were and the optimum values were selected the lowest IDD validation error. The hyperparameter search was conducted only for the GCN model on short-term predictions for the H3.6M dataset and used for all future experiments hence demonstrating generalisability of the architecture.
6.2 RESULTS
Basketball (ID)
Basketball Signal (OoD)
Average (of 7 for OnD)
millisecunds
80
160
320
400
1000
80
160
320
400
1000
80
160
320
400
1000
GCN
0.40
0.67
L.I1
1.25
1,63
0.27
0.58
1.14
1.42
2.18
0.36
0.65
141
1.49
2.17
ours
0.40
0.66
1.12
1.29
1.76
0.28
0.37
1.5
1.4
2,07
0.34
0.62
1.35
1.41
2.10
Table 3: Eucildean distance between predicted and ground truth joint angles on CMU. Full table in appendix, table 7.
Consistent with the literature we report short-term (< 500ms) and long-term (> 500ms) predic- tions. In comparison GCN, we take short term history into account (10 frames, 400ms) for both
7

---

Under review as a conference paper at ICLR 2021
Basketball
Baskethull Signal
Average (of 7 for OoD)
milliseconds
80
160
320
400
1000
160
320
400
1000
80
160
320
400
1000
(OoD)
15.7
28.9
54.1
65.4
108.4
14.4
30.4
63.5
78.7
114.8
20.0
43.8
86.3
105.8
169.2
ours (OoD)
16.0
30.0
54.5
65.3
98.1
12.8
26.0
53.7
67.6
103.2
21.6
42.3
84.2
103.8
164.3
Table 4:
Mean Joint Per Position Error (MPIPE) between predicted and ground truth 3D Cartesian coordinates of joints on CMU. Full table in appendix, table 8.
datasets to predict both short- and long-term motion. In comparison to attention GCNN, we take long term history (50 frames, seconds) predict the next 10 frames, and predict futher into the future by recursively applying the predictions as input to the model as in Wei et al. (2020). In this way a single short term prediction model may produce long term predictions.
We use Euclidean distance between the predicted and ground-truth joint angles for the Euler angle representation. For 3D joint coordinate representation we use the MPJPE as used for training (equa- tion 5). Table reports the joint angle error for the short term predictions on the H3.6M dataset. Here we found the optimum hyperparameters to be Párop 0.5 for GCN, and À 0.003, with Pdrop = 0.3 for our augmentation of GCN. The latter of which was used for all future experi- ments, where for our augmentation of attention- GCN we removed dropout altogether. On average, our model performs convincingly better both ID and OoD. Here the generative branch works well as both regulariser for small datasets and by creating robustness to distributional shifts. We see similar and consistent results for long-term predictions in table 2.
From tables and 4, we can see that the superior OoD performance generalises to the CMU dataset with the same hyperparameter settings with similar trend of the difference being larger for longer predictions for both joint angles and 3D joint coordinates. For each of these experiments n, = 8
Table 5, shows that the effectiveness of the generative branch generalises to the very recent motion attention architecture. For attention GCN we used n, = 32. Here, interestingly short term pre- dictions poor but long term predictions are consistently better. This supports our assertion that information relevant to generative mechanisms are more intrinsic to the causal model and thus, here, when the predicted output is recursively used, more useful information is available for the future predictions.
Walking (II)
Eating (Ool))
Smoking (Ool)
Average for Ool))
milliscconds
560
720
880
1000
560
720
850
1000
560
720
880
1000
560
720
880
1000
aN-GGN (Ool)
55.4
60.5
65.2
68.7
87.6
103.6
113.2
120.3
81.7
93.7
102.9
108.7
112.1
129.6
140.3
147.8
OUrS (OaD)
58.7
65.5
69.1
81.7
94.4
102.7
104.1
113.1
137.9
145.3
Table 5: Long-term prediction of 3D joint positions on H3.6M. Here, ours is also trained with the attention-GCN model. Full table in appendix, table 9.
CONCLUSION
We draw attention to the need for robustness to distributional shifts in predicting human motion, and propose a framework for its evaluation based major open source datasets. We demonstrate that state-of-the-ard discriminative architectures can hardened to extreme distributional shifts by augmentation with a generative model, combining low in-distribution predictive error with maximal generalisability. The introduction of a surveyable latent space further provides a mechanism for model perspicuity and interpretability, and explicit estimates of uncertainty facilitate the detection of anomalies: both characteristics are of substantial value in emerging applications of motion pre- diction, such as autonomous driving, where safety is paramount. Our investigation argues for wider use of generative models in behavioural modelling, and shows it can be done with minimal or no performance penalty, within hybrid architectures of potentially diverse constitution.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Emre Aksan, Manuel Kaufmann, and Otmar Hilliges. Structured prediction helps 3d human motion modelling. In Proceedings of the IEEE International Conference on Computer Vision, pp. 7 7144- 7153, 2019.
Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social Istm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 961-971,2011
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Con- crete problems in ai safety. arXiv preprint 1606v: 2016.
Apratim Bhattacharyya, Mario Fritz, and Bemt Schiele. Long-term on-board prediction of people in traffic scenes under uncertainty. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4194 4202, 2018.
Anthony Bourached and Parashkev Nachev. Unsupervised videographic analysis of rodent be- haviour. arXiv preprint arXiv:1910.11065, 2019.
Judith Butepage, Michael J Black, Danica Kragic, and Hedvig Kjellstrom. Deep representation learning for human motion prediction and classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6158-6166, 2017.
Yujun Cai, Lin Huang, Yiwei Wang. Tat-Jen Cham, Jianfei Cai, Junsong Yuan, Jun Liu, Xu Yang, Yiheng Zhu, Xiaohui Shen, et al. Learning progressive joint propagation for human motion pre- diction. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Openpose: realtime multi- person 2d pose estimation using part affinity fields. arXiv preprint arXiv: / /8/000008 2018.
Chien Yen Chang, Belinda Lange, Mi Zhang, Sebastian Koenig, Phil Requejo, Noom Somboon, Alexander Sawchuk, and Albert A Rizzo. Towards pervasive physical rehabilitation using microsoft kinect. In 2012 6th international conference on pervasive computing technologies for healthcare (Pervaiiveeaalth) and workshops, pp. 159-162. IEEE, 2012.
Nutan Chen, Justin Bayer, Sebastian Urban, and Patrick Van Der Smagt. Efficient movement repre- sentation by embedding dynamic movement primitives in deep autoencoders. In 2015 TEEE-RAS 15th International Conference on Humanoid Robots (Humanoids), pp. 434 440. IEEE, 2015.
Erik Daxberger and José Miguel Hernández-Lobato Bayesian variational autoencoders for unsu- 2019.
pervised out-of onttof-diisrrbbttoon detection. arXiv preprint
Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik. Recurrent network models for human dynamics. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4354, 2015.
Evelien E Geertsema, Roland D Thijs, Therese Gutter, Ben Vledder, Johan B Arends, Frans Leijten, Gerhard H Visser, and Stiliyan N Kalitzin. Automated biseo-based detection of nocturnl convulsive seizures in residential care setting. Epilepsia, 59:53-60, 2018.
Anand Gopalakrishnan, Ankur Mali, Dan Kifer, Lee Giles, and Alexander G Ororbia. A neural tem- poral model for human motion prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 121 2019.
Sebastian Grassia. Practical parameterization of rotations using the exponential map. Journal of graphics tools, 3(3):29-48, 1998.
Will Grathwohl, Kuan-Chiek Wang, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. arXiv preprint arXiv: 2019.
Jacobsen, David Duvenaud, Mohammad Norouzi,
Liang- Yan Gui, Yu-Xiong Wang, Xiaodan Liang, and José MF Moura. Adversarial geometry- aware human motion prediction. Proceedings of the European Conference on Computer Vision (ECCV). pp. 186-803, 2018a.

---

Under review as a conference paper at ICLR 2021
Liang- Yan Gui, Kevin Zhang, Yu-Xiong Wang, Xiaodan Liang, José MF Moura, and Manuela Veloso. Teaching robots to predict human motion. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 562-567. IEEE, 2018b.
Xiao Guo and Jongmoo Choi. Human motion prediction via learning local structure representations and temporal dependencies. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 2580-2587, 2019.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint 161.v21610.02153 2016.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure, arXiv preprint 1812: 181211842..66, 2018.
Catalin lonescu, Fuxin Li, and Cristian Sminchisescu. Latent structured models for human pose estimation. In 2011 International Conference on Computer Vision, pp. 2220-2227, IEEE, 2011.
Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.6m Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions on pattern analysis and machine intelligence, 36(7): (117)::322-139, 2013.
Ashesh Jain, Amir R Zamir, Silvio Savarese, and Ashutosh Saxena. Structural-rnn. Deep learning on spatio-temporal graphs. Proceedings of the ieee conference on computer vision and pattern recognition, pp. 5308-5317, 2016.
Manish Kakar, Häkan Nyström, Lasse Rye Aarup, Trine Jakobi Nøttrup, and Dag Rune Olsen. Respiratory motion prediction by using the adaptive neuro fuzzy inference system (anfis). Physics in Medicine & Biology, 50(19):4721, 2005.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, pp. 5574-5584 2017.
Dachee Kim and Paik. Gait recognition using active shape model and motion prediction. IET Computer Vision, 4(1):25-36, 2010.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1eXiv:141.66980 2014.
Diederik Kingma and Max Welling. arXiv:1312.61 2013.
Auto-encoding variational bayes. arXiv preprint
Thomas N Kipf and Max Welling. arXiv:1611.07308, 2016.
Variational graph auto-encoders.
arXiv preprint
Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.
Hema Koppula and Ashutosh Saxena. Learning spatio-temports structure from rgb-d videos for human activity detection and anticipation. In International conference on machine learning, pp- 792 800, 2013a.
Hema Swetha Koppula and Ashutosh Saxena. Anticipating human activities for reactive robotic response. In IROS, pp. 2071. Tokyo, 2013b.
Rynson WH Lau and Addison Chan. Motion prediction for online gaming. International Work- shop on Motion in Games, pp. 104-114. Springer, 2008.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):36-444, 2015.
Andreas M Lehrmann, Peter V Gehler, and Sebastian Nowozin. Efficient nonlinear markov models for human motion. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1314-1321, 2014.
10

---

Under review as a conference paper at ICLR 2021
Chen Li, Zhen Zhang, Wee Sun Lee, and Gim Hee Lee. Convolutional sequence to sequence model for human dynamics. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5226-5234, 2018.
Dongxu Li, Cristian Rodriguez, Xin Yu, and Hongdong Li. Word-level deep sign language recog- nition from video: A new large-scale dataset and methods comparison. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), March 2020a.
Maosen Li, Siheng Chen, Yangheng Zhao, Ya Zhang, Yanfeng Wang, and Qi Tian. Dynamic mul- tiscale graph neural networks for 3d skeleton based human motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 214 114-223, 2020b.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv preprint arXiv:1706.0220 2017.
Zhuo Ma, Xinglong Wang, Ruijie Ma, Zhuzhu Wang, and Jianfeng Ma. Integrating gaze tracking and head-motion prediction for mobile device authentication: A proof of concept. Sensors, 18(9): 2894, 2018
Wei Mao, Miaomiao Liu, Mathieu Salzmann, and Hongdong Li. Learning trajectory dependencies for human motion prediction. In Proceedings of the IEEE International Conference on Computer Vision, pp. 9489-9497, 2019.
Julieta Martinez, Michael Black, and Javier Romero. On human motion prediction using recur- rent neural networks. In Proceedings of the IEEE Conference Computer Vision and Patlern Recognition, pp. 2891-2900, 2017.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint 1802.0360 2018.
2018.
Yuichiro Motegi, Yuma Hijioka, and Makoto Murakami. Human motion generative model using variational autoencoder. International Journal of Modeling and Optimization, 8(1), 2018.
Andriy Myronenko. 3d mri brain tumor segmentation using autoencoder regularization. In Interna- tional MICCAI Brainlesion Workshop, pp. 31 1-320. Springer, 2018.
Eric Nalisnick, Akihiro Matsukawa, Yee whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? arXiv preprint arXiv:18/0.091338 2018.
Brian Paden, Michal Cáp, Sze Zheng Yong. Dmitry Yershov, and Emilio Frazzoli. A survey of motion planning and control techniques for self-driving urban vehicles. IEEE Transactions on intelligent vehicles, 1(1):33-55 2016.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Dario Pavllo, David Grangier, and Michael Auli Quaternet: quaternion- recurrent model for human motion. arXiv preprint 1805.08800 2018.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. In International Conference on Machine Learn- ing, pp. 1278-1286, 2014.
Ahmadreza Reza Rofougaran, Maryam Rofougaran, Nambirajan Seshadri. Brima B Ibrahim, John Walley, and Jeyhan Karaoguz. Game console and gaming object with motion prediction modeling and methods for use therewith, April 2018. US Patent 9,943,760.
Akihiko Shirai, Erik Geslin, and Simon Richir. Wiimedia: motion analysis methods and applications using a consumer video game controller. In Proceedings the 2007 ACM SIGGRAPH symposium on Video games, pp. 133-140, 2007.
Ilya Sutskever, Geoffrey E Hinton, and Graham W Taylor. The recurrent temporal restricted boltz- mann machine. In Advances in neural information processing systems, pp. 1601-1608, 2009.

---

Under review as a conference paper at ICLR 2021
Petr Svec, Atul Thakur, Eric Raboin, Brual C Shah, and Satyandra K Gupta. Target following with motion prediction for unmanned surface vehicle operating in cluttered environments. Autonomous Robots, 36(4):383-400 2014.
Graham W Taylor, Geoffrey E Hinton, and Sam T Roweis. Modeling human motion using binary latent variables. In Advances in neural information processing systems, pp. 1345-1352, 2007.
Yijing Wang, Zhengxuan Liu, Zhiqiang Zuo, Zheng Li, Li Wang, and Xiaoyuan Luo. Trajectory planning and safety assessment of autonomous vehicles based on motion prediction and model predictive control. IEEE Transactions on Vehicular Technology, 68(9):8556-8855 2019.
David Webster and Ozkan Celik. Systematic review of kinect applications in elderly care and stroke rehabilitation. Journal of neuroengineering and rehabilitation, 11(1):108, 2014.
Mao Wei, Liu Miaomiao, and Salzemann Mathieu. History repeats itself: Human motion prediction via motion attention. In ECCV, 2020.
APPENDIX
The appendix consists of 4 parts. We provide a brief summary of each section below.
Appendix A: provide results from our experimentation to determine the optimum way of defining separable distributions on the H3.6M, and the CMU datasets.
Appendix B: we provide the full results tables which are shown in part in the main text.
Appendix C: we inspect the generative model by examining its latent space and use it to consider the role that the gnerative model plays in learning well as possible directions of future work.
Appendix D: we provide larger diagrams of the architecture of the augmented GCN.
A DEFINING Out-of-Distribution (OoD)
Here we describe in more detail the empirical motivation for our definition of Out-of-Distrriutiin (OoD) on the H3.6M and CMU datasets.
Figure shows the distribution of actions for the h3.6M and CMU datasets. We want our ID data to be small in quantity, and narrow in domain. Since this dataset is labelled by action are provided with a natural choice distribution being one of these actions. Moreover, it is desirable that the action be quantifiably distinct from the other actions.
To determine which action supports these properties we train a simple classifier to determine which action is most easily distinguished from the others based on the DCT inputs: (+x
DCT(( N 1 (+T] where Fs,n = Tk,N for n 2 N. We make no as- sumption the architecture that would be optimum to determine the separation, and use simple fully connected model with 4 layers. Layer I: input dimensions x 1024, layer 2: 1024 x 512, layer 3: 512 128, layer 4: 128 15 (or 128
for CMU). Where the final layer uses softmax to predict the class label. Cross entropy is used as a loss function on these logits during training. We used ReLU activations with a dropout probability of 0.5.
We trained this model using the last 10 historic frames (N = 10, T = 10) with 20 DCT coefficients for both the H3.6M and CMU datasets, as well as (N 50, T 10) with 20 DCT coefficients additionally for H3.6M (here we select only the 20 lowest frequency DCT coefficients). We trained each model for epochs with a batch size of 2048, and a learning rate of 0.00001. The confusion matrices for the H3.6M dataset are shown in figures 3, and 4 respectively. Here, use the same train set as outlined in section 6.1. However, we report results on subject 11- which for motion prediction was used as the validation set. We did this because the number of instances much greater than subject 5, and no hyperparameter tuning was necessary. For the CMU dataset we used the same train and test split as for all other experiments.
In both cases, for the H3.6M dataset, the classifier achieves the highest precision score (0.91, 0.95 respectively) for the action walking well recall score of 0.83 and 0.81 respectively. Further-
12

---

Under review as a conference paper at ICLR 2021
17500
15000
Count
10000
7100
2500
matling
tatong
snsting
dscospin
drectiont
Waming
masin
nuchaaed
aeting
ottingdoom
bkingesoald
mikknggog
moltingtyynteer
Action
(a) Distribution of short-term training instances for actions in h3.6M.
20000
Count
5000
4393
2030
Inserbal Simar
desting Postz
Amenan
labiin
Action
(b) Distribution of training instances for actions in CMU.
Figure 2
more, in both cases walking together dominates the false negatives for walking (50%, and 44% in each case) well as the false positives (33% in each case).
The general increase in the distinguishability that can be seen figure 4 increases the demand to be able to robustly handle distributional shifts the distribution of values that represent different actions only gets more pronounced as the time scale is increased. This is true with even the näive DCT transformation to capture longer time scales without increasing vector size.
As we can see from the confusion matrix in figure 5, the actions in the CMU dataset are even more easily separable. In particular, our selected ID action in the paper, Basketball, can be identified with 100% precision and recall on the test set.
13

---

Under review as a conference paper at ICLR 2021
35.00%
1.2%%
500%
0.704
0.21%
5.07%
6.7%%
0.00%
0.00%
0.2%
040%
5.2%
5.00%
000%
3112%
132%
67.22%
500%
122%
442%%
4.00%
3.7%
252%
040%
8.44%
132%
4.22%
0.0%%
332%
2.225
243%
1.5%
0.0%%
0.0%%
040N
2.2%
212%
000%
0100%
0.00%
3,0%%
12 5.%%%
25.227
23
6.0%
5.00%
0.0%
022%
048%
2.2%%
2.53%
000%
000%
132%
0.00%
12 20%
339
00%
830%
0.2%%
042%
0.22%
230%
0.0%%
000
3.22%
1315
43.33
0404
3.0%
232%
0.00%
3.0%%
3.54%
28. 34%
332%
6.0%
0.3%%
13.73%
022%
040%
0.2%
2.5%
844%
0.00%
137
3.15%
0.244
0.0%%
3.00%
13%%
3.4h%
0.04%
23 025
6.4%%
5.1%%
0.0%%
0.0
332%
4.00%
13.5%%
4.7%%
240%
30.227
6.24%
4.2%
-12
NAT
230%
2.41%
120%
8.15%
0.0%
5.0%%
145%
30,00h
120%
822%
042%
732%
22%
000%
1.0%
13%%
0.10%
33%
25 73%
4.2%%
8.0%%
0.00%
042%
3.2%
0.00%
0.0%
the same input convention 7g = [ TA,N + FK,N+T] where #R,n TA.N for n 2 N. Such that in each case input the classifier is 48 x 20 960. The classifier has 4 fully connected layers. Layer 1: input dimensions x 1024, layer 2: 1024 512, layer 3: 512 x 128, layer 4: 128 15 128 8 for CMU). Where the final layer uses a sofumax to predict the class label. Cross entropy loss is used for training and ReLU activations with a dropout probability of 0.5. We used batch size of 2048, and a learning rate of 0. 1.0000..3..0 dataset. N 10, T 10. Number of DCT coefficients = 20 (lossesless transformation).
Figure 3: Confusion matrix for multi-class classifier for action labels. In each
14

---

Under review as a conference paper at ICLR 2021
94.30%
6.5%
500%
0.41%
0.00%
5.00%
61.5%
0.52%
0000%
0.20%
040%
5.22%
4.33%
000%
3.50%
132%
500%
244%%
1225
273%
0.00%
040%
0.02%
0.00%
5.60%
0.00%
332%
0.00%
0.0%%
1.10%
0.07%
0.00%
3.7%%
3.22%
000%
0.00%
3,00%
13
24.44%
55.92%
6.00%
0.00%
0.0%%
020%
0485
0.22%
40%
000%
0.00
132%
0.00%
1:24
33%
1.%%
0.0%
147%
040%
0.0%%
179%
0.00%
0.0%
06
300
3.22%
1.22%
03.33
0404
3.30%
2.0%%
2.0%
0.00%
3.0%
3.24%
21.07%
130%
6.0%
0.0%%
02 44%
822%
040%
2.2%
0.0%
842%
0.00%
14
13%
2.50%
2.4%%
0.00%
3.00%
4.42%
125%
1..99
30%
100%%
0.0%%
0.0%
332%
0.11%
30.23%
1.4%
3.22%
25.07%
12.57%
382%
230%
1.5%%
200%
12 12.30%
3.00%
1.2%
1.2%%
64%
0.0%%
022%
042%
522%
5..5%
000%
0100%
23%%
0.00%
333%
39%
3.7%%
8.0%%
0.00%
022%
042%
3.%
0.00%
0.0%
the same input convention 7g = [ [= TA,N + FK,N+T] where #R,n TA,N for n 2 N. Such that in each case input the classifier is 48 x 20 960. The classifier has 4 fully connected layers. Layer 1: input dimensions x 1024, layer 2: 1024 512, layer 3: 512 x 128, layer 4: 128 15 128 8 for CMU). Where the final layer uses a sofumax to predict the class label. Cross entropy loss is used for training and ReLU activations with a dropout probability of 0.5. We used batch size of 2048, and learning rate of 0.00001. H3.6M dataset. N 50, T 10. Number of DCT coefficients = 20, where the 40 highest frequency DCT coefficients are culled.
Figure 4: Confusion matrix for multi-class classifier for action labels. In each
15

---

Under review as a conference paper at ICLR 2021
10
0.00%
5.0%
0007
0100%
20.00%
12.11%
0.00%
008
0.00%
0.00%%
0.00%
0.00%
066
0.00%
0.00%
0.00%
0.00%
0.00%
0.00%
C.0%
89%
0.00%
0.00%
004
13000
0.00%
0.00%
0.00%
0.00%
0.00% 0.00%
0.01%
0.00%
50%
0.0%
segnal
Figure 5: Confusion matrix for multi-class classifier for action labels. In each case we use the same input convention Zs [#k,1...., TK,N F+IN++1. N+1. ..... FK.N+T] where 2k,n n TK.N for N. Such that in each case input to the classifier is x 20 = 960. The classifier has fully connected layers. Layer impul dimensions 1024, layer 2: 1024 512, layer 3: 512 128, layer 4: 128 15 (or 128 x 8 for CMU). Where the final layer uses a softmax to predict the class label. Cross entropy loss used for training and ReLU activations with a dropout probability of 0.5. We used batch size of 2048, and a learning rate of 0.00001. CMU dataset. = 10, T = 25. Number of DCT coefficients 35 (lossesless transformation).
16

---

Under review as a conference paper at ICLR 2021
B FULL RESULTS
Walking (ID)
Eating (OoD)
Smoking (OoD)
Discussion
(OoD)
milliscconds
100
320
400
80
100
320
400
80
160
320
400
80
160
320
400
GCN (OnD)
0.22
0.37
0.60
0.65
0.22
0.38
0.65
0.79
0.28
01.55
1.08
1.10
0.29
0.65
0.98
1.18
0.008
0.008
0.003
0.03
0.04
0.01
0.01
0.02
0.004
0.01
0.04
ours (OeD)
0.23
0.37
0.59
0.64
0.21
0.37
0.59
0.72
0.28
0.54
1.01
0.99
0.31
0.65
0.97
1.07
Sed Dev
0.003
0.004
0.03
0.03
0.008
0.01
0.03
0.04
0.005
0.01
0.01
0.02
0.005
0.009
0.02
0.01
Directions (OoD)
Greeting (OoD)
Phoning (OnD)
Posing (OnD)
milliseconds
320
320
160
320
GCN (Oai))
0.38
0.59
0.82
0.92
0.48
0.81
1:25
1.44
0.58
1.12
1.52
1.61
0.27
0.59
1,26
1.53
Dey
0.01
0.05
0.006
0.01
0.02
0.006
0.01
0.01
0.1
ours (Oel3
0.38
0.58
0.79
0.90
0.49
0.81
1.24
1.43
0.57
1.10
1.52
1.61
0.33
0.68
1.25
1.5T
Dev
0.007
0.0
0.006
0.005
0.02
0.02
0.004
0.003
0.01
0.02
0.03
Purchases (OoD)
Sitting (OoD)
Sitting
Down (OnD)
Taking Photo (OoD)
millisecoedss
80
160
320
400
80
160
320
400
80
160
320
400
80
160
320
400
GCN (OoD)
0.62
0.90
1.34
1.2
0.40
0.66
1,15
1.33
0,46
0.94
1.52
1.69
0.26
0.53
0.82
0.93
Std Dev
0,001
0.001
0.02
0.03
0.003
0.007
0.02
0.03
0.01
0.05
0.04
0.05
0.005
0.01
0.01
0.02
ours (OaD)
0.62
1.23
0.39
105
0.40
1.3
0.26
0.52
0.81
Std Des
0.001
0.002
0.005
0.01
0.001
0.001
0.004
0.005
0.007
0.009
0.01
0.02
0.005
0.01
0.01
0.01
Waiting (Ool)
Walking Dop
(OoD)
Walking Together (OoD)
Average for OnD)
milliseconds
80
160
320
400
80
160
320
400
80
160
320
400
80
160
320
400
GCN (OuD)
1,06
1.30
0.67
1.09
1.27
Std Dev
0.01
0.03
0.05
0.05
0.01
0.02
0.02
0.03
0,005
0.02
0.03
0.03
0.007
0.02
0.04
0.04
ours (OoD)
0.29
0.58
1,06
1.29
0.52
0.88
1.17
1.34
0.21
0.44
0.66
0.74
0.38
0.68
1.07
1.21
Std Dev
0.0007
0.003
0.001
0.006
0.006
0.01
0.008
0.01
0.01
0.01
0.01
0.01
0.006
0.01
0.01
0.02
Table 6: Short-term prediction of Eucildean distance between predicted and ground truth joint angles on H3.6M. Each experiment conducted 3 times. We report the mean and standard deviation. Note that we have lower variance in results for ours.
Basketball (ID)
Baskethall Signal (OoD)
Directing Traffic (OoD)
milliseconds
80
160
320
400
1000
80
160
320
400
1000
80
160
320
400
1000
GCN
0.40
0.67
1.11
1.25
1.63
0.27
0.55
1.14
1.42
2.18
0.31
0.62
1.05
1.24
2.49
OuTs
0,40
0.66
1.12
1.29
1.76
0.28
0.57
1.15
1.3
2.07
0.28
0.56
0.96
1.10
2.33
Jumping (OoD)
Running (OoD)
Soccer (OoD)
milliseconds
80
160
320
400
1000
80
160
320
400
1000
80
160
320
400
1000
GCN
0.42
0.73
1.72
1.98
2.66
0.46
0.84
1.50
1.72
1.57
0.29
0.54
1.15
1.41
2.14
ours
0.38
0.72
1.74
2.03
2.70
0.46
0.81
1.36
1.53
2.09
0.28
0.53
1.07
1.27
1.99
Walking (OoD)
Washing
window (OnD)
Average
(of 7 for OoD)
milliseconds
80
320
400
1000
80
160
320
400
1000
160
320
400
1000
GCN
0.40
0.61
0.97
1.18
1.85
0.36
0.65
1.23
1.51
2.31
0.36
0.65
1.41
1.49
2.17
ours
0,38
0.54
0.82
0.99
1,27
0.35
0.63
1.20
1.51
2.26
0.34
0.62
1.38
1.1
2.10
Table 7: Eucildean distance between predicted and ground truth joint angles on CMU. Full table.
Basketball (ID)
Basketball Signal (OoD)
Directing Traffic (OoD)
milliseconds
80
160
320
400
1000
80
160
320
400
1000
80
160
320
400
1000
GCN
13.7
28.9
54.1
65.4
108.4
14.4
30.4
63.3
78.7
114.8
18.5
374
75.6
93.6
210.7
ours
16.0
30.0
54.5
65.5
98.1
12.8
26.0
53.7
67.6
103.2
18.3
37.2
75.7
93.8
199.6
Jumping (OoD)
Running (OoD)
Soccer (OoD)
milliseconds
80
160
320
400
1000
80
160
320
400
1000
80
160
320
400
1000
GCN
24.6
51.2
111.4
139.6
219.7
32.3
54.8
85.9
99.3
99.9
22.6
46.6
92.8
114.3
192.5
ouTs
25.0
52.0
110.3
136.8
200.2
29.8
50.2
83.5
98.7
107.3
21.1
44.2
90.4
112.1
202.0
Walking (OoD)
Washing window (OoD)
Average of 7 for (OoD)
milliseconds
80
160
320
400
1000
80
160
320
400
1000
80
160
320
400
1000
GCN
10.8
20.7
42.9
53.4
86.3
17.1
36.4
77.6
96.0
151.6
20.0
43.8
86.3
105.8
169.2
ours
10.5
18.9
39.2
48.6
72,2
17.6
37.3
82.0
103.4
167.5
21.6
42.3
84.2
103.8
164.3
Table 8: Mean Joint Per Position Error (MPJPE) between predicted and ground truth 3D Cartesian coordinates of joints on CMU. Full table.
17

---

Under review as a conference paper at ICLR 2021
Walking (ID)
Eating (Oel)
Smaking (OoD)
Discussion (OoD)
milliseconds
560
720
$80
1000
560
720
880
1000
560
720
880
1000
560
720
880
attention-XC (OoD)
120.3
102.9
108.7
130.0
outs (DoD)
58.7
60.6
65.5
69.1
81.7
94.4
102.7
109.3
80.6
89.9
99.2
164.1
113.4
129.0
134.3
139.
Greeting (OnD)
Posing
millisoconds
720
Mennoo--GNN (O6D)
107.0
1236
1327
138.4
127.4
142.0
1554
138.6
98.7
117.3
120.9
1384
131.0
176.0
189.
199.6
ours (OnD)
107.1
120.6
129.2
136.6
128.0
140.3
150.8
155.7
95.8
111.0
122.7
1314
158.7
1813
194.4
203.4
(Ool))
Sining
(OaD)
Siming Down
Taking
(OaD)
milliseconds
720
880
1000
560
720
880
1000
720
880
1000
720
850
1000
Mlonhoe--UCN (Oab)
1366
144.0
154.3
162.1
118.1
141.1
154.
164.0
1368
1623
177.
1879
1137
137.5
14077
1900
Our (Chal3)
1280
143.2
134.7
164.3
118.
137.7
149.7
157.5
1368
157.6
1708
1804
116.3
134.
1456
1554
Waiting (OoD)
Walking Dog (Ohl))
Walking Together (OoD)
Average
OnD)
milliseaonds
560
720
880
1000
560
720
880
1000
560
720
880
1000
560
720
880
1000
Jilenion--CN GCN (OoD)
109.9
125
135.3
142
131.
146.9
161.1
1714
64.5
71.1
768
80.8
112.1
129. 1403
1478
Table 9: Long-term prediction of 3D joint positions on H3.6M. Here, ours is also trained with the attention-GCN model.
18

---

Under review as a conference paper at ICLR 2021
C LATENT SPACE OF THE VAE
One of the advantages of having generative model involved is that we have a latent variable which represents distribution over deterministic encodings of the data. We considered the question of whether or not the VAE was learning anything interpretable with its latent variable as was the case in Kipf & Welling (2016).
The purpose of this investigation was two-fold. First to determine if the generative model was learn- ing comprehensive internal state, or just a non-linear average state as is common to see in the training of VAE like architectures. The result of this should suggest key direction of future work. Second, an interpretable latent space may be of paramount usefulness for future applications of human motion prediction. Namely, if dimensionality reduction of the latent space to inspectable number of dimensions yields actions, or behaviour that are close together kinematically or teleolgi- cally similar, as in Bourached & Nachev (2019), then human experts may find unbounded potential application for a interpretation that is both quantifiable and qualitatively comparable to all other classes within their domain of interest. For example, medical doctor may consider patient to have unusual symptoms for condition, say, A. It may be useful to know that the patient's 's deviation from classical case of A, is in the direction of condition, say, B.
We trained the augmented GCN model discussed the main text with all actions, for both datasets. We use Uniform Manifold Approximation and Projection (UMAP) (McInnes al., 2018) project the latent space of the trained GCN models onto 2 dimensions for all samples in the dataset for each dataset independently. From figure 6 can that for both models the 2D project relatively closely resembles a spherical gaussian. Further, we can see from figure 6b that the action walking does not occupy a discernible domain of the latent space. This result further verified by using the same classifier as used in appendix A, which achieved no better than chance when using the latent variables as input rather than the raw data input.
This result implies that the benefit observed in the main text is by using the generative model is significant even the generative model has poor performance itself. In this case we can be sure that the reconstructions are at least not good enough to distinguish between actions. It is hence natural for future work to investigate if the improvement on OoD performance greater trained in such way to ensure that the generative model performs well. There are multiple avenues through which such an objective might achieve. Pre-training the generative model being one of the salient candidates.
19

---

Under review as a conference paper at ICLR 2021
(b) H13.6M. All actions in blue: opacity=0.1. Walking red: opacity=l.
(a) H3.6M. All actions, opacity=0.1.
(c) CMU. All actions in blue: opacity=0.1.
Figure 6: Latent embedding of the trained model on both the H3.6m and the CMU datasets inde- pendently projected in 2D using UMAP from 384 dimensions for H3.6M, and 512 dimensions for CMU using default hyperparameters for UMAP.
20

---

Under review as a conference paper at ICLR 2021
D ARCHITECTURE DIAGRAMS
21

---

Under review as a conference paper at ICLR 2021
100
130
91
7,29
9956
$000
995
100
730
20
20
00
22
Figure 7: Network architecture with discriminative and VAE branch.

---

Under review as a conference paper at ICLR 2021
GCB
GCL(
20
20
K
20
2066
Figure 8: Graph Convolutional layer (GCL) and a residual graph convolutional block (GCB).
23


Reviews:
[{"review_id": "sK5JWeWRW_C", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes a method and benchmark for out-of-distribution modeling and evaluation of human motion. They evaluate against state-of-the-art human motion methods, and show favorable performance against them.\n\n\nPros\n+ Generative model formulation for human motion prediction\n+ Benchmark for testing out of distribution performance in Human 3.6M and CMU-Mocap\n+ Proposed generative model outperforms baselines\n\n\nComments / Suggestions:\n- Interpretability claim:\nThe authors talk about facilitating interpretability in the abstract, however, I fail to find any clear experiments suggesting this. For example, I cannot find analysis of the different dimensions in the learned latent space or anything of that nature. I see section B in the supplementary material discusses interpretability, but I fail to find any clear cut results about this. Can the authors clarify how this claim is reflected in the paper?\n\n- Evaluations:\nThe evaluations provided in this paper are based on euclidean distance measured with respect to the ground truth. While this metric is reasonable, it may also not be enough to evaluate a generative model of motion (e.g., there are multiple plausible futures given a single past). Given that there are clearly defined actions in the used datasets, I would suggest using a metric that measures the generated sequences as a whole. For example, one can train a motion recognition network which given a motion tells us what type of motion we are observing. The authors could train this type of network and test it on their generated motion to see if the predicted / generated motion is recognized as the right category. Another similar evaluation would be FID, where the authors can see if the predicted / generated motion distribution in feature space is close to the ground truth distribution.\n\n- Differences with Kipf & Welling, 2016:\nThe authors mention that they adopt VGAE from Kipf and Welling, but I fail to find where the authors mention what are the specific differences of their method in comparison to Kipf & Welling, 2016. Can the authors clarify this or point out where the specific differences are mentioned?\n\n\nConclusion:\nThe proposed benchmark is interesting and useful for out-of-distribution evaluations, however, some evaluations may be missing to make this more comprehensive. The differences between the method used by the authors and the related work need to be clarified. I am willing to change my score if the authors successfully address the issues mentioned above.\n\n###########################\n  Post Rebuttal Comments\n###########################\n\nAfter reading the rebuttal, I am keeping my original score. For my first concern, they authors mention space as being a limitation for not providing analysis on the \"surveyable\" latent space, but as far as I know, additional experiments addressing my concern could have been added to the supplementary material. For my second concern, the authors talk about excelling in synthetic fidelity, however, there are fidelity measures for generative models that were not used in this submission. MSE is not a fidelity metric. I suggest that the authors address the concerns raised by the reviewers in future submissions, and it's highly likely that the work will be more solid.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "k6EK7t1UUkU", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents, firstly, a new benchmark (based on Human3.6M and CMU datasets) for human activity and motion with a high degree of out-of-distribution examples, and secondly a hybrid framework for human motion prediction which is more robust to out-of-distribution samples. \n\nOn a positive note, the presented view of human activity as highly compositional and without a clear ontology of actions and sub-actions is highly relevant, and the observed issues with the state of the art methods are completely correctly characterized. The proposed behchmark is also highly valuable to the community.\n\nHowever, the paper suffer from two flaws that renders it unfit for publication in ICLR in its current form. \n* Firstly, the contribution - to combine GCN with the approach of (Myronenko 2018) to regularize the training with a generative model that takes unlabeled samples into accound, and to replace their VAE framework with a corresponding VGAE one -  is not on its own significant enough to serve as a basis for an ICLR paper. \n* Secondly, the experiments are not adequate in that the method is only compared to a GCN without the generative model - and not with any of the other state-of-the-art in motion prediction. Moreover, results are presented without standard deviations which makes it hard to determine if the improvements are significant.\n\nThese two flaws leads to a Reject recommendation, but the authors are highly encouraged to expand the experiments to empirically verify that the proposed contribution is significant enough to warrant publication, and resubmit to a later conference. While addressing the second flaw, it would also help convince the reader of the significance of the method contribution.", "extra_scores": {}, "extra_text": {}}, {"review_id": "blQNVIlMJGk", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a generative model to solve the OoD problem in human motion prediction. It extends the GCN and attention-GCN works with VGAE for predicting human motions that are  different from ones used in training. Experiments are performed on H36M and CMU benchmarks for illustrating the efficacy of the proposed approach.\n\nPros:\n1. The paper is good in writting and easy to follow the idea\n2. The perspective of using generative model to deal with OoD problem in human motion is novel.\n\nCons.\n1. My major concern is the effectiveness of the proposed approach. From the results shown in Table 3 to Table 5, we could find that the proposed approach fail to solve the OoD problem for some actions when comparing with the baseline attention-GCN. For example, in Table 5, the proposed generative model achieve poor performance than attention-GCN, such as Discussion, Posing, Purchases, Walking Dog and Walking Together (5 out of 14 acitions). These experiments could not provide convinced results to depict the efficacy of the proposed approach. I think the authors should provide more explanations on this which should make this paper stronger. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "6KUeTkR9rOk", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary:\n---\nThis paper raises and studies concerns about the generalization of 3D human motion prediction approaches across unseen motion categories. The authors address this problem by augmenting existing architectures with a VAE framework. More precisely, an encoder network that is responsible for summarizing the seed sequence is shared by two decoders for the reconstruction of the seed motion and prediction of the future motion. Hence, the encoder is trained by using both the ELBO of a VAE and the objective of the original motion prediction task. \n\n## Pros:\n--- \nThe paper has a novel and interesting direction as robustness to distribution shifts has not been studied before in 3D human motion modeling. It is implemented around one of the SoTA models based on Graph Convolutional Networks (GCN) using discrete cosine transformation (DCT) features extracted from the motion sequence. To simulate the out-of-distribution scenario, the baselines and the proposed extension are trained on a single action category such as walking and tested on the remaining actions such as eating and sitting. Experiment results on the H3.6M and CMU datasets show that the proposed approach is useful on out-of-distribution (OoD) test cases. \n\n## Cons:\n--- \nI have two main concerns on the proposed benchmark and the models. \n \n-- OoD Benchmark--\n- It looks like there is a significant underfitting problem. The performance of GCN on the walking category is 0.56 at 400 ms while the in-distribution (ID) performance with OoD training is 0.66 (Table 1). The training split for the OoD setup proposed by the authors is possibly too small. I also can not grasp the motivation for selecting a training set “as small in quantity, and narrow in domain as possible” (Section 3). While there is not enough or barely enough training samples, the comparisons might be misleading. We do not know how the proposed extension behaves on the standard task. The authors should compare their models on the main task as well.\n \n- Motion samples from different categories (i.e., walking, eating, etc.) can still be useful for the models in learning the 3D human motion prior. In fact, it has been shown by Martinez et al. (2017) [4] that training motion models with _all_ available actions improves the performance significantly compared to a single-action models as done in this paper. While the proposed approach outperforms the baselines in average performance, it is not always or substantially better on the fine-grained actions. \n \n- It is a tedious setup, but a leave-one-action-out strategy can be more reliable. In my opinion a better option would be training on one dataset and testing on another one. This would allow for an evaluation of the existing (and even pre-trained) models directly where the proposed extension would remain as the only factor for evaluation. In the context of H3.6M and CMU datasets, this might not be straightforward due to different skeletal configurations. Yet there exists a much larger benchmark for 3D motion prediction: AMASS [1]. This would be a suitable candidate for this task as it is a collection of several diverse mocap datasets with different motion categories. It would be very easy to train on a subset of datasets and test on the remaining ones as all the datasets follow a unified skeletal configuration. Note that this is only a suggestion to improve the current work and I am not asking for running experiments on AMASS for the rebuttal as it would drastically change the submission.\n \n-- Quantifying the OoD--\n- The existing architectures are augmented with a VAE latent space and a decoder, which is not technically novel. However, regularization of the representation space and reconstruction of the inputs as auxiliary tasks seem like helpful to the motion prediction task and a good contribution under the “limited” training/evaluation protocol. At the same, time the proposed evaluation protocol is not orthogonal to the task but instead it just follows the existing task protocol. In other words, it is not an independent metric/method/framework for assessing the existing models’ OoD performance. \n\nI am asking the following questions as the proposed approach is presented as a “framework”:\n\n- The authors hypothesize that motion prediction in generative modeling frameworks can alleviate the OoD problems. I find it too broad as generative modelling can be applied in various frameworks. Although it is conceptually very different, [1] uses an auto-regressive model, which is a generative model by design, and trains the model by predicting both the seed (i.e., loosely reconstruction) and the future frames similar to the proposed approach. Can we say that they also deal with the OoD problems implicitly? How do the authors position their “framework” compared to this line of work? \n\n- The authors only focus on the Seq2seq-based methods for motion prediction and choose a baseline with an implicit temporal model (i.e., using DCT to encode the motion sequences). Hence, the proposed approach seems to be limited to this GCN-based architecture. How can the sequential deterministic models [1, 3, 4] be addressed? \n \n-- Additional comments --\n- Figures should be improved. Especially the text is hardly readable. \n\n- I find it very hard to follow Section 3. It was clear only after I read the section A in the appendix. It would be clearer if some of the findings are discussed in Section 3 already. \n\n- The losses in the tables are too high compared to the actual task. I am not sure if there is a qualitative difference between the models as the authors did not present any qualitative results. \n\n- Missing related work on 3D motion modelling. I list a s,all collection of SoA representatives below:\n \n[1] Aksan, Emre, Manuel Kaufmann, and Otmar Hilliges. \"Structured prediction helps 3d human motion modelling.\" Proceedings of the IEEE International Conference on Computer Vision. 2019.\n[2] Gui, L. Y., Wang, Y. X., Liang, X., & Moura, J. M. (2018). Adversarial geometry-aware human motion prediction. In Proceedings of the European Conference on Computer Vision (ECCV) (pp. 786-803).\n[3] Pavllo, D., Grangier, D., & Auli, M. (2018). Quaternet: A quaternion-based recurrent model for human motion. arXiv preprint arXiv:1805.06485.\n[4] Martinez, J., Black, M. J., & Romero, J. (2017). On human motion prediction using recurrent neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2891-2900).\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for out-of-distribution modeling and evaluation in  the human motion prediction task. Paper was reviewed by four expert reviewers who identified the following pros and cons.

> Pros:
- New benchmark for testing out of distribution performance [R1]
- Compelling performance with respect to the baselines [R1,R4]
- Paper is well written and easy to  follow  [R2]
- Generative model in the context  of  out-of-distribution modeling of human motion is novel [R1,R2,R4]

> Cons:
- Lack of support for interpretability claim  [R1]
- Validity and usefulness of the metric [R1]
- Lack of "effectiveness" of the proposed approach [R2,R4]
- Technical contributions are not significant [R3,R4]
- Experimental validation lacks comparisons to other state-of-the-art in motion prediction  methods [R3] 
- Lack of evaluation on additional datasets and for the main task [R4]

Authors tried to address the comments in the rebuttal, but  largely unconvincingly to the  reviewers.  On balance, reviewers felt that negatives outweighed the positives and unanimously suggest rejection. AC concurs and sees no reason to overturn this consensus. 

Decision: Reject