Action-guided 3D Human Motion Prediction

1 Introduction . Imagining and predicting the future motion of a person doing some actions ( e.g. , walking , standing up ) is a natural ability of human beings . However , it is still very challenging for machines to achieve reasonable forecasting . In this work , we focus on 3D human motion prediction task which aims to forecast the future state of the 3D human body conditioned on several observed past video frames . Future motion prediction is important for many real-world applications like human-machine interaction and autonomous driving . The main challenge is that the future motion can vary greatly ( i.e. , the prediction space is large ) , which would increase the uncertainty and ambiguity of future motion prediction . We observe that there exists a strong correlation between human motion and action category , which could be exploited to reduce such uncertainty and ambiguity . As illustrated in Figure 1 , human action always contains some representative sub-processes such as swinging arm , throwing something out , etc . We name these sub-processes as motion dynamics . By exactly examining the videos in Figure 1 , we further find that the motion dynamics could be shared across different subjects performing the same action . Motivated by the above observations , we propose to exploit representative motion dynamics for each action class to guide motion prediction and reduce the learning difficulty . ∗This work was partially done when Jiangxin Sun was an intern at Huya Inc. †Jian-Fang Hu is the corresponding author . Hu is also with Guangdong Province Key Laboratory of Information Security Technology , Guangzhou , China and Key Laboratory of Machine Intelligence and Advanced Computing , Ministry of Education , China . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Specifically , we propose a novel framework to perform future 3D motion prediction . In this framework , we construct an action-specific memory bank to store the representative motion dynamics for each action class . The constructed memory bank stores both short-term and long-term motion dynamics to capture motions with different velocities . And based on the memory bank , we further design a query-read process to query past observed motion and read candidate future motion dynamics from the memory bank . The motion dynamics read from the memory bank are used to explicitly guide future motion prediction . To ensure global motion consistency , we further formulate an action constraint loss to keep the semantic consistency between the predicted motion and the observed past motion . The proposed approach achieves promising prediction results and outperforms previous methods by a considerable margin , which demonstrates the effectiveness of using action-specific motion dynamics to guide motion prediction . Overall , our main contributions are : 1 ) we construct an action-specific memory bank to exploit the action-specific representative motion dynamics ; 2 ) we propose a novel framework that can effectively utilize the action-specific dynamics stored in the memory bank for guiding motion prediction ; 3 ) our approach achieves state-of-the-art performance on 3D human motion prediction . 2 Related work . 3D human motion prediction . 3D human motion prediction has attracted long-standing interest in the community [ 5 , 32 , 41 ] . 3D human motion is typically modeled with 3D poses [ 23 , 10 , 25 , 24 , 28 ] or parametric 3D body models [ 1 , 11 , 16 , 31 , 17 ] . Earlier methods mainly perform 3D human motion prediction using techniques such as Hidden Markov Models [ 4 ] , linear dynamical systems [ 29 ] and Gaussian process latent variable models [ 32 , 33 ] . Recently , several deep learning-based approaches ( e.g . methods employing recurrent neural networks or LSTMs [ 13 ] ) are proposed to predict human poses from past 3D human skeletons [ 6 , 45 , 9 , 41 , 2 , 7 ] . Since it is difficult to obtain in-the-wild 3D annotations , Zhang et al . [ 42 ] further proposed an autoregressive model named PHD to forecast future 3D human meshes from past 2D video inputs . However , it ignores the action information depicted in the observed video frames , which can serve as a strong prior to facilitate the prediction . In contrast , we aim to explore action context for motion prediction by designing an action-specific memory bank to discover the representative motion dynamics within each action class . With a query-read process , we explicitly utilize the information stored in the memory bank of each action class for guiding motion prediction . Memory networks . Internal memory has been widely used to capture long-term temporal dependencies within sequences ( e.g. , LSTMs [ 13 ] , GRUs [ 8 ] ) and can be implicitly updated in a recurrent process . In order to address the inherent instability of internal memory over long timescales , an external memory component has been developed to augment the sequence modeling and achieved improved performance on related tasks [ 34 , 30 , 18 , 26 ] . The memory component can be read or written with an attention-based procedure and it can store useful information depicted in a sequence or a set of sequences . Recently , memory-based approaches have been developed in many vision tasks such as image captioning [ 37 , 40 ] , tracking [ 46 , 19 ] , and video understanding [ 35 , 27 ] . Unlike traditional memory networks using a single memory to store all the information , we construct a memory bank to store independent memory components for each action category . Input Image 3 Our approach . The goal of 3D human motion prediction is to forecast the future state of 3D human body presented in a given video . We follow [ 16 , 42 ] to decompose the representation of 3D human body into three independent components θ , β , and Π. θ ∈ R72 and β ∈ R10 denote the pose and shape parameters of SMPL model [ 21 ] , respectively . The pose parameters θ represent the global body rotation and the relative rotation of 23 body joints in axis-angle , which is closely related to action dynamics . The shape parameters β are the first 10 coefficients of a PCA space modeling body shapes , thus containing personal information such as height or waistline . Based on the constructed θ and β , the SMPL model generates a triangulated mesh M ( θ , β ) ∈ RN×3 with N = 6890 vertices . With a pre-trained linear regressor , the 3D coordinates of body joints X can be calculated from M ( θ , β ) . Π = [ s , tx , ty ] is the weak-perspective camera parameter encoding the scaling , x-offset and y-offset information of an orthographic projection , which is used to infer the 2D coordinates of body joints x from 3D coordinates X . We address the problem of 3D human motion prediction by learning mappings Φ , Ψ , and Θ to predict the 3D human body representations θt+1 : T , βt+1 : T and Πt+1 : T of future un-observed video frames , respectively , from the representations θ1 : t , β1 : t and Π1 : t of the observed past frames . The prediction process can be mathematically expressed as : θt+1 : T = Φ ( θ1 : t ) , βt+1 : T = Ψ ( β1 : t ) , Πt+1 : T = Θ ( Π1 : t ) . ( 1 ) The overall framework is presented in Figure 2 . We follow the common practice in future prediction tasks [ 45 , 9 , 2 , 7 ] to develop an encoder-LSTM-decoder architecture to perform the prediction . Specifically , we first employ an encoder to extend the representations θ , β , Π to higher dimensional latent codes B ( θ ) , B ( β ) , B ( Π ) , respectively . Then , three independent LSTMs are used to predict the latent representations for future frames . Finally , the decoder is employed to transform the predicted representations back to the space of θ , β , Π , which are fed into the SMPL model to construct 3D mesh/skeleton presentations . Since the human motion has a strong correlation with the corresponding action category ( e.g. , one could better infer the future body motion when given the performing action ) , we seek to explore the action context with a novel action memory bank to guide the motion prediction in this work . The action memory bank is only used to help the prediction of pose parameters θ as they are closely related to the action category , while for the prediction of action-irrelevant representations like shape parameters β and camera parameters Π , we employ the standard encoder-LSTM-decoder architecture . In the following , we will mainly describe details of our framework for predicting SMPL pose parameters ( i.e. , learning mapping Φ in Eq . 1 ) with our proposed action memory bank . 3.1 Action-guided feature prediction . Actions of the same category always contain specific sub-processes and share some common dynamics ( see Figure 1 ) , which are useful to guide future motion prediction . In this work , in order to take advantage of the action prior knowledge , we propose to construct a class-specific action memory bank to store the common dynamics of each action class and define a query-read process to utilize the dynamics stored in the memory bank as illustrated in Figure 3 . In the query process , we calculate the similarity between the observed data and the dynamics contained in the pre-constructed memory bank conditioned on a predicted action class . In the read process , we combine the motion dynamics from the memory bank using the similarity as a soft weight . And the combined motion dynamic is finally used to guide motion prediction . 3.1.1 Action memory bank construction . Our action memory bank is constructed to store the representative motion dynamics appeared in each action class . Specifically , we represent the short-term motion dynamic as st : t+∆t−1 = [ θt , θt+1 , θt+2 , ... , θt+∆t−1 ] , which consists of the initial pose θt and the pose dynamics of the following ∆t − 1 frames . That is to say , the proposed representation describes motion as the dynamics of human body joint rotations . In order to further capture long-term motion dynamics , we extend the sampling time step to K and represent the long-term motion dynamic as lt : t+∆t−1 = [ θt , θt+K×1 , θt+K×2 , ... , θt+K× ( ∆t−1 ) ] . Then for each action class , we utilize the Affinity Propagation ( AP ) Clustering algorithm to cluster the short-term and long-term motions over all the training videos . We use L1 distance to measure the similarity of different motion patterns . We discard the clusters containing too few elements ( less than 1 % ) and regard the remaining cluster centers as the representative motion dynamics of the corresponding action class to form our action memory bank M : M = { ( Si , Li ) | i = 1 , 2 , ... , I } , ( 2 ) where Si ∈ R ( ∆t×72 ) ×N short i , Li ∈ R ( ∆t×72 ) ×N long i correspond to short-term motion and long-term motion memory for the action category i , respectively . Each item in the memory bank is a clustering center and ∆t is the time length of each item . Nshorti and N long i are the number of short-term and long-term items for action i in the memory bank , respectively . I denotes the number of action classes .

Summary: This paper proposes a system to use predicted action class and action-specific memory to guide human motion prediction. It constructs an action memory bank that consists of observed short-term and long-term human dynamics that can be later queried and used to predict future human motion. The proposed method achieves state-of-the-art human motion forecasting on the H3.6M and PennAction datasets using only videos.