Action-guided 3D Human Motion Prediction

1 Introduction . Imagining and predicting the future motion of a person doing some actions ( e.g. , walking , standing up ) is a natural ability of human beings . However , it is still very challenging for machines to achieve reasonable forecasting . In this work , we focus on 3D human motion prediction task which aims to forecast the future state of the 3D human body conditioned on several observed past video frames . Future motion prediction is important for many real-world applications like human-machine interaction and autonomous driving . The main challenge is that the future motion can vary greatly ( i.e. , the prediction space is large ) , which would increase the uncertainty and ambiguity of future motion prediction . We observe that there exists a strong correlation between human motion and action category , which could be exploited to reduce such uncertainty and ambiguity . As illustrated in Figure 1 , human action always contains some representative sub-processes such as swinging arm , throwing something out , etc . We name these sub-processes as motion dynamics . By exactly examining the videos in Figure 1 , we further find that the motion dynamics could be shared across different subjects performing the same action . Motivated by the above observations , we propose to exploit representative motion dynamics for each action class to guide motion prediction and reduce the learning difficulty . ∗This work was partially done when Jiangxin Sun was an intern at Huya Inc. †Jian-Fang Hu is the corresponding author . Hu is also with Guangdong Province Key Laboratory of Information Security Technology , Guangzhou , China and Key Laboratory of Machine Intelligence and Advanced Computing , Ministry of Education , China . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Specifically , we propose a novel framework to perform future 3D motion prediction . In this framework , we construct an action-specific memory bank to store the representative motion dynamics for each action class . The constructed memory bank stores both short-term and long-term motion dynamics to capture motions with different velocities . And based on the memory bank , we further design a query-read process to query past observed motion and read candidate future motion dynamics from the memory bank . The motion dynamics read from the memory bank are used to explicitly guide future motion prediction . To ensure global motion consistency , we further formulate an action constraint loss to keep the semantic consistency between the predicted motion and the observed past motion . The proposed approach achieves promising prediction results and outperforms previous methods by a considerable margin , which demonstrates the effectiveness of using action-specific motion dynamics to guide motion prediction . Overall , our main contributions are : 1 ) we construct an action-specific memory bank to exploit the action-specific representative motion dynamics ; 2 ) we propose a novel framework that can effectively utilize the action-specific dynamics stored in the memory bank for guiding motion prediction ; 3 ) our approach achieves state-of-the-art performance on 3D human motion prediction . 2 Related work . 3D human motion prediction . 3D human motion prediction has attracted long-standing interest in the community [ 5 , 32 , 41 ] . 3D human motion is typically modeled with 3D poses [ 23 , 10 , 25 , 24 , 28 ] or parametric 3D body models [ 1 , 11 , 16 , 31 , 17 ] . Earlier methods mainly perform 3D human motion prediction using techniques such as Hidden Markov Models [ 4 ] , linear dynamical systems [ 29 ] and Gaussian process latent variable models [ 32 , 33 ] . Recently , several deep learning-based approaches ( e.g . methods employing recurrent neural networks or LSTMs [ 13 ] ) are proposed to predict human poses from past 3D human skeletons [ 6 , 45 , 9 , 41 , 2 , 7 ] . Since it is difficult to obtain in-the-wild 3D annotations , Zhang et al . [ 42 ] further proposed an autoregressive model named PHD to forecast future 3D human meshes from past 2D video inputs . However , it ignores the action information depicted in the observed video frames , which can serve as a strong prior to facilitate the prediction . In contrast , we aim to explore action context for motion prediction by designing an action-specific memory bank to discover the representative motion dynamics within each action class . With a query-read process , we explicitly utilize the information stored in the memory bank of each action class for guiding motion prediction . Memory networks . Internal memory has been widely used to capture long-term temporal dependencies within sequences ( e.g. , LSTMs [ 13 ] , GRUs [ 8 ] ) and can be implicitly updated in a recurrent process . In order to address the inherent instability of internal memory over long timescales , an external memory component has been developed to augment the sequence modeling and achieved improved performance on related tasks [ 34 , 30 , 18 , 26 ] . The memory component can be read or written with an attention-based procedure and it can store useful information depicted in a sequence or a set of sequences . Recently , memory-based approaches have been developed in many vision tasks such as image captioning [ 37 , 40 ] , tracking [ 46 , 19 ] , and video understanding [ 35 , 27 ] . Unlike traditional memory networks using a single memory to store all the information , we construct a memory bank to store independent memory components for each action category . Input Image 3 Our approach . The goal of 3D human motion prediction is to forecast the future state of 3D human body presented in a given video . We follow [ 16 , 42 ] to decompose the representation of 3D human body into three independent components θ , β , and Π. θ ∈ R72 and β ∈ R10 denote the pose and shape parameters of SMPL model [ 21 ] , respectively . The pose parameters θ represent the global body rotation and the relative rotation of 23 body joints in axis-angle , which is closely related to action dynamics . The shape parameters β are the first 10 coefficients of a PCA space modeling body shapes , thus containing personal information such as height or waistline . Based on the constructed θ and β , the SMPL model generates a triangulated mesh M ( θ , β ) ∈ RN×3 with N = 6890 vertices . With a pre-trained linear regressor , the 3D coordinates of body joints X can be calculated from M ( θ , β ) . Π = [ s , tx , ty ] is the weak-perspective camera parameter encoding the scaling , x-offset and y-offset information of an orthographic projection , which is used to infer the 2D coordinates of body joints x from 3D coordinates X . We address the problem of 3D human motion prediction by learning mappings Φ , Ψ , and Θ to predict the 3D human body representations θt+1 : T , βt+1 : T and Πt+1 : T of future un-observed video frames , respectively , from the representations θ1 : t , β1 : t and Π1 : t of the observed past frames . The prediction process can be mathematically expressed as : θt+1 : T = Φ ( θ1 : t ) , βt+1 : T = Ψ ( β1 : t ) , Πt+1 : T = Θ ( Π1 : t ) . ( 1 ) The overall framework is presented in Figure 2 . We follow the common practice in future prediction tasks [ 45 , 9 , 2 , 7 ] to develop an encoder-LSTM-decoder architecture to perform the prediction . Specifically , we first employ an encoder to extend the representations θ , β , Π to higher dimensional latent codes B ( θ ) , B ( β ) , B ( Π ) , respectively . Then , three independent LSTMs are used to predict the latent representations for future frames . Finally , the decoder is employed to transform the predicted representations back to the space of θ , β , Π , which are fed into the SMPL model to construct 3D mesh/skeleton presentations . Since the human motion has a strong correlation with the corresponding action category ( e.g. , one could better infer the future body motion when given the performing action ) , we seek to explore the action context with a novel action memory bank to guide the motion prediction in this work . The action memory bank is only used to help the prediction of pose parameters θ as they are closely related to the action category , while for the prediction of action-irrelevant representations like shape parameters β and camera parameters Π , we employ the standard encoder-LSTM-decoder architecture . In the following , we will mainly describe details of our framework for predicting SMPL pose parameters ( i.e. , learning mapping Φ in Eq . 1 ) with our proposed action memory bank . 3.1 Action-guided feature prediction . Actions of the same category always contain specific sub-processes and share some common dynamics ( see Figure 1 ) , which are useful to guide future motion prediction . In this work , in order to take advantage of the action prior knowledge , we propose to construct a class-specific action memory bank to store the common dynamics of each action class and define a query-read process to utilize the dynamics stored in the memory bank as illustrated in Figure 3 . In the query process , we calculate the similarity between the observed data and the dynamics contained in the pre-constructed memory bank conditioned on a predicted action class . In the read process , we combine the motion dynamics from the memory bank using the similarity as a soft weight . And the combined motion dynamic is finally used to guide motion prediction . 3.1.1 Action memory bank construction . Our action memory bank is constructed to store the representative motion dynamics appeared in each action class . Specifically , we represent the short-term motion dynamic as st : t+∆t−1 = [ θt , θt+1 , θt+2 , ... , θt+∆t−1 ] , which consists of the initial pose θt and the pose dynamics of the following ∆t − 1 frames . That is to say , the proposed representation describes motion as the dynamics of human body joint rotations . In order to further capture long-term motion dynamics , we extend the sampling time step to K and represent the long-term motion dynamic as lt : t+∆t−1 = [ θt , θt+K×1 , θt+K×2 , ... , θt+K× ( ∆t−1 ) ] . Then for each action class , we utilize the Affinity Propagation ( AP ) Clustering algorithm to cluster the short-term and long-term motions over all the training videos . We use L1 distance to measure the similarity of different motion patterns . We discard the clusters containing too few elements ( less than 1 % ) and regard the remaining cluster centers as the representative motion dynamics of the corresponding action class to form our action memory bank M : M = { ( Si , Li ) | i = 1 , 2 , ... , I } , ( 2 ) where Si ∈ R ( ∆t×72 ) ×N short i , Li ∈ R ( ∆t×72 ) ×N long i correspond to short-term motion and long-term motion memory for the action category i , respectively . Each item in the memory bank is a clustering center and ∆t is the time length of each item . Nshorti and N long i are the number of short-term and long-term items for action i in the memory bank , respectively . I denotes the number of action classes .

Summary: The paper introduces an external memory component to the baseline method PHD on the task of human action prediction. This external memory component is designed so that it clusters common actions across videos conditioned on the action class. A query-read mechanism is designed to train and retrieve similar actions using attention. This memory component is then used to provide information to the LSTM that predicts future parameters pertaining to action conditioned on the past. A combined loss of different components pertaining to the training of the different parameters is used. This method achieves promising results, and the action memory component is able to cluster relevant actions. An ablation study is also provided to check the contribution of the action memory component.