Action-guided 3D Human Motion Prediction
Authors: Jiangxin Sun
Zihang Lin
Xintong Han
Jian-Fang Hu
Jia Xu
Wei-Shi Zheng
Venue: NeurIPS 2021 Poster (2021)
Keywords: human motion prediction
3D human motion
future prediction

Abstract: The ability of forecasting future human motion is important for human-machine interaction systems to understand human behaviors and make interaction. In this work, we focus on developing models to predict future human motion from past observed video frames. Motivated by the observation that human motion is closely related to the action being performed, we propose to explore action context to guide motion prediction. Specifically, we construct an action-specific memory bank to store representative motion dynamics for each action category, and design a query-read process to retrieve some motion dynamics from the memory bank. The retrieved dynamics are consistent with the action depicted in the observed video frames and serve as a strong prior knowledge to guide motion prediction. We further formulate an action constraint loss to ensure the global semantic consistency of the predicted motion. Extensive experiments demonstrate the effectiveness of the proposed approach, and we achieve state-of-the-art performance on 3D human motion prediction.

Action-guided 3D Human Motion Prediction
Jiangxin Sun' Sun Yat-sen University sunjx56mai12. sysu. edu.
Zihang Lin
Xintong Han Huya Inc
Sun Yat-sen University
1inzh59@mai12. sysu. sysu.edu.c. edu.cn cn
hanxintongdhuya.com
Jian-Fang Hu
Jia Xu
Wei-Shi Zheng Sun Yat-sen University wszhengdiees org
Sun Yat-sen University hujf5@mail sysu edu.
Huya Inc xuj sujiahhuya.com
Abstract
The ability of forecasting future human motion is important for human- machine interaction systems to understand human behaviors and make interaction. In this work, we focus developing models to predict future human motion from past observed video frames. Motivated by the observation that human motion is closely related to the action being performed, guide motion prediction. Specifically, we construct an action-specific memory bank representative motion dynamics for each action category, and design a query-read process to retrieve some motion dynamics from the memory bank. The retrieved dynamics video frames and serve as a strong prior knowledge to guide motion prediction. We further formulate an action constraint loss ensure the global semantic consistency of the predicted motion. Extensive experiments demonstrate the effectiveness of the proposed approach, and we achieve sate-of-the-ar performance on 3D human motion prediction.
propose explore action context
consistent with the action depicted in the observed
Introduction
Imagining and predicting the future motion of person doing some actions (e.g., walking, standing up) is a natural ability of human beings. However, it is still very challenging for machines to achieve reasonable forecasting. In this work, focus 3D human motion prediction task which aims to forecast the future state of the 3D human body conditioned on several observed past video frames. Future motion prediction is important for many real-world applications like human-machine interaction and autonomous driving. The main challenge is that the future motion can vary greatly (i.e.. the prediction space large), which would increase the uncertainty and ambiguity of future motion prediction.
We observe that there exists strong correlation between human motion and action category, which could be exploited to reduce such uncertainty and ambiguity. As illustrated in Figure [ human action always contains some representative sub-pra sub-processes such swinging arm, throwing something out, etc. We name these sub-processes as motion dynamics. By exactly examining the videos in Figure 1. further find that the motion dynamics could be shared across different subjects performing the same action. Motivated by the above observations, we propose to exploit representative motion dynamics for each action class to guide motion prediction and reduce the learning difficulty.
"This work partially done when Jiangxin Sun intern Huya Inc.
$ Jiin-FFng Hu is the corresponding author. Hu is also with Guangdong Province Key Laboratory of Information Security Technology. Guangzhou, China and Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Goll Swing
Boxting
Resdaal Pich
Figure I: Illustration of representative motion dynamics. This figure shows that the motion dynamics could be shared across different subjects who are performing the same action.
Specifically, we propose a novel framework to perform future 3D motion prediction. In this framework, we construct an action-specific memory bank store the representative motion dynamics for each action class. The constructed memory bank stores both short-term and long-term motion dynamics capture motions with different velocities. And based on the memory bank, we further design query-read process to query past observed motion and read candidate future motion dynamics from the memory bank. The motion dynamics read from the memory bank are used to explicitly guide future motion prediction. To ensure global motion consistency, we further formulate an action constraint loss to keep the semantic consistency between the predicted motion and the observed past motion. The proposed approach achieves promising prediction results and outperforms previous methods by considerable margin, which demonstrates the effectiveness of using action-specific motion dynamics to guide motion prediction.
Overall, our main contributions are: construct an action-specific memory bank exploit the action-specific representative motion dynamics; 2) we propose a novel framework that can effectively utilize the -ettiinceecciicc dynamics stored in the memory bank for guiding motion prediction; 3) our approach achieves state-of of-the-an performance on 3D human motion prediction.
2
Related work
3D human motion prediction. 3D human motion prediction has attracted long-standing interest in the community [5] 32 41 3D human motion is typically modeled with 3D poses [23] 10 25 24
28
parametric 3D body models [I] 16 31
17] Earlier methods mainly perform 3D human motion
prediction using techniques such as Hidden Markov Models [4, linear dynamical systems (2 and
Recently, several deep learning- based approaches (e.g. methods employing recurrent neural networks or LSTMs (13 are proposed to predict human 41 2 7 Since it is difficult obtain in-the-wild 3D annotations, Zhang et al. [42 further proposed an autoregressive model named PHD to forecast future 3D human meshes from past 2D video inputs. However, ignores the action information depicted in the observed video frames, which can serve as a strong prior to facilitate the prediction. In contrast, aim explore action context for motion prediction by designing an action-specific memory bank to discover the representative motion dynamics within each action class. With a query-read process, we explicitly utilize the information stored in the memory bank of each action class for guiding motion prediction.
Gaussian process latent variable models 32 33
poses from past 3D human skeletons 6 45
Memory networks. Internal memory has been widely used to capture long-term temporal depen- dencies within sequences (c.g., LSTMs [3]. GRUs (8) and can be implicitly updated in a recurrent process. In order address the inherent instability of internal memory over long timescales, an external memory component has been developed to augment the sequence modeling and achieved improved performance on related tasks [34 30
18 26]. The memory component can be read or written with an attention- -bsed procedure and it can store useful information depicted in a sequence or a set of sequences. Recently, memory-based approaches have been developed in many vision tasks such as image captioning 37 40 tracking [46] 19 and video understanding [5 27 Unlike traditional memory networks using a single memory to store all the information, we construct
memory bank to store independent memory components for each action category.

---

Input Image Sequeace
Action Memory Bank
Clasdifer
U
LST M
SMPL
3D Par Paarmeeer Regressor
Model
Time
F1tc
LSTM
Projection
11:
LST
Predicted
Predicted Sequence
Sequence
Figure 2: An overview of the proposed framework. We first extract 3D human body parameters from input videos with 3D Parameter Regressor. Then we develop an encoder-LSTM--econ architecture to perform prediction from these parameters. In order to utilize action context to improve the prediction, we further design an action memory bank guide the prediction of the action-relate parameters (i.e., the predicted SMPI. pose parameters Q).
3 Our approach
The goal of 3D human motion prediction is to forecast the future state of 3D human body presented in given video. We follow [16 42 to decompose the representation of 3D human body into three independent components 0, 3, and II. T2 and /3 E IR denote the pose and shape parameters of SMPL model [21], respectively. The pose parameters O represent the global body rotation and the relative rotation of 23 body joints in axis-angle, which closely related to action dynamics. The shape parameters G are the first 10 coefficients of a PCA space modeling body shapes, thus containing personal information such as height or waistline. Based on the constructed 0 and 3, the SMPL model generates a triangulated mesh M(0,, .) E RNx3 with N 6890 vertices. With pre-trained linear regressor, the 3D coordinates of body joints X can be calculated from MM(0, B). II =I [s, tur ty] is the weak peak-perspeetive camera parameter encoding the scaling, 2-offset and y-offset information of an orthographic projection, which is used to infer the 2D coordinates of body joints x from 3D coordinates X.
We address the problem of 3D human motion prediction by learning mappings 4, V, and to predict the 3D human body representations 0r+1:7, 01+1:7,B.-1.77 and 4 of future un-observed video frames, respectively, from the representations 0 0ist,1r. PI... and I1: of the observed past frames. The prediction process can be mathematically expressed as:
1.+1.7 (0)).)) B.+1.T (Bi.),, N4 = (I...)
(1)
The overall framework is presented in Figure 2 We follow the common practice in future prediction
tasks 45 2 develop an encoder-LSS Sirroder-LSTM---- LSTM-decoder architecture perform the prediction. Specifically, we first employ an encoder to extend the representations O, B, II to higher dimensional latent codes B (0), (()), (I),, respectively. Then, three independent LSTMs are used to predict the latent representations for future frames. Finally, the decoder employed transform the predicted representations back to the space of 0, B, II, which are fed into the SMPL model to construct 3D mesh/skeleton presentations.
Since the human motion has a strong correlation with the corresponding action category (e.g., one could better infer the future body motion when given the performing action), seek to explore the action context with a novel action memory bank to guide the motion prediction in this work. The action memory bank is only used help the prediction of pose parameters as they closely related to the action category, while for the prediction of action-irrelevant representations like shape parameters G and camera parameters II, we employ the standard Lrooder--STM-decooee architecture. In the following, we will mainly describe details of our framework for predicting SMPL pose parameters (i.e., learning mapping & in Eq. I with our proposed action memory bank.
3.1 Action-guided feature prediction
Actions of the same category always contain specific sub-processes and share some common dynamics (see Figure I) which are useful to guide future motion prediction. In this work, in order to take
3

---

0+ Decoder
0++z Decoder
0 Decoder
Q uery-read
Query-read
Cless
of hort
4p
long
B
B (o) F-2
B (0)
c+1
6+7
B
Class
Call
Class 2
Class
Q uery-read
short-term dy dynamic long-term dynamic
B
B
B
Encoder
Encoder
Predicted Action Label
02
Figure 3: The architecture of the proposed action-guided predictor (left). We develop action memory bank to store action-specific motion dynamics which are coupled with a query-read process (right) guide motion prediction.
advantage of the action prior knowledge, we propose to construct class-specific action memory bank to store the common dynamics of each action class and define query-read process to utilize the dynamics stored in the memory bank as illustrated in Figure 3 In the query process, we calculate the similarity between the observed data and the dynamics contained in the pre-constructed memory bank conditioned on a predicted action class. the read process, we combine the motion dynamics from the memory bank using the similarity soft weight. And the combined motion dynamic is finally used to guide motion prediction.
3.1.1 Action memory bank construction
Our action memory bank is constructed to store the representative motion dynamics appeared in each action class. Specifically, we represent the short-term motion dynamic as $1:+-r-1 = [0,, 01+1, 0:42, .... 01+ which consists of the initial pose O2 and the pose dynamics of the following At
frames. That is to say, the proposed representation describes motion as the dynamics of human body joint rotations. In order to further capture long-term motion dynam- ics, we extend the sampling time step to K and represent the long-term motion dynamic as Then for each action class, we utilize the Affinity Propagation (AP) Clustering algorithm to cluster the short-term and long-term motions over all the training videos. We use LI distance measure the similarity of different motion patterns. We discard the clusters containing too few elements (less than I 1%) and regard the remaining cluster centers as the representative motion dynamics of the corresponding action class to form our action memory bank M
14441-1 00, 0 0x+Kx2 .... x
M {SS, L.) li 1,2,..., I.. 1}.
where S, E N( L; E IR (dxxT2) x Nouns correspond to short-term motion and long-term motion memory for the action category i, respectively. Each item the memory bank is a clustering center and At the time length of each item. Nohort and Nzlong are the number of short-term and long-term items for action i in the memory bank, respectively. I denotes the number of action classes.
3.1.2 Motion feature prediction
With the constructed action memory bank, we take advantage of action-specific representative motion dynamics guide the prediction. Specifically, design query-read process, which queries the past motion and then read the corresponding future motion dynamics from the memory bank. Inspired by previous memory based methods 30 36 38 7. the query-read process is designed based on an attention mechanism. First, we employ an action recognition network [22] on the observed frames to predict the probability of action classes and take the class with the maximum classification confidence. Then, we compute the key matrices K Khhort E X.Nhhor as Kehort W&S! and value matrices Vehort E X.C.NNNaann Vehort
W.S, where S E 1 N

---

represent the former and latter half motion in the memory bank, respectively. We E RCxX x72) and W. E RC 2 x72) are two learnable matrices. And for each time step t, we compute the query vector de E RCc from the encoded motion feature B(()) RC2 by q. W.B. +
where B (C is the concatenation [B (,+ B (8)
B After that, in the query
b +21
process, we compute the similarity between the query vector qr and the key vectors in K Kahort as:
( Khhhrt
sim
qi) (K:hort)"a qi.
(3)
In the read process, we read the future short-term motion dynamics of othrrt RC from the memory bank using the similarity as soft weight, which can be formulated as:
short
so V.hort so fooamar (sim ( short ,q:))-
of
Similarly, we can get the future long- term motion dynamics o long with the memory bank by replacing
S, with L, Finally, we concatenate the initial latent feature B ( of ophort and o Long as the input to the LSTM to predict the latent features (0)
141447 of future un-observed video frames.
3.1.3 Motion feature decoding
Once we get the predicted future latent feature ( I for future frames + 1: T, the pose parameters 11+19TT can be obtained by a decoding process. Similarly, we define our motion feature decoding process based on the query-read process. The main difference is that the employed key matrices Kehort,dec WdeeS, and value matrices Vshort,dec Wdees, S are computed based on
the complete motion S, (A rather than partial motion S/, S E R x72) x in the memory bank, as the full motion features are available in the decoding stage. The query vector chort,dec read from the
formulated as q = W (t-Ar,)) Thus, short-term motion dynamics of memory bank can be formulated as:
short,dec V-short,dee softmar((K), dhert,deey? dedeeeeooyoa qda)).
(5)
The long-term motion dynamics 0, Jong.doc can be obtained by replacing S; with L. Then the predicted latent features (,(d) Of generating 3D pose parameters 0(+144 (0.+1, 0.42...... .... ....).
short,dec and o, Long.dec are concatenated together the input of the decoder,
3.2 Model training
Our objective in future motion prediction is to learn the model parameters such that the gap between the predicted motion and the ground truth motion is minimized. To achieve this, we define the prediction loss Cpred as a series of MSE losses formulated as follows:
Epred = Lparameter + Liatent + Ljoints
(6)
T
Cparameter
Σ
0,
0 l
+ ||
-B1l3- -
+ III 11.
1=2
T
Liatent
Σ
Bi00
+
|B(0) -B(33// + |B!.)) -B.00)
(8)
2
12
1=2
T Σ
x, x.I1 11v, 2,) 11.
Djoint
Here, Cporameters Ltatent are the MSE losses of the body parameters 0, G, II and their corresponding latent feature ( B(9), B(I),, respectively. Cjosnt the MSE loss for 3D keypoints and 2D keypoints. V, is the visibility indicator over ground truth 2D keypoints.
Considering the importance of prior knowledge, we follow [42 15] to employ a prior loss Epriors which is commonly used constraint to ensure the reasonability of the predicted human body accord- ing to human prior. It includes human pose reasonability prior loss [15] 16], shape regularization prior loss 3 and shape consistency loss [16].
5

---

In order to keep the semantic consistency between predicted motion and ground tround-truth motion, we further propose an action constraint loss Laction. We train a 3D skeleton-basse based action classification network & [20] on the training videos, and force the action classification results C = ((X) using the predicted 3D joints X to be similar to the results C = (xX) obtained with ground-truth 3D joints X. Here, we calculate a Kullback-eeibler divergence loss between C and C to constraint motion prediction as Lantion
K LCC C). Thus, the complete loss formulation is:
L = Lpred + Lprior + Laction-
(10)
Since directly training the whole Loauder-LSTM--eeee parameters can be unstable, we introduce a two-stage optimization approach. In the first stage, we remove all the LSTMs in the framework and only train the encoders and decoders by minimizing
architecture with randomly initialized
the body parameter loss Cparameter =
0 3
3
- ft) 2 This achieves a
good initialization of the encoder and decoder parameters. In the second stage, we train the whole framework to optimize 3D human motion prediction with loss function L defined in Eq. (IO).
4 Experiments
Datasets. We evaluate the proposed method on Human3.66 6. 14] and Penn Action 44 datasets, which contain human action videos captured under different views with various motions. Human3.61
[4] contains 3.6 million 3D human poses and corresponding images. To construct this set, 11 professional actors are invited to perform 17 activities in indoor laboratory: discussion, smoking, taking photos, etc. This set is well annotated with 3D joint coordinates, joint angles, person bounding boxes, and 3D laser scans of each actor. Following [22,, we use the data of Subjects 1, 6, 7, and 8 as the training set, data of Subject as the validation set, and data of Subjects 9 and 1 as the test set. The Penn Action 44 dataset consists of 2326 in-the-wild thil sport videos involving 15 actions such as baseball pitch, bowling, golf swing, etc. Since all the videos are collected in the wild, the ground truth 3D poses are not available in this set. The annotations only contain action class labels, 2D bounding boxes, and human body joints (2l) locations and visibility). We use the methods in 42 to generate the corresponding pseudo 3D annotations and use them to train the model.
Evaluation metries. The same as that in 422, for the Human3.66 6M dataset with 3D human pose annotations, we report the PA-MPJPE (mean error of per joint position after applying Procrustes Alignment 12] in mm. For the Penn Action dataset with 2D pose annotations, we measure the PCK (percentage of correct keypoints 39 at a 0.05. Following 42), we input 15 past frames to the model and predict the motion of the next 30 frames. We start the prediction from every 25th frame for Human3.66 6M dataset and start from every frame for Penn Action dataset. For fair comparison with PHD (42), we also provide the results of our approach using Dynamic Time Warping (DTW), which can reduce the prediction error caused by unreliable velocity prediction. The DTW intends to find an optimal matching between the ground truth and predicted future motion sequence after time-warping so that the similarity between them maximized. However, we would like to point out that using DTW for evaluation is unreasonable, since we have chance of obtaining the ground truth to warp the prediction in real-world applications.
Network architecture. We employ the 3D human reconstruction model in PHD 42 as our 3D human body parameter regressor to extract the decoupled 3D motion representations. It consists of a ResNet50 image encoder, a causal ID temporal convolutional encoder and regressor with iterative optimization. The dimensions of the encoded 3D human body parameters (() B()), (!I)) are 1024, 256, 64, respectively. Both the encoder and decoder are designed as 3 stacked FC-ReI.U-BN blocks. The number of LSTM hidden units for predicting 0, B, II are 1024, 256, 64, respectively. The temporal length At of each memory item is and sampling time interval K of long-term motion is set to 3. The channels of both key vector C3 and value vector C, are kept same and set to 512.
Learning details. We train our predictors using SGD algorithm with a Nesterov momentum of 0.9, where the 3D human body parameter regressor is frozen with the parameters pre-trained in PHD 42) In the first training stage, we train the encoder and decoder with learning rate 0.01 for 6 epochs. In the second stage, we train the whole prediction block in end-to-end manner for 30 epochs. The

---

Table 1: Comparison results with recent approaches for human motion prediction.
Human 3.6M (PA-MPIPE ))
Penn Action (PCK t)
Method
1
10
20
30
1
10
20
30
Oracle
56.9
56.9
56.9
56.9
56.9
81.6
81.6
81.6
81.6
81.6
Nearest Neighbor
90.3
95.1
100.6
108.6
114.2
63.2
61.5
60.6
58.7
57.8
Constant
59.7
65.3
72.8
84.3
90.4
78.3
71.7
64.9
56.2
49.7
With DTW
PHD
42
57.7
59.5
61.1
62.1
65.1
81.2
80.0
79.0
78.2
77.2
Ours
57.3
58.9
60.3
61.4
62.8
81.4
80.6
79.7
79.1
78.5
Nearest Neighbor
90.3
99.8
110.3
124.7
133.3
62.5
57.6
53.7
44.6
41.1
Constant
59.7
71.4
85.9
101.4
102.8
78.3
65.5
54.6
42.3
32.7
Without DTW
PHD
42
57.7
61.2
64.4
67.1
81.1
81.2
77.2
72.4
67.9 60.1
Ours
57.3
59.6
61.7
62.5
75.9
81.4 79.1
76.7
72.8 66.5
learning rate is initialized to 0.01 and decreased 0.001 after 12 epochs. It took about 3 days for Human 3.6M dataset and 20 hours for Penn Action dataset to train our framework using 4x V100 GPUs. For training the 3D skeleton-based action classifier, use the publicly available code of 20 with default parameters.
4.1 Experimental results
We compare our approach with the state-of-the-art approach PHD [42] and two other baselines (Nearest Neighbor and Constant discussed in 442) The Nearest Neighbor baseline takes the input conditioning frames to find the closest sequence in the training set based on the Euclidean distance of 3D joint locations, and regards the motion reconstructed from the following frames the prediction result. The Constant baseline regresses the 3D human body from the last observed frame and regards it as a constant prediction result for future frames. In addition, we also implement an Oracle approach by directly reconstructing 3D human motion using the 3D parameter regressor in our model from the corresponding ground-trui frames, which can be regarded as an upper bound for our prediction.
We report our experimental results on the Human3.61 [4] dataset and the Penn Action 44 dataset in Table 1 For Human3.6M 3.6M set, our method consistently outperforms all the competitors by considerable margin for both five-frame prediction (0.6mm improvement with DTW and 1.6mm improvement without DTW) and thirty-framm prediction (3.7mm improvement with DTW and 5.6mm improvement without DTW). The results verify that predicting features with action-related prior knowledge can achieve better prediction results. The performance improvement gets larger as the length of prediction increases. This is as expected, as the developed action memory bank and action loss constraint can be used strong guidance for motion prediction, which effectively narrow the prediction space. Thus, the prediction difficulty could be greatly reduced, especially for the prediction of long sequences. Our results on the Penn Action 44 dataset show that our method also achieves state-of-the-art performances for human motion prediction. Specifically, our method achieves 80.6% for five-frame prediction and 78.5% for thirty-frame prediction with DTW. Without DTW, our method can perform 79. 1 for five-frame prediction and 66.5% 5% for the thirty-frame prediction, which are 1.9% and 6.4% higher than the results reported in PHD 42 The promising results show that the action context provide rich information for the 3D motion prediction. We also note that the performance improvement obtained by our method using DTW is smaller than that PHD. This is because that the developed action memory bank with both short-term and long-term dynamics has already captured motion velocity information and better resolved the velocity ambiguity.
In Figure we present some visualization results for the predicted 3D human mesh. For the first example, we can observe that our predictions align much better to the ground truth than predictions of PHD 42 It indicates that the developed action memory bank can help predict reasonable motion velocity for the ongoing action sequences. For the second example, the poses predicted by our method more accurate, which demonstrates the proposed method can learn the motion variation better.
7

---

Imput
L-5
I=10
I-15
L=20
t=25
t-30
t=35
Ground
Frame
PHID
Ground Truth Prame
Aaaala Make
PHD
Ours
Figure 4: Visualization of 3D human motion prediction. From left right: the observed input video and motion predicted at different time steps. We provide samples of baseball pitch and bowling. For each sample, the top row contains the ground-truth frames. The results obtained by PHD 42 and our approach are presented in the mid and bottom row, respectively.
Figure 5: Each row visualizes an item in our action memory bank for the baseball pitch class.
provide some visualization results of the motion dynamics in memory bank. As shown, our memory bank contains some representative motion dynamics for baseball pitch, which could provide strong prior for motion prediction.
In Figure 5
4.2 Ablation studies
In this section, we conduct extensive ablation experiments for future human motion prediction on the Human3.6X, 6M dataset [14] to study the influence of each component in framework. We report the results without DTW, which is more reasonable than the model with DTW in real-world applications.
Influence of action context for prediction. We provide an in-depth analysis the proposed motion prediction method by gradually adding the components and tabulated the results in Table We start from baseline that only contains the encoder, LSTM, and decoder. Then we query
2

---

Table 2:
Evaluation of our action memory bank
Table 5; Evaluation of utilizing the long-term
and other components on Human3. 6M dataset.
motion dynamies on Human3.6M dataset.
Reconstruction error y
Reconstruction error +
Method
10
20
30
Method
5
10
20
30
Baseline
63.1
66.5
68.9
83.3
Prediction with Bank 61.2
63.8
65.7
79.6
Only Short
60.1
62.4
63.7
77.3
+ Decoding with Bank 59.6
61.9
63.4
77.1
K 2
59.4
61.9
63.1
76.8
Action Constraint
59.6
61.7
62.5
75.9
K 3
59.6
61.7
62.5
75.9
Table 3: Evaluation of action context modeling Human3.6M dataset.
K 4
60.0
62.1
63.0
76.4
K 5 60.1
62.2
63.5
77.0
Reconstruction error 4
Table Influence of the predicted action label on Human3. 3.6M dataset.
Method
5
10
30
Baseline
63.1
66.5
68.9
83.3
Action Label
61.8
64.9
66.5
80.2
Reconstruction error
Action Memory Bank
59.6
61.7
62.5
75.9
Accuracy
30
Table 4: Evaluation of our action-specific mem- ory bank on Human3.6M dataset
Ground-truth (100%)
59.6
61.4
61.9
74.1
Classifier (87%)
59.6
61.7
62.5
75.9
Reconstruction error +
80%
59.9
62.0
62.8
76.3
Method
5
10
20
30
75%
60.1
62.2
63.4
76.9
Baseline
63.1
66.5
68.9
83.3
70%
60.7
62.6
64.1
78.2
65%
61.1
63.4
65.2
79.7
Action-agnostic Bank 61.4
64.6
66.3
80.7
Action-specific Bank 59.6
61.9
63.4
77.1
60%
61.8 64.2 67.0
81.5
and read motion dynamics from the memory bank and treat as an additional input to the L.TTM (termed as +Prediction with Bank). We find that it can bring 3.7mm improvement for the prediction of 30-frame prediction. Further, we use the motion dynamics captured in the memory bank to guide our decoding process (termed as +Decoding with Bank). The performance is improved by more than 1.6mm. Finally, we add the action constraint loss Laction to capture more global action context, which measures the consistency of the predicted motion sequence from global perspective. We can observe more improvement in the prediction, especially for the prediction of longer frames (20 and 30 frames). The results demonstrate that the action context captured in the action memory bank and action constraint loss can serve as strong guidance for motion prediction. To further demonstrate the importance of modeling action context and the effectiveness of the proposed memory bank, implement another simple one-hot baseline which models action context by taking the action label (i.e., one-hot vector) indicating the action class additional input to the motion predictor. As shown in Table 3 the one-hot baseline can also reduce the reconstruction error, indicating that injecting action context helps motion prediction. However, this one-hot baseline much less accurate than our proposed approach, which further demonstrates the effectiveness of our proposed memory bank.
aation-agnostic vs. action-specific memory bank. In our action memory bank, we perform clustering for each action class separately (i.e., our memory bank is constructed in action-specific manner). We can also construct an action-agnostic memory bank by clustering over all the dynamic samples without considering the action labels. The experimental results are presented in Table For a fair comparison, we report our approach without the action constraint. As shown, our approach with the aetioonagnossic bank already bring consistent improvement over the baseline. And our proposed action-specific memory bank further promotes the performance, which demonstrates the effectiveness of exploring the action-specific context for future motion prediction.
Benefits of long-term dynamics in memory bank. In our memory bank, short-term dynamic memory item and long-term dynamic memory item (with a sampling interval K) capture motion dynamics at different temporal scales. Here, we study the influence of K as shown in Table As can be seen, compared with the setting that only uses short-term motion dynamics, adding long-term motion dynamics can always improve the prediction performance, which indicates that the long-term

---

motion dynamic complements well with the short-term dynamics. The combination of them can provide more useful information to guide future motion prediction. Setting K achieves the best prediction performance among all the cases.
Impact of the predicted action label. In the prediction process, we employ an action classifier to obtain the action label of the sequence, so that the predictor can query and read relevant dynamic information from the memory bank of the corresponding action. However, the action label can be wrongly recognized in some hard cases. To study the influence of the predicted action labels, we conduct experiments by replacing some low confidence predictions (but correctly predicted) with wrong labels. Our classifier can achieve an accuracy of 87% and we gradually reduce the accuracy to 80%, 75%, 70%, 65% and 60%. The detailed results are presented in Table 6 The motion prediction performance will increase when ground truth labels are provided. Meanwhile, the performance decreases the recognition accuracy goes down. is worth noting that when the accuracy is between 75% and 100%, the prediction performance relatively stable with a difference less than Imm in terms of 30-frame prediction, which demonstrates the robustness of the proposed method.
Conclusion
In this work, we propose to exploit the representative motion dynamics of each action class for 3D human motion prediction. Specifically, we construct an action-specifer memory bank to store the representative motion dynamics. Based on the constructed memory bank, we formulate a query-read process to read motion dynamics from the memory bank guiding motion prediction. An action constraint loss is formulated to enhance the semantic consistency of the predicted 3D motion. Extensive experiments on two benchmarks demonstrate the effectiveness of using action-specific motion dynamies to guide motion prediction. For social impact, we believe that our model can be used to predict human motion for autonomous driving, human- computer interaction, and augmented/virtual reality applications. We also acknowledge that there exists potential negative social impact such as generating malicious fake videos conditioned on the predicted human motion with GANs.
Limitations and Future Work. Our proposed method faces three main limitations: 1) our model relies heavily on the action classification, 2) we develop models to predict the motion features out- putted by fixed 3D parameter regressor without optimizing it, 3) we predict future 3D human motion for each appeared person independently without considering its interactions with the environment. In the future, we would consider learning the recovery and prediction of human shape and motion jointly in an end-to-end manner, as well as modeling human- caatnnhuman/hhhe human/human-obhee objjcct interactions as in [3 Also, we intend to develop a more intricate design for the memory bank such as action hierarchy, temporal pyramid and part-level body motion.
Acknowledgments and Disclosure of Funding
This work was supported partially by the NSFC (U1911401, U1811461, 62076260, 61772570), Guangdong Natural Science Funds Project (2020B1515 1510120018, (188030312002) Guangdong NSF for Distinguished Young Scholar (2018803330605)), Guangzhou Research Project (201902010037), Research Projects of Zhejiang Lab (2019KDOAB03), and the Key-Area Research and Development Program of Guangzhou (202007030004).
References
Anurag Amab, Carl Doersch, and Andrew Zisserman. Exploiting temporal context for 3d human pose estimation in the wild. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 3395-3404, 2019.
[2] Neeraj Battan, Yudhik Agrawal, Sai Soorya Rao, Aman Goel, and Avinash Sharna. Glocalnet: Class-awart long-tern term human motion synthesis. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 879-888, 2021.
[3] Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J Black. Keep it smpl: Automatic estimation of 3d human pose and shape from a single image. In European conference computer vision, pages 561-578. Springer, 2016.
10

---

[4] Matthew Brand and Aaron Hertzmann. Style machines. In Proceedings of the annual conference on Computer graphics and interactive techniques, pages 183-192, 2000.
[5] Christoph Bregler. Learning and recognizing human dynamics in video sequences. In Proceedings of IEEE Computer Society Conference on Computer Vision and Puttern Recognition, pages 568-574, 1997.
[6] Judith Butepage, Michael J Black, Danica Kragic, and Hedvig Kjellstrom. Deep representation learning for human motion prediction and classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 15586166. 2017.
[7] Yujun Cai, Lin Huang. Yiwei Wang. Tat-Jen Cham, Jianfei Cai, Junsong Yuan, Jun Liu, Xu Yang. Yiheng Zhu, Xiaohui Shen, et al. Learning progressive joint propagation for human motion prediction. European Conference on Computer Vision, pages 226-242. 2020.
[8] Junyoung Chung. Caglar Gulcchre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint 141.:514 2014.
[9] Enric Corona, Albert Pumarola, Guillem Alenya, and Francesc Moreno-Noguer. Context-aware human mo- tion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2020.
[10]
Rishabh Dabral, Anurag Mundhada, Uday Kusupati, Safeer Afaque, and Arjun Jain. Structure-aware and temporally coherent 3d human pose estimation. arXiv preprint 17Xiv:/711..02500 3(4):6, 2017.
[11] Carl Doersch and Andrew Zisserman. Sim2real transfer learning for 3d human pose estimation: motion to the rescue. arXiv preprint arXiv:1907.02-4: 2019.
[12] John C Gower. Generalized procrustes analysis. Psychometrika, 40(1):33-51, 1975.
[13] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 98)):133 17800077-7780. 1997.
[14] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. JEEE transactions on pattern analysis and machine intelligence, 36(7):1325-133
2013.
[15] Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1122-7131. 2018.
[16] Angjoo Kanazawa, Jason Y Zhang, Panna Felsen, and Jitendra Malik. Learning 3d human dynamics from video. In Proceedings of the IEEE/CVP Conference Computer Vision and Pattern Recognition, pages 5614-5623, 2019.
[17] Muhammed Kocabas, Nikos Athanasiou, and Michael J Black. Vibe: Video inference for human body pose and shape estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 5253-5263, 2020.
[18] Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit lyyer, James Bradbury, Ishaan Guirajani, Victor Zhong, Romain Paulus, and Richard Socher. Ask me anything: Dynamic memory networks for natural language processing. In International conference on machine learning, pages 1378-1387. PMLR, 2016.
[19] Zihang Lai, Erika La, and Weidi Xie. Mast: A memory- augmente self-supervised tracker. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 6479-6488, 2020.
[20] Ziyu Liu, Hongwen Zhang, Zhenghao Chen, Zhiyong Wang, and Wanli Ouyang. Disentangling and unifying graph convolutions for skeleton-based action recognition. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 143-152, 2020.
[21] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. ACM fransactions graphics (TOG), 34(6):1-16, 2015.
[22] Diogo Luvizon, David Picard, and Hedi Tabia. Multi-task deep learning for real-time 3d human pose estimation and action recognition. IEEE transactions pattern analysis and machine intelligence, 2020.
[23] Julieta Martinez, Rayat Hossain, Javier Romero, and James J Little. A simple yet effective bascline for 3d human pose estimation. In Proceedings of the IELE International Conference on Computer Vision, pages 2640-2649, 2017.
[24] Dushyant Mehta, Oleksandr Sotnychenko, Franziska Mueller, Weipeng Xu, Mohamed Elgharib, Pascal Fua, Hans-Peter Seidel, Helge Rhodin, Gerard Pons-Moll, and Christian Theobalt. Xnect: Real-time multi-person 3d human pose estimation with single rgb camera. arXiv preprint urXiv:1907.0082 2019.
[25] Dushyant Mehta, Srinath Sridhar, Oleksandr Sotnychenko, Helge Rhodin, Mohammad Shafiei, Hans-Peter Seidel, Weipeng Xu, Dan Casas, and Christian Theobalt. Vnect: Real-time 3d human pose estimation with a single rgb camera. ACM Transactions on Gruphics (TOG), 36(4):1-14, 2017.
[26] Alexander Miller. Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. Key-value memory networks for directly reading documents. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1400-1409, 2016.
[27] Scoung Wug Oh, Joon- Young Lee, Ning Xu, and Seon Joo Kim. Video object segmentation using space- time memory networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9226 9235, 2019.
11

---

[28] Dario Pavilo, Christoph Feichtenhofer, David Grangier, and Michael Auli. 3d human pose estimation in video with temporal convolutions and semi-supervised training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7753-7762, 2019.
[29] Vladimir Pavlovic. James M Rehg. and John MacCormick. Learning switching linear models of human motion. In Proceedings of the 13th International Conference on Neural Information Processing Systems, pages 942-948. 2000.
[30] Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. End-to-end memory networks. In Proceedings of the International Conference Neural Information Processing Systems, pages 2440-2444 -4408 2015.
[31] Hsiao- Yu Fish Tung, Hsiao-Wei Tung, Ersin Yumer, and Katerina Fragkiadaki. Self-supervised learning of motion capture. In Proceedings of the 31 International Conference Neural Information Processing Systems, pages 5242-5230 5252, 2017.
[32] Raquel Urtasun, David J Fleet, Andreas Geiger, Jovan Popovié, Trevor Darrell, and Neil D Lawrence. Topologicallly-costriinn
latent variable models. In Proceedings of the 25th international conference on Machine learning, pages 1080-1087, 2008
[33] Jack M Wang, David J Flect, and Aaron Hertzmann. Gaussian process dynamical models for human motion. IEEE transactions pattern analysis and machine intelligence, 30(2):283-25) 2007. [34] Jason Weston, Sumit Chopra, and Antoine Bordes, Memory networks. arXiv preprint /410v 2014.
[35] Chao-Yuue Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming Hc, Philipp Krahenbuhl, and Ross Girshick. Long-term feature banks for detailed video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 284-293, 2019.
[36]
Caiming Xiong, Stephen Merity, and Richard Socher. Dynamic memory networks for visual and textual question answering. In International conference on machine learning, pages 2397-2406, 2016.
[37] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pages 2048-2057. PMLR, 2015.
[38] Xu Yang, Kaihua Tang, Hanwang Zhang, and Jianfei Cai. Auto-encoding scene graphs for image captioning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10685-10694, 2019.
[39] Yi Yang and Deva Ramanan. Articulated human detection with flexible mixtures of parts. IEEE transactions
on pattern analysis and machine intelligence, 35(12))2878-288 22990, 2012.
[40] Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher Pal, Hugo Larochelle, and Aaron Courville. Describing vidcos by exploiting temporal structure. In Proceedings of the IEEE international
conference on computer vision, pages 4507-4515, 2015.
[41] Ye Yuan and Kris Kitani. Dlow: Diversifying latent flows for diverse human motion prediction. In European Conference on Computer Vision, pages 346-364. Springer, 2020.
[42] Jason Y Zhang, Panna Felsen, Angjoo Kanazawa, and Jitendra Malik. Predicting 3d human dynamics from vidco. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7114-7123, 2019.
[43] Jason Y Zhang, Sam Pepose, Hanbyul Joo, Deva Ramanan, Jitendra Malik, and Angjoo Kanazawa. Perceiving 3d buman-object spatial arrangements from a single image in the wild. In European Conference Computer Vision, pages 34-51, 2020.
[44] Weiyu Zhang, Menglong Zhu, and Konstantinos G Derpanis. From actemes to action: A strongly- supervised representation for detailed action understanding. In Proceedings of the IEEE International
Conference on Computer Vision, pages 2248-2255, 2013.
[45] Zhou, Zimo Li. Shuangjiu Xiao, Chong He. Zeng Huang, and Hao Li. Auto-conditioned recurrent networks for extended complex human motion synthesis. In International Conference on Learning Representations, 2018.
[46] Xizhou Zhu, Yujie Wang, Jifeng Dai, Lu Yuan, and Yichen Wei. Flow-guided feature aggregation for video object detection, In Proceedings of the IEEE International Conference on Computer Vision, pages
408 417, 2017.
12


Reviews:
[{"review_id": "WBjzj2VI4wH", "reviewer": "Reviewer_HR9t", "summary": "The authors of the paper proposed to exploit the representative motion dynamics of each action class for 3D human motion prediction. Specifically, they constructed an action-specific memory bank to store representative motion dynamics for each action category, with a query-read process to retrieve relevant motion dynamics from the memory bank, which are consistent with the action in the observed video frames and are used as a prior knowledge to guide motion prediction. The experiment results demonstrated that the proposed approach achieved and exceeded state-of-the-art performance. The ablation analysis demonstrated that each component's contribution to the performance improvements, especially the contribution by the action context captured in the memory bank and the action constraint loss to the motion prediction performance.  ", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The framework proposed by the authors of this paper to capture action context and categories as motion dynamics in the memory bank, to guide the motion prediction, was new and the results demonstrated performance improvement. The paper was well organized. The analysis of the experiments were thorough. The new framework in this paper was based on the encoder-LSTM-decoder architecture, and recently there have been other models being proposed to overcome some disadvantages of this architecture, such as the transformer based methods and etc. in a few recent publications. It is going to be better if the authors can compare their method with these new methods. Also, it is better if the authors could include thoughts on how to improve the performance of long term motion prediction. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is not any significant concerns on negative societal impact.  ", "review_text": "", "extra_scores": {"ethical_concerns": "There is not any significant concerns on ethical issues. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 "}, "extra_text": {}}, {"review_id": "LcVGauwUYen", "reviewer": "Reviewer_3rjL", "summary": "The paper introduces an external memory component to the baseline method PHD on the task of human action prediction. This external memory component is designed so that it clusters common actions across videos conditioned on the action class. A query-read mechanism is designed to train and retrieve similar actions using attention. This memory component is then used to provide information to the LSTM that predicts future parameters pertaining to action conditioned on the past. A combined loss of different components pertaining to the training of the different parameters is used. This method achieves promising results, and the action memory component is able to cluster relevant actions. An ablation study is also provided to check the contribution of the action memory component.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is very clear in its content. There seems to be an original contribution on top of the baseline method PHD, that of a memory component. The relevant related work is detailed, both in the human action prediction space and in the memory component space.\n\nThe main contribution is the addition of the memory component to the typical human action prediction pipeline. The memory component is well motivated, and its details are clearly outlined. Figure 5 clearly shows the actions clustered in the memory component. An ablation is also provided to check the contribution of the memory component to the overall performance.\n\nThe total loss has relevant components that contribute to performance. Each component is described and its relevance explained.\n\nAlthough the performance gains with the memory component are not too big, it is however important that a memory component can help guide the training of models for this task. This aspect has been explored to some level with reasonable experiments.\n\n---\n\nUpdate : I did not have many objections to begin with. However, I see that the authors have sufficiently responded to the other reviewers' comments. In particular, additional experiments were conducted to improve on ablation studies, which proved the benefit of the proposed method. The performance of the proposed method was not too big in the first place, however this does not diminish the merit of the method, as indicated by other reviewers as well. I am happy to increase my rating after reading the authors' responses to all the reviewers' comments.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "A section is devoted to the limitations of the method, however this is present in the appendix. It might be prudent to mention some of the limitations in the main text itself.\n\nThe computational resources used for the experiments has been clearly detailed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "GmI_lbTYW0O", "reviewer": "Reviewer_7fUc", "summary": "This paper proposes a system to use predicted action class and action-specific memory to guide human motion prediction. It constructs an action memory bank that consists of observed short-term and long-term human dynamics that can be later queried and used to predict future human motion. The proposed method achieves state-of-the-art human motion forecasting on the H3.6M and PennAction datasets using only videos.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n- The insight that action classes contain crucial information that can help further motion/video generation is well-motivated and intuitive. It is clear that action class could play a central role in estimating and forecasting human motion from videos.\n- The idea of decomposing action dynamics into short-term and long-term memory banks is interesting and the proposed memory bank design could facilitate future research in this direction.\n\nWeakness:\n\n- The most straightforward way of incorporating action class into motion prediction could be using a simple one-hot vector to indicate the action class to the motion predictor.  This simple baseline seems quite an essential method to compare against and is missing from the ablation studies.\n- The improvement of motion forecasting (number provided in Table 1) is unconvincing against the PHD method. While the proposed method is better in all accounts, providing a few millimeters of motion prediction accuracy boost while using more information (namely, a pre-computed memory bank that uses motion from the same dataset) is not significant enough. In the H3.6M dataset, each actor is instructed to perform the same task and has a very similar motion pattern, so having access to the same motion class (memorized) from different subjects is a non-trivial extra source of information.\n- The proposed motion prediction network, memory bank, and pose regressor seem largely derivative of prior arts and lack novelty.\n\nMinor issues:\n\n- I am not sure why the shape parameter beta needs to be predicted per time-step, since the inherent body shape of the person in question should not change over time. Especially when both PennAction and H3.6M are single-person datasets.\n- What would be more interesting to show is if the action memory bank can be used to mix and generate different future motions from the same starting video. The idea of using action to guide motion prediction is intriguing, but evaluating this task only on motion reconstruction accuracy seems like not fulfilling its full potential.\n- I am curious to see how well the method can perform in clean, non-video-based Mocap datasets (e.g. AMASS[1]). The proposed method purely operates on estimated pose using PHD, while a clean MoCap dataset can be of much larger scale and better motion quality. The video aspect of the proposed work seems a bit less convincing in the light of state-of-the-art methods [2][3], often generative, that can generate future human motion based on past motions.\n\nWriting issues:\n\n- L31: \"in specific\" should be \"specifically\".\n- L34: with different \"velocities\" seem out of space.\n\n[1] Mahmood, Naureen, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Black. 2019. “AMASS: Archive of Motion Capture as Surface Shapes.” Proceedings of the IEEE International Conference on Computer Vision 2019-Octob: 5441–50.\n\n[2] Yuan, Ye, and Kris M. Kitani. 2019. “Diverse Trajectory Forecasting with Determinantal Point Processes.” arXiv, 1–14.\n\n[3] Xu, Jingwei, Huazhe Xu, Bingbing Ni, Xiaokang Yang, Xiaolong Wang, and Trevor Darrell. 2020. “Hierarchical Style-Based Networks for Motion Synthesis.” Lecture Notes in Computer Science 12356 LNCS: 178–94.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No, the author did not address any potential negative social impact of their work. \n\nHuman motion prediction can be used maliciously to generate videos and content based on a short clip of video input. As more and more methods can generate photorealistic human videos based on just human pose input, there is growing significance in analyzing the potential impact of such works. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "0RKM_v-UoRd", "reviewer": "Reviewer_CSBT", "summary": "This paper introduces a new pipeline for human motion prediction with the help of an action dynamics memory bank. Based on the previous work PHD, the stored action motion dynamics from the training data clustering and action classification task can guide the prediction better. In experiments, the proposed method outperforms its counterpart and several baselines on two widely-used benchmarks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "originality: \nThe main framework is following the PHD. The main novelties come from the action dynamics memory and proxy action classification.\n\nquality: \nSound design and good results.\n\nclarity: \nWell written and easy to follow.\n\nsignificance: \nImportant field, good improvements.\n\nDetailed comments\n\nPros:\n+ Sound design of the action dynamics memory, many actions indeed have some common primitives.\n\n+ Good results compared to the PHD.\n\n+ It is interesting to apply the memory module to motion prediction.\n\nCons:\n- Relying heavily on the action classification. A coin always has two sides. This is one of my main concerns.\n\n1, we must classify all actions. Unseen ones are not considered, while the other methods do not have this concern. Moreover, when we should process many actions, scaling the memory bank would be tricky.\n\n2, in use, how to decide class-i? One-hot vector via thresholding or soft probability? It looks that the authors used the latter one. It should be clarified.\n\n3, a person can perform many different actions simultaneously. How to handle this very common situation? For example, eating and talking. \n\n4, did not consider the action hierarchy. In clustering, should we consider the inclusion relations between actions? It looks very vital to me, e.g., the atomic dynamics of bowing and holding-sth. Though we can ignore this and directly see that all actions are mutually exclusive, then the rationality of motion prediction seems robust for different actions. In addition, many actions usually share similar atomic motion dynamics. Why not consider this phenomenon in the memory module design?\n\n5, the atomic dynamics claim is the critical insight to support the whole work, but just several visualizations are shown. I suggest the authors provide more analysis to prove the atomic and common properties of the found dynamics in the memory module. And how the query works in some ideal situations, can the dynamics in memory well match the different cases.\n\n6, the possible bias from the clustering. The long-tailed distribution is common in action datasets. How this affects the clustering and representative dynamics finding?\n\n7, usually, many part-level actions are the essential elements in actions instead of the whole-body motions. Can finer-grained dynamics further improve the prediction?\n\n- The proposed method is based on the PHD. The novel parts are the memory module and action classification task. Though I think this would not hurt much, more discussions about the difference between the two methods are essential.\n\n- Using the pre-trained regressor from PHD. Is this fair for the comparison between the two methods? How about training from scratch? Would the improvement be marginal? As the main competitor is the PHD, this would be very important. BTW, the author has honestly mentioned this in the limitation discussion.\n\n- The memory only works for the pose prediction. The pose and shape are highly correlated. Can we omit this or leave it to the shared backbone or SMPL? An ablation study should be supplemented.\n\n- v_t: how to get the visibility indicator? Need clarification.\n\n- w/ and w/o DTW, the performance improvements differ significantly. Please analyze this phenomenon.\n\n- Fig 4: maybe some indications like red boxes on the unreasonable predicted human parts would highlight the comparison more clear. The current figure is complicated for me to find the difference and how the proposed method outperforms PHD.\n\nOverall, this is an interesting work to utilize more prior knowledge from motion data to advance motion prediction. But many clarifications and discussions are missing. \nI will decide the final rating according to the response from the authors.\n\n------Post-rereview\nThanks for the response from the authors.\nAfter reading all reviews and responses, my most concerns were addressed.\nI keep the rating and positive to this wors. For more please refer to the below response.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers recommend accepting this paper.
The author response was well received and cleared up some of the minor points about the work.
This paper examines the idea of creating an action-specific memory bank to store motion dynamics different action categories.
The work explores a query-read process to retrieve information from the memory bank. The work provides a good set of experiments to demonstrate the effectiveness of the proposed approach, and asserts that they have achieved state-of-the-art performance on 3D human motion prediction.

The AC recommends accepting this paper.
Decision: Accept (Poster)