Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper introduces an external memory component to the baseline method PHD on the task of human action prediction. This external memory component is designed so that it clusters common actions across videos conditioned on the action class. A query-read mechanism is designed to train and retrieve similar actions using attention. This memory component is then used to provide information to the LSTM that predicts future parameters pertaining to action conditioned on the past. A combined loss of different components pertaining to the training of the different parameters is used. This method achieves promising results, and the action memory component is able to cluster relevant actions. An ablation study is also provided to check the contribution of the action memory component.
main_review: The paper is very clear in its content. There seems to be an original contribution on top of the baseline method PHD, that of a memory component. The relevant related work is detailed, both in the human action prediction space and in the memory component space.

The main contribution is the addition of the memory component to the typical human action prediction pipeline. The memory component is well motivated, and its details are clearly outlined. Figure 5 clearly shows the actions clustered in the memory component. An ablation is also provided to check the contribution of the memory component to the overall performance.

The total loss has relevant components that contribute to performance. Each component is described and its relevance explained.

Although the performance gains with the memory component are not too big, it is however important that a memory component can help guide the training of models for this task. This aspect has been explored to some level with reasonable experiments.

---

Update : I did not have many objections to begin with. However, I see that the authors have sufficiently responded to the other reviewers' comments. In particular, additional experiments were conducted to improve on ablation studies, which proved the benefit of the proposed method. The performance of the proposed method was not too big in the first place, however this does not diminish the merit of the method, as indicated by other reviewers as well. I am happy to increase my rating after reading the authors' responses to all the reviewers' comments.
limitations_and_societal_impact: A section is devoted to the limitations of the method, however this is present in the appendix. It might be prudent to mention some of the limitations in the main text itself.

The computational resources used for the experiments has been clearly detailed.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors of the paper proposed to exploit the representative motion dynamics of each action class for 3D human motion prediction. Specifically, they constructed an action-specific memory bank to store representative motion dynamics for each action category, with a query-read process to retrieve relevant motion dynamics from the memory bank, which are consistent with the action in the observed video frames and are used as a prior knowledge to guide motion prediction. The experiment results demonstrated that the proposed approach achieved and exceeded state-of-the-art performance. The ablation analysis demonstrated that each component's contribution to the performance improvements, especially the contribution by the action context captured in the memory bank and the action constraint loss to the motion prediction performance.  
main_review: The framework proposed by the authors of this paper to capture action context and categories as motion dynamics in the memory bank, to guide the motion prediction, was new and the results demonstrated performance improvement. The paper was well organized. The analysis of the experiments were thorough. The new framework in this paper was based on the encoder-LSTM-decoder architecture, and recently there have been other models being proposed to overcome some disadvantages of this architecture, such as the transformer based methods and etc. in a few recent publications. It is going to be better if the authors can compare their method with these new methods. Also, it is better if the authors could include thoughts on how to improve the performance of long term motion prediction. 
limitations_and_societal_impact: There is not any significant concerns on negative societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a system to use predicted action class and action-specific memory to guide human motion prediction. It constructs an action memory bank that consists of observed short-term and long-term human dynamics that can be later queried and used to predict future human motion. The proposed method achieves state-of-the-art human motion forecasting on the H3.6M and PennAction datasets using only videos.
main_review: Strength:

- The insight that action classes contain crucial information that can help further motion/video generation is well-motivated and intuitive. It is clear that action class could play a central role in estimating and forecasting human motion from videos.
- The idea of decomposing action dynamics into short-term and long-term memory banks is interesting and the proposed memory bank design could facilitate future research in this direction.

Weakness:

- The most straightforward way of incorporating action class into motion prediction could be using a simple one-hot vector to indicate the action class to the motion predictor.  This simple baseline seems quite an essential method to compare against and is missing from the ablation studies.
- The improvement of motion forecasting (number provided in Table 1) is unconvincing against the PHD method. While the proposed method is better in all accounts, providing a few millimeters of motion prediction accuracy boost while using more information (namely, a pre-computed memory bank that uses motion from the same dataset) is not significant enough. In the H3.6M dataset, each actor is instructed to perform the same task and has a very similar motion pattern, so having access to the same motion class (memorized) from different subjects is a non-trivial extra source of information.
- The proposed motion prediction network, memory bank, and pose regressor seem largely derivative of prior arts and lack novelty.

Minor issues:

- I am not sure why the shape parameter beta needs to be predicted per time-step, since the inherent body shape of the person in question should not change over time. Especially when both PennAction and H3.6M are single-person datasets.
- What would be more interesting to show is if the action memory bank can be used to mix and generate different future motions from the same starting video. The idea of using action to guide motion prediction is intriguing, but evaluating this task only on motion reconstruction accuracy seems like not fulfilling its full potential.
- I am curious to see how well the method can perform in clean, non-video-based Mocap datasets (e.g. AMASS[1]). The proposed method purely operates on estimated pose using PHD, while a clean MoCap dataset can be of much larger scale and better motion quality. The video aspect of the proposed work seems a bit less convincing in the light of state-of-the-art methods [2][3], often generative, that can generate future human motion based on past motions.

Writing issues:

- L31: "in specific" should be "specifically".
- L34: with different "velocities" seem out of space.

[1] Mahmood, Naureen, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Black. 2019. “AMASS: Archive of Motion Capture as Surface Shapes.” Proceedings of the IEEE International Conference on Computer Vision 2019-Octob: 5441–50.

[2] Yuan, Ye, and Kris M. Kitani. 2019. “Diverse Trajectory Forecasting with Determinantal Point Processes.” arXiv, 1–14.

[3] Xu, Jingwei, Huazhe Xu, Bingbing Ni, Xiaokang Yang, Xiaolong Wang, and Trevor Darrell. 2020. “Hierarchical Style-Based Networks for Motion Synthesis.” Lecture Notes in Computer Science 12356 LNCS: 178–94.
limitations_and_societal_impact: No, the author did not address any potential negative social impact of their work. 

Human motion prediction can be used maliciously to generate videos and content based on a short clip of video input. As more and more methods can generate photorealistic human videos based on just human pose input, there is growing significance in analyzing the potential impact of such works.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces a new pipeline for human motion prediction with the help of an action dynamics memory bank. Based on the previous work PHD, the stored action motion dynamics from the training data clustering and action classification task can guide the prediction better. In experiments, the proposed method outperforms its counterpart and several baselines on two widely-used benchmarks.
main_review: originality: 
The main framework is following the PHD. The main novelties come from the action dynamics memory and proxy action classification.

quality: 
Sound design and good results.

clarity: 
Well written and easy to follow.

significance: 
Important field, good improvements.

Detailed comments

Pros:
+ Sound design of the action dynamics memory, many actions indeed have some common primitives.

+ Good results compared to the PHD.

+ It is interesting to apply the memory module to motion prediction.

Cons:
- Relying heavily on the action classification. A coin always has two sides. This is one of my main concerns.

1, we must classify all actions. Unseen ones are not considered, while the other methods do not have this concern. Moreover, when we should process many actions, scaling the memory bank would be tricky.

2, in use, how to decide class-i? One-hot vector via thresholding or soft probability? It looks that the authors used the latter one. It should be clarified.

3, a person can perform many different actions simultaneously. How to handle this very common situation? For example, eating and talking. 

4, did not consider the action hierarchy. In clustering, should we consider the inclusion relations between actions? It looks very vital to me, e.g., the atomic dynamics of bowing and holding-sth. Though we can ignore this and directly see that all actions are mutually exclusive, then the rationality of motion prediction seems robust for different actions. In addition, many actions usually share similar atomic motion dynamics. Why not consider this phenomenon in the memory module design?

5, the atomic dynamics claim is the critical insight to support the whole work, but just several visualizations are shown. I suggest the authors provide more analysis to prove the atomic and common properties of the found dynamics in the memory module. And how the query works in some ideal situations, can the dynamics in memory well match the different cases.

6, the possible bias from the clustering. The long-tailed distribution is common in action datasets. How this affects the clustering and representative dynamics finding?

7, usually, many part-level actions are the essential elements in actions instead of the whole-body motions. Can finer-grained dynamics further improve the prediction?

- The proposed method is based on the PHD. The novel parts are the memory module and action classification task. Though I think this would not hurt much, more discussions about the difference between the two methods are essential.

- Using the pre-trained regressor from PHD. Is this fair for the comparison between the two methods? How about training from scratch? Would the improvement be marginal? As the main competitor is the PHD, this would be very important. BTW, the author has honestly mentioned this in the limitation discussion.

- The memory only works for the pose prediction. The pose and shape are highly correlated. Can we omit this or leave it to the shared backbone or SMPL? An ablation study should be supplemented.

- v_t: how to get the visibility indicator? Need clarification.

- w/ and w/o DTW, the performance improvements differ significantly. Please analyze this phenomenon.

- Fig 4: maybe some indications like red boxes on the unreasonable predicted human parts would highlight the comparison more clear. The current figure is complicated for me to find the difference and how the proposed method outperforms PHD.

Overall, this is an interesting work to utilize more prior knowledge from motion data to advance motion prediction. But many clarifications and discussions are missing. 
I will decide the final rating according to the response from the authors.

------Post-rereview
Thanks for the response from the authors.
After reading all reviews and responses, my most concerns were addressed.
I keep the rating and positive to this wors. For more please refer to the below response.
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: All reviewers recommend accepting this paper.
The author response was well received and cleared up some of the minor points about the work.
This paper examines the idea of creating an action-specific memory bank to store motion dynamics different action categories.
The work explores a query-read process to retrieve information from the memory bank. The work provides a good set of experiments to demonstrate the effectiveness of the proposed approach, and asserts that they have achieved state-of-the-art performance on 3D human motion prediction.

The AC recommends accepting this paper.
Decision: Accept (Poster)