Title: Harmonizing Stochasticity and Determinism: Scene-responsive Diverse Human Motion Prediction
Venue: NeurIPS 2024
Accepted: yes
Keywords: Human Motion Prediction

Diverse Motion Prediction

3D Scene-aware

Multimodal
Abstract: Diverse human motion prediction (HMP) is a fundamental application in computer vision that has recently attracted considerable interest. Prior methods primarily focus on the stochastic nature of human motion, while neglecting the specific impact of external environment, leading to the pronounced artifacts in prediction when applied to real-world scenarios. To fill this gap, this work introduces a novel task: predicting diverse human motion within real-world 3D scenes. In contrast to prior works, it requires harmonizing the deterministic constraints imposed by the surrounding 3D scenes with the stochastic aspect of human motion. For this purpose, we propose DiMoP3D, a diverse motion prediction framework with 3D scene awareness, which leverages the 3D point cloud and observed sequence to generate diverse and high-fidelity predictions. DiMoP3D is able to comprehend the 3D scene, and determines the probable target objects and their desired interactive pose based on the historical motion. Then, it plans the obstacle-free trajectory towards these interested objects, and generates diverse and physically-consistent future motions. On top of that, DiMoP3D identifies deterministic factors in the scene and integrates them into the stochastic modeling, making the diverse HMP in realistic scenes become a controllable stochastic generation process. On two real-captured benchmarks, DiMoP3D has demonstrated significant improvements over state-of-the-art methods, showcasing its effectiveness in generating diverse and physically-consistent motion predictions within real-world 3D environments.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper present a method for human motion prediction that take into account the environment (3d point cloud) in which the person moves. The method work with several steps: first from the past motion an area of interest is estimated and several object of interest (bed, chair...) with which the person might interact. after selecting the most likely object the method will estimate an interaction with the object (e.g sitting on the bed or lying on the bed) and plan a trajectory to reach the object. Finally with the trajectory and the estimated interaction the model generate the 3d motion from the end of the past motion to the end of the interaction. Since this is a novel task the author compare to two type of method : motion prediction and environment aware motion synthesis and outperform the state of the art on both tasks.\n\nSTRENGTHS:\nThe paper introduce a new task of environment aware motion prediction where the environment is a static 3d point cloud.\n\nInteresting approach where the models first find the most likely item to be used then create a trajectory and finally generate the future motion based on that trajectory.\n\nThe paper provides extensive and detailed experiments and ablation study on two datasets\n\nThe qualitative results look very good.\n\nDespite tackling a new task the authors compare with sota methods for two related tasks.\n\nThe paper is clear,  well written and well detailed.\n\nWEAKNESSES:\nAt least some results of the scene aware motion synthesis from the appendix should be moved to the main paper. Especially since there is no qualitative comparison with other scene aware method in the main paper.\n\nIn figure 1 it is not clear that diffusion is being used.\n\nA qualitative comparison with bifu would have been interesting since its results are the best after the proposed method.\n\nQUESTIONS:\nVideos of the generated motion would have been appreciated\n\nline 261 introduces => introduced\n\nLIMITATIONS:\nLimitations and impacts are discussed in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-25"}

comments: SUMMARY:
This work studies human motion prediction in 3D scenes. The proposed DiMoP3D leverages context-aware intermodal interpreter, behaviorally-consistent stochastic planner, and self-prompted motion generator to solve the task. The authors conduct experiments on GIMO and CIRCLE to demonstrate a superior performance.

STRENGTHS:
- The paper is well-organized and easy to follow.
- In extensive experiments,  DiMoP3D demonstrates a superior performance over the baselines with analysis.

WEAKNESSES:
- I question the novelty of this work. Novelties in Context Awareness and Autonomous Intention Estimation are overclaimed. They have been studied in previous works but authors fail to provide a discussion:
[29] uses path planning to perform context-aware navigation and GoalNet to estimate the end-point state. 
[98] uses 3D scene point to encode the scene context and gaze to estimate the intention.
Still more work to supplement...
I believe the novelty is very overclaimed.


- The system estimates the motion of a fixed length $\Delta L$. Is it possible to not reach an end state? Taking Fig.2. as a case, how do you guarantee the human is sitting on the chair but not still walking to it after $\Delta L$? It may lead to unnature speed and motions.


- How do you evaluate the diversity and natureness of motions?

QUESTIONS:
See weakness part.

LIMITATIONS:
As discussed in Appendix G, the system is not as efficient as baselines.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel task that incorporates real-world 3D scene information into the existing Human Motion Prediction task. The authors propose a model (DiMoP3D) that, starting from the observed motion sequence and the 3D scene, stochastically predicts the future poses and the interactions with the context; in particular, DiMoP3D infers the individual intention and the human-object interactions. Upon benchmarking on two datasets, GIMO and CIRCLE, the model surpasses all the baselines on most of the considered metrics.\n\nSTRENGTHS:\nThe proposed task lies in the intersection of existing ones while capturing a problem setting that hasn’t been explored yet. Similar tasks either considered only the contact points within the scene but with no stochasticity in the prediction ([1,2,3]) or focused on stochastically generating human motion conditioning on the observed pose sequence and the 3D scene but fixing the goal (e.g., [4]). As such, the discussion is technically sound and original. The proposed model is convincingly designed as the role of its submodules is properly motivated and analyzed both in the method’s section and in the ablation studies. The results on CIRCLE and GIMO confirm the effectiveness of the proposal.\n\n[1] Mao, Wei, Richard I. Hartley, and Mathieu Salzmann. \"Contact-aware human motion forecasting.\" Advances in Neural Information Processing Systems 35 (2022): 7356-7367.\n\n[2] Luca Scofano, Alessio Sampieri, Elisabeth Schiele, Edoardo De Matteis, Laura Leal-Taixé, Fabio Galasso. “Staged Contact-Aware Global Human Motion Forecasting.” BMVC 2023: 589-594\n\n[3]  Xing, Chaoyue, Wei Mao, and Miaomiao Liu. \"Scene-aware Human Motion Forecasting via Mutual Distance Prediction.\" arXiv preprint arXiv:2310.00615 (2023).\n\n[4] Hassan, Mohamed, et al. \"Stochastic scene-aware motion prediction.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n\nWEAKNESSES:\nW1. While it is clear how their proposed task diverges from human motion synthesis in 3D scenes, human-object and scene interaction prediction (e.g., [5]), and social navigation, it is not clear enough how it differs from scene-aware 3D human motion forecasting or synthesis. \n\n[5] Xu, Sirui, et al. \"Interdiff: Generating 3d human-object interactions with physics-informed diffusion.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\nQUESTIONS:\nI acknowledge that Sec. B.1 of the supplementary extends the literature review in Sec. 2 of the main manuscript, though the discussion could benefit from a tighter and more direct comparison with similar tasks in [1,2,3,4].\n\nTypos and minor issues with the writing:\n\nL99: forgot the full stop at the end of the sentence.\n\nL119: “intermodal insights. utilizing…”\n\nL181: “To tackle ress these”\n\nL211: “When \\alpha_T is approaches 0…”\n\nL261: “...is introduces…”\n\nL304: “...by the subject en route…”\n\nLIMITATIONS:\nYes, they included the discussion about limitations and societal impact in Section G of the supplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a task that is scene-aware diverse human motion prediction. To be specific, given a 3D scene and history motion, this task aims to predict diverse future human motions that are consistent with the scene and history motion. This paper also proposed a model, DiMoP3D, to tackle this task. This model includes a Context-Aware Intermodal Interpreter to encode the scene and find the goal object, a  Behaviorally-Consistent Stochastic Planner to generate the end pose and an obstacle-free trajectory, and a diffusion model to generate the motions on the path.\n\nSTRENGTHS:\n1. This paper tackles an interesting task.\n2. The numerical results are good.\n\nWEAKNESSES:\n1. As my understanding, if the author follows the original data split of CIRCLE and GIMO, there are no multiple possible future motions for history motion. How to make sure the model can generate diverse future motions trained on such datasets?\n\n2. Please compare the proposed task with [9, 29]. These two works have already tackled the task of scene-aware stochastic prediction. Why do the authors claim they proposed a new task?\n\n3. The author claims that the predictions must adhere to deterministic constraints, including physical consistency. The proposed method only considers the obstacle-free trajectory, but there is no constraint for the generated poses, e.g., there could be penetration with the ground for the walking sequence. \n\n4. What is the insight and novelty of DiMoP3D to make it different from the previous paper? Most techniques are similar to previous work. The scene encoder is from [71], the prediction of diverse goals is similar to [9], the HOI estimation is similar to [1, 2], the trajectory planning is similar to [77, 82, 29], the motion generator is similar to [87].\n\n5. Some parts are hard to read, e.g., the introduction.\n\n6. For the 3D human motion prediction/generation tasks, I prefer to see a video rather than only some figures for the visualization part.\n\n[1] Zhao, Kaifeng, et al. \"Compositional human-scene interaction synthesis with semantic control.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n\n[2] Hassan, Mohamed, et al. \"Populating 3D scenes by learning human-scene interaction.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\nQUESTIONS:\nThe question is the same as Weaknesses. Overall, the paper seems to propose a valuable task and has good results. Addressing the limitations mentioned above would significantly strengthen the work and I am positive to change the score.\n\nLIMITATIONS:\nThe authors addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-15"}