Harmonizing Stochasticity and Determinism: Scene-responsive Diverse Human Motion Prediction

Authors: zhenyu  lou
qiongjie  cui
tuo  wang
zhenbo  song
luoming  zhang
cheng  cheng
haofan  wang
xu  tang
huaxia  li
hong  zhou
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 10:17:40

Abstract: Diverse human motion prediction (HMP) is a fundamental application in computer vision that has recently attracted considerable interest. Prior methods primarily focus on the stochastic nature of human motion, while neglecting the specific impact of external environment, leading to the pronounced artifacts in prediction when applied to real-world scenarios. To fill this gap, this work introduces a novel task: predicting diverse human motion within real-world 3D scenes. In contrast to prior works, it requires harmonizing the deterministic constraints imposed by the surrounding 3D scenes with the stochastic aspect of human motion. For this purpose, we propose DiMoP3D, a diverse motion prediction framework with 3D scene awareness, which leverages the 3D point cloud and observed sequence to generate diverse and high-fidelity predictions. DiMoP3D is able to comprehend the 3D scene, and determines the probable target objects and their desired interactive pose based on the historical motion. Then, it plans the obstacle-free trajectory towards these interested objects, and generates diverse and physically-consistent future motions. On top of that, DiMoP3D identifies deterministic factors in the scene and integrates them into the stochastic modeling, making the diverse HMP in realistic scenes become a controllable stochastic generation process. On two real-captured benchmarks, DiMoP3D has demonstrated significant improvements over state-of-the-art methods, showcasing its effectiveness in generating diverse and physically-consistent motion predictions within real-world 3D environments.

Main paper: 
 introduction: 
 Human motion prediction (HMP), i.e., forecasting future human poses based on observation, is crucial for applications including autonomous vehicles and human-robot collaboration [15, 20, 39, 44, 54, 60, 66] . Many existing works [1, 22, 40, 49, 76, 89] formulate HMP as a deterministic problem, aiming to generate a single future sequence. However, it fails to capture the inherent stochasticity of human motion, where multiple plausible outcomes can arise from a single observation. Recent research has shifted towards diverse or stochastic HMP, to achieve multiple plausible predictions [3, 6, 30, 59, 72] , which holds the potential in real-world applications and is the focus of our work.Recent advances in diverse HMP primarily focus on stochastic predictions, where a random factor from the latent space conditions the diversity of predictions alongside observed motions [3, 12, 30, 59, 71, 72, 82] . While these methods predict multiple plausible futures from a single past motion, they typically disregard the 3D environment, operating within an idealized, context-free framework. This limitation becomes apparent in real-world applications, where motion must conform to physical and semantic scene constraints [29, 67, 68, 78, 90] , leading to issues like obstacle penetration and unrealistic interactions, as in Figure 1 . This gap underscores the need for a new task that merges diverse HMP within real-world 3D scenes, enhancing both practicality and applicability.Recognizing existing limitations, this work introduces a novel task, making diverse HMP within real-world 3D scenes. Its objective is to break the previous idealized context-free setup towarding a realistic and practical setting, which involves several key challenges: (1) Harmonizing Stochasticity and Determinism: This task necessitates a delicate harmonization between the stochastic nature of human motion and the deterministic constraints from 3D scenes, thereby broadening the scope of traditional HMP; (2) Scene-Motion Intermodal Coordination: It requires analyzing coordination between human motion and scene dynamics to align predictions with contextual elements, which involves identifying human intentions and potential interactive objects; (3) Behaviorally Coherent Physical Consistency: The predictions must adhere to deterministic constraints, including physical consistency (e.g., avoid collision) and behavior coherence (e.g., sitting on a chair, not lying).To tackle these challenges, we introduce DiMoP3D (Diverse Motion Prediction in 3D Scenes), a framework for generating diverse, physically consistent, and plausible human motion predictions in real-world 3D scenes, which comprises three main components: (1) Context-aware Intermodal Interpreter analyzes potential areas of human interest and goals, essentially intentions within a scene. Our method enhances traditional scene understanding by integrating 3D point clouds with observed motions for context-aware intermodal analysis. It first encodes the point cloud, segments object instances, and then pinpoints potential interaction targets, emphasizing intended factors while filtering out less probable ones. This strategy further transforms the task of diverse HMP into a controllable stochastic generation process; (2) Behaviorally-consistent Stochastic Planner then constructs behaviorally consistent action plans, representing stochastic conditional factors. Recognizing that human interaction with specific objects often follows deterministic patterns, we prioritize predicting the final human pose upon reaching each target, and generate obstacle-free trajectories toward it; (3) Self-prompted Motion Generator harmonizes the stochastic nature of human motion with deterministic constraints in a self-prompted manner to produce varied predictions based on the conditional factor. To ensure coherence with this factor, it employs a denoising diffusion model, guiding the motion denoising process toward a deterministic, obstacle-free final state.Our contributions are threefold: (1) We introduce a novel and challenging task of predicting diverse human motions within real-world 3D scenes, advancing beyond the traditional scope of diverse HMP from an idealized context-free setting to a more realistic and practical one. (2) We propose DiMoP3D to tackle this task. It harmonizes the deterministic constraints of 3D scenes with the stochastic nature of human motion, enabling diverse and plausible motion predictions in real-world scenarios. (3) Evaluations on two real-captured benchmarks, GIMO and CIRCLE, show that DiMoP3D significantly outperforms existing SoTA methods, particularly in terms of physical consistency.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: Section 3 tell the method, Section 4 demonstrates results, proving the effectiveness of the method. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
stochastic human motion prediction: 
 Human motion prediction diverges into deterministic and stochastic methods. Deterministic models aim to predict a singular sequence that closely aligns with future movements [18, 19, 40, 49, 76 ], yet often encounter quality degradation over longer time spans (> 1-sec) due to the stochastic nature of human movement. In contrast, stochastic HMP [2, 3, 5, 44, 82] embraces this variability by generating a range of plausible future motions and modeling the distribution of human behaviors. Notably, it also encompasses the most probable future sequence targeted by deterministic models, as a likely scenario within its broader distribution. Such methodologies enhance applications across autonomous driving [7, 33, 43] , patient care [10, 42] , and human flow prediction [32, 36] , by embracing a wider range of potential scenarios and have become a focal point of contemporary research. Dominant approaches include VAEs, GANs, flow networks, and diffusion models [3, 6, 12, 30, 38, 59, 71, 72, 82] . Despite promising progress in modeling stochastic human motions, a critical challenge persists: human motion is not only stochastic but also heavily influenced by the external environment. In real-world settings, human motion is intricately intertwined with surrounding scenes, necessitating that future trajectories comply with the scene's physical constraints (e.g., avoid object penetration). Additionally, predicted motions should be semantically consistent with the expected human-object interactions (e.g., sit for a chair, lie for a bed). Addressing these requirements calls for a sophisticated approach to diverse HMP that balances the stochastic aspect of human motion with the deterministic factors imposed by the surrounding 3D scenes, which is the focus of our work.3D scene understanding is essential in various applications, prompting extensive research on representations like RGB-D maps [9, 52, 61] , scene graphs [77, 79, 84] , 3D voxels [23, 48, 74] , and notably, 3D point clouds [55, 56, 65, 87] . Recognizing the importance of 3D scene information in human motion prediction [14, 21, 26, 35, 80] , our work integrates 3D point clouds as the scene representation due to their direct derivation from sensing technologies.To enhance fidelity of HMP in real-world scenarios, the connection between human actions and scene context has made scene-aware motion generation a major research focus [8, 17, 28, 88] . Early methods [8, 17, 67] relied on object bounding boxes, 2D images, and depth maps, which are insufficient for capturing real 3D environments. Recent advances use 3D point clouds for scene representation [29, 69, 88] : GIMO [88] employs a bidirectional transformer to fuse human motion and scene features, [50] predicts future contact maps, and [46] extracts global and local salient points.However, these methods predict a single sequence [4, 50, 88] , whereas our approach models a distribution of potential outcomes. Some stochastic methods add diversity, but [8] relies on 2D inputs, limiting 3D interactions, and [28] uses predefined objects without inferring targets. In contrast, our model parses the 3D scene through cross-modal, object-specific human interest analysis and predicts scene-aware motions in real 3D scenarios, greatly enhancing adaptability and authenticity.Recent advancements in paired scene-motion data [4, 27, 88] have sparked a new research direction in synthesizing motions in 3D scenes [29, 41, 47, 53, 69, 78] . Specifically, SAMP [28] employs a conditional variational autoencoder (cVAE) to generate one frame per forward pass within a threestage stochastic pipeline, COUCH [86] introduces a human-chair dataset with an auto-regressive, contact-satisfying method, and DN-Synt [68] proposes a hierarchical framework for effective sceneaware human motion synthesis. With the rise of language models, methods utilizing natural language prompts have emerged [16, 69, 75, 85] . HUMANISE [69] proposes an attention-based languageprompted synthesis method. Additionally, diffusion models have shown significant promise [16, 29, 37, 63, 64, 70] . Notably, AffordMotion [70] employs scene affordance as an intermediate representation, achieving state-of-the-art performance in object interaction synthesis.Our novel task of scene-aware diverse HMP is partly inspired by advances in motion synthesis but is tailored for distinctly different applications and challenges. This task is further distinguished by key innovations: (1) Temporal-Dependent Prompting. Contrary to motion synthesis, which often lacks temporal context and follows static user instructions, our approach conditions predictions on the unique prompt of historical human motion, enabling autonomous inference of human intentions.(2) Context-Aware Scene Parsing. Moving beyond traditional scene understanding, our method infers potential movement targets through the integration of context-aware intermodal insights. Utilizing scene determinism to enhance the fidelity of diverse HMP predictions. Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Secon 5 analyzes the efficiency limitation of the proposed DiMoP3D. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
problem setup: 
 Given L historical human posesX 1:L = {x 1 , x 2 , • • • , x L } within 3D scenes represented by point clouds S ∈ R np×6 , our goal is to predict K different scene-consistent future motions { X(i) L:L+∆L } K i=1 .Here, n p denotes the number of points, each described by 3D spatial coordinates and RGB color information. L and ∆L denote the lengths of the observed and predicted sequences, respectively. Each pose is described as the SMPL-X representation x l = (t l , o l , p l ) [88] , where t l ∈ R 3 denotes the global translation, o l ∈ SO(3) denotes the orientation, and p l ∈ R 32 refers to the body pose embedding. We set L = 3-sec and ∆L = 5-sec to achieve a long-term prediction [40, 49] . Then, the task can be formulated as:EQUATIONThe stochastic process θ is sampled jointly from the past motion X 1:L and the scene S, and then utilized to condition the prediction motion XL:L+∆L . We propose DiMoP3D to solve this novel task, which involves the following novelties: Context awareness: unlike traditional diverse HMP methods [12, 59, 71] that focus solely on human motion, our task is more challenging as it requires harmonizing the stochastic nature of human motion with the deterministic constraints of the surrounding 3D scenes. Autonomous intention estimation: different from motion synthesis, our task requires independent intention estimation based on past motion, to prompt future motion prediction.Scene information plays a crucial role in predicting future motion [8, 17, 67] . Despite the stochastic manner of human motion, it is still feasible to deduce human interests and likely goals within a scene. For instance, in Figure 2 , the door behind the person is unlikely to be the target based on their trajectory away from it, while the sofa, coffee table, and distant chair may emerge as potential points of interest. Diverging from traditional scene parsing methods, our approach integrates a scene-motion intermodal coordination to better suppose human intentions in real-world settings.In machine vision, 3D point clouds sourced directly from sensing devices have become fundamental for scene representation [55, 65, 87] and serve as the input for our scene interpreter. Acknowledging the significant influence of past motions and scene context on future human movements, we emphasize the need for a context-aware intermodal analysis that integrates historical motion with scene point clouds to infer potential human intentions. Furthermore, since human movements typically involve interactions with target objects [11, 24, 81] , our approach identifies objects within the scene and computes a human interest score for each, rather than analyzing isolated points. This helps determine specific interactive targets, making the motion prediction process more controllable and enabling diverse prediction by sampling different interactive targets.To achieve this, our scene interpreter employs a UNet-like encoder-decoder architecture [58] , comprising an instance segmenter for object recognition and an interest net for human interest inference. These two modules share the same point cloud encoder for efficiency but use different decoders, as illustrated in the yellow box in Figure 2 . The shared encoder processes and downsamples the point cloud S into a compact feature representation F s ∈ R n ′ p ×c , with n ′ p < < n p (8 ≤ n ′ p ≤ 50 in our cases), and c represents the feature dimension. Subsequently, two decoders are employed to segment objects O, and predict human interests M in the scene, respectively:F s = SceneEncoder(S), O = InstanceSegmenter(F s ), M = InterestNet(F s , X 1:L ). (2)Here, O = {O 1 , O 2 , ..., O no } denotes the set of n o segmented objects, each being a subset of the scene pointcloud (O i ⊆ S). M ∈ R np×1 denotes the per-point interest map, with higher values indicating a greater likelihood of targeting specific scene elements. Once M is obtained, we compute the probability P i of each object O i ∈ O being selected as an interaction target:EQUATIONEQUATIONwhere len(O i ) denotes the number of points in O i , M [p] is the interest value of point p, and ϕ = 0.5 represents the temperature factor that controls randomness in sampling. During inference, we sample the target object O g (g ∈ {1, 2, ..., n o }) based on the probability distribution {P 1 , P 2 , ..., P no }.In cases where there is no human-object interaction or objects are beyond reach within ∆L, no explicit target object exists. To handle this, we include the ground as a potential target, voxelized into smaller patches to improve granularity. Each ground patch, treated as an individual object, has a side length of s = 0.5 meters to balance accuracy and efficiency. The interest scores and interaction probabilities for ground patches are then calculated similarly to other objects, as in Eq.3 and Eq.4.Our scene interpreter aligns observed motions with scene context, filtering out less likely engagement areas and identifying potential targets. We note that this approach makes the diverse HMP controlled by those deterministic elements of the scene, thereby enhancing the physical consistency of predictions. Representing human intention through scene-motion intermodal analysis, the selected target object O g directs the subsequent planning process, as outlined below.Ensuring collision-free and scene-consistent human behavior is crucial but often overlooked, while learning these patterns directly requires impractically large datasets. To tackle these challenges, we employ an action planner to plan obstacle-free trajectories toward the target O g and deterministically predict the interactive human end-pose xend associated with O g . These intermediate predictions serve as stochastic conditional factors, guiding to craft future motions that respect physical constraints and typical human-environment interactions while incorporating motion diversity.To enable effective navigation and collision avoidance, we utilize a scene height map S H ∈ R ns×1 to delineate accessible areas and detect obstacles, inspired by [68, 73] . We first compute obstacle-free trajectories toward the target object O g using a modified A* algorithm (details on generating diverse trajectories are in Appendix B), and then employ a single-layer transformer ψ to predict per-frame human velocity, sampling discretize points from the planned trajectory:EQUATIONHere, τ plan denotes the continuous trajectory, τ plan = { tplan L+1 , tplan L+2 , ..., tplan L+∆L } represents the sampled discretize trajectory points, and t end is the estimated timestamp for the end of the interactive motion. Since the length of this trajectory varies across sequences, the human may not reach the target object exactly at the prediction horizon ∆L = 5-sec. In these cases, if the target object is too distant to reach within ∆L (t end > ∆L), we truncate the trajectory τ plan to fit within ∆L and adjust the target to the nearest ground patch. Conversely, if the target is reached too early (t end < ∆L), we keep the subject relatively static after t end . This adjustment ensures that the planned trajectory aligns with the prediction horizon. We then discretize the continuous trajectory to obtain the per-frame global translation τ plan to guide subsequent motion generation:EQUATIONIn addition to physical constraints, human interactions with specific objects often follow deterministic patterns despite potential action diversity. For instance, "set," and "wipe," are reasonable actions for a table, whereas "sit" and "lie" are not. Traditional diverse HMP methods typically overlook these Human-Object Interaction (HOI) patterns, leading to motion and scene inconsistencies. Differing from these methods, our approach predicts the target object O g in advance, enabling us to predict the interactive HOI end-pose xend before the full-sequence prediction:EQUATIONThis end-pose represents the final state of the prediction, secures appropriate human interaction with the scene. To this end, our planner constructs an obstacle-free trajectory, and the predicted end-pose (along with t end ) as a stochastic conditional factor θ from a scene-motion intermodal perspective:EQUATIONThis factor further prompts the motion generator to predict behaviorally consistent motions.By constructing a stochastic conditional factor θ in advance, DiMoP3D operates as a self-prompted motion generator, which harmonizes the stochastic factor with deterministic motion generation rooted in θ. To generate diverse predictions that closely align with the predicted conditional factor θ, we utilize a motion diffuser, taking advantage of the diffusion model's ability to effectively guide intermediate results [12, 59, 71] . Additionally, to further maintain semantic coherence and physical consistency, we propose a semantic alignment inspector to supervise the denoising process.For simplicity, we denote the sequence at noising step t as X t . Diffusion is modeled as a Markov noising process {X t } T t=0 , with X 0 drawn from the data distribution, and:EQUATIONHere α t ∈ (0, 1). When α T approaches 0, we approximate X T 1:L+∆L ∼ N (0, I), where 0 and I represent the zero matrix and the identity matrix, respectively. To effectively integrate scene and motion features in our predictions, our diffusion model M D employs a transformer decoder to model distribution akin to the reversed diffusion process, leveraging its capacity for cross-modal attention. Instead of predicting noise, we predict the clean sample directly, following [63, 64, 83] :EQUATIONwhere X0 represents the intermediate denoising result at each denoising step.To align the predicted sequence with the observed X 1:L , the planned trajectory τ plan , and the forecasted end-pose xend at time t end , we adjust the corresponding segments after each denoising step. To be sepecific, for each framex0 i = ( t0 i , ō0 i , p0 i ): x0 i =          x 0 i if i ≤ L, ( tplan i , ō0 i , p0 i ) if L < i < L + t end , xL+∆L if i + t end , x0 i if i > L + t end ,for each frame i.(EQUATIONThis modified prediction is then noised back before the next denoising step:EQUATIONwhere γ t represents the posterior variance based on α at step t. Upon completing T denoising steps, the motion generator yields a cohesive sequence X0 , which integrates smoothly with the observed sequence and aligns with the planned trajectory and goals.To enhance the consistency between the predicted motion and the target object, we introduce a semantic alignment inspector leveraging MotionCLIP [62] . It computes a HOI semantic loss via natural language descriptions as follows:EQUATIONHere, MC M otion , MC T ext denote MotionCLIP's motion and text encoders, respectively, with D g signifying the interaction description template related to the class of the sampled target object O g . Acknowledging that HOI predominantly occurs later in motion sequences, our semantic loss formulation weights later motion frames more heavily to accurately capture these interactions. We also notice other related datasets [27, 28, 50] , yet find limitations precluding their use (e.g., jittering, sequence length, absence of human meshes).Baselines. Our DiMoP3D is compared with four contemporary methods: DLow [82] , SmoothDMP [72] , BeLFusion [5] , and BiFU [88] . DLow [82] applies a flow network, SmoothDMP [72] is VAEbased, and BeLFusion [5] is diffusion-based, which achieves SoTA performance in diverse HMP. These three, however, do not focus on scene-aware diverse HMP, setting BiFU [88] , a deterministic scene-aware method, apart as an essential control for our analysis.Metrics. To align with existing literature that evaluates human skeleton metrics, we utilize the SMPL model [45] to convert body parameters X 1:L+∆L into skeletons J 1:L+∆L = {j 1 , j 2 , ..., j L+∆L }, where each j i ∈ R nj ×3 represents a skeleton with n j = 22 joints, following [25] .DiMoP3D is evaluated for diversity, accuracy, and physical consistency in scene-aware predictions.We begin with the well-established pipeline in [82] : Prediction diversity is quantified using the Average Pairwise Distance (APD) by computing the L2 distance across predicted sequences. The Average Displacement Error (ADE) measures the reconstruction accuracy among the whole predicted sequence, while the Final Displacement Error (FDE) measures accuracy of the furthest frame, alongside their multimodal counterparts, MMADE and MMFDE, for diverse HMP scenarios.To measure the physical consistency of the predicted motion within 3D scenes, an additional metric, the Average Cumulated Penetration Depth (ACPD) is introduced, following [78, 83] :EQUATIONwhere j[n] denotes the position of the n-th joint in the skeleton, and SDF(•, S) refers to the signed distance function [51] of the scene point cloud S. For training details, please refer to Appendix A. (5-sec) motion forecasting. Despite their higher scores in diversity (APD), this is attributed to their erratic and unpredictable predictions, disregarding the scene context (detailed in Sec 4.4).DiMoP3D's enhanced performance stems from three key factors: (1) Diversity. The stochastic conditional factor introduces diversity through multiple mechanisms: the intermodal interpreter sets broad motion objectives, the stochastic planner generates a variety of end-poses and trajectories, and the motion generator achieves diverse motion poses. This multi-faceted approach ensures a breadth of plausible actions are considered, enabling DiMoP3D to achieve a considerable APD score. (2) Accuracy. DiMoP3D outperforms every baseline in ADE, FDE, MMADE, and MMFDE for a large margin, even the deterministic BiFU. By estimating future human action based on a scene-motion intermodal analysis, DiMoP3D implicitly infers the subject's intent. This boosts the probability of accurately identifying the subject's genuine intent as the basis for prediction, thereby improving the prediction precision. The combination of accurate intermodal scene interpreting and stochastic planning ensures precise motion prediction for each sequence. (3) Physical consistency. Our motion generator employs a diffusion model, prompted by the predicted stochastic factors. It also ensures motion coherence through priors overwrite at each denoising step. This dual focus on deterministic constraints enables DiMoP3D to achieve superior motion-scene consistency.This superior performance demonstrates the efficacy of our DiMoP3D in predicting diverse human motion in 3D scenes, as further evidenced by subsequent ablation studies and visualizations.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes]Answer: [Yes] Justification: Appendix A tells the training details and the supplemental material provides our source code. Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
ablation studies: 
 In Table 2 , we dissect the impact of excluding four pivotal components from DiMoP3D. First, eliminating InterestNet markedly decreases performance across ADE, FDE, MMADE, and MMFDE (first row). This decline stems from the process of selecting the target object O g from O, which reverts to random sampling without scene-motion crossmodal analysis, impairing DiMoP3D's ability to deduce human intentions. Consequently, the accuracy of predicting real actions diminishes, highlighting the significance of integrating multimodal scene-motion analysis for scene-aware HMP. Addressing the role of stochastic planner, its absence undermines the planning of actions, including the prediction of end-poses by the HOI-Estimator and trajectory planning via the A* Trajectory-Planner. Without these two components, the motion generator struggles to predict end-poses or future trajectories with scene consistency. Notably, omitting the HOI-Estimator results in imprecise interactive end-poses, often causing the subject to intersect with the target object in later frames, as evidenced by increased ACPD and reduced FDE and MMADE (second row). Similarly, excluding the TrajectoryPlanner significantly elevates ACPD (third row), indicating frequent subject penetrations into the scene context while approaching the end-pose. These findings underscore the vital role of coordinated end-pose and trajectory prediction in predicting motion within 3D scenes effectively.Finally, the SemanticInspector enhances the scene-motion alignment through natural language, with its omission resulting in higher ADE and MMADE. Please refer to Appendix C for further ablations.To delve deeper into DiMoP3D, in Figure 3 , we showcase DiMoP3D's predictions across two scenarios, contrasting them with the SoTA baseline BeLFusion [5] .In bedroom scenario, the subject stands still behind the door. BeLFusion's predictions show notable issues with object and wall penetrations. Furthermore, sample-1 and 3 are marked by abrupt, illogical movements, and sample-3 and 4 display glaring scene inconsistencies: sample-3 has the subject sitting on the bare floor, and sample-4 involves the subject reaching for non-existent items. Conversely, DiMoP3D ensures physical consistency, directing each prediction towards a specific movement goal: opening a window, lying on the bed, accessing a cabinet, and sitting on a chair.In the seminar room scene, the subject is moving forward. BeLFusion struggles again, with sample-1, 3, and 4 depicting the subject unrealistically exiting the room, and sample-2 showing an inconsistent motion of picking up a curtain. DiMoP3D, however, delivers high-fidelity predictions, depicting the subject walking through a door, grasping items, sitting, and pulling down curtains, respectively.To emphasize DiMoP3D's predictive diversity, we further visualize various end-poses generated by the HOI-Estimator in Figure 4 . Overall, DiMoP3D consistently delivers diverse, realistic, and physically-consistent motion predictions with clear objectives, benefiting from our conditional factor prediction schema which models human-object interactions and navigates obstacle-free trajectories. In this section, we compare our DiMoP3D with three scene-aware motion synthesis approaches on GIMO [88] : SAMP [28] , DN-Synt [68] , and AffordMotion [70] . SAMP and DN-Synt utilize VAE architectures, while AffordMotion employs a diffusion-based model.To adapt these methods for our diverse HMP task, we initialize motion synthesis from the last observed frame x L and encode the complete observed sequence X 1:L into a unified embedding for historical motion conditions using a 2-layer transformer encoder, similar to the embedding technique in [57] . Additionally, we introduce the FID metric, commonly used in motion synthesis, to evaluate the discrepancy between the distributions of generated and original dataset motions.The results in Table 3 reveal an intriguing pattern: synthesis methods exhibit higher ADE than FDE. This occurs because, although these methods include explicit end-pose estimators yielding accurate final pose predictions, they struggle to condition on past motion. Consequently, they produce motion incoherence and significant prediction errors along the trajectory from the observation to the final pose. Notably, AffordMotion achieves the best APD and FID scores, which we attribute to its design that prioritizes fidelity over accuracy and allows a higher degree of freedom. Meanwhile, DiMoP3D also demonstrates competitive performance in these metrics. These findings underscore DiMoP3D's capability to harmonize the stochastic nature of human motion and the deterministic constraints from the scene and the past motion, achieving superior performance in diverse scene-aware HMP, and maintaining competitive diversity and fidelity even when compared to SoTA synthesis method. 
conclusion and limitation: 
 This work introduces a novel task of predicting diverse human motion in 3D scenes, along with a novel framework, DiMoP3D, to address it. By incorporating multimodal motion-scene analysis, DiMoP3D identifies areas or objects the subject is likely to interact with, enabling diverse, accurate, and physically consistent human motion prediction. Evaluated on the GIMO and CIRCLE datasets, DiMoP3D reduces ADE and FDE by nearly half compared to the state-of-the-art baseline BeLFusion, while maintaining high physical consistency. These results underscore the importance of scene awareness in diverse human motion prediction for real-world applications.Despite its strong performance, DiMoP3D predicts motion in a fixed sequence length. When the actual sequence length differs, it either keeps the subject relatively static or truncates the sequence. Future work could explore variable-length motion prediction or end-to-end prediction, where the motion generator predicts sequence length and generates motion simultaneously. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes]Justification: We have reviewed the NeurIPS Code of Ethics and ensured that our research conforms to it.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: The proposed DiMoP3D may have broader impacts on robotics, autonomous vehicles, video games, assistive technologies, etc., as detailed in Appendix E.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: There is no high risk for misuse in the data or models. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: The licenses and terms of use are explicitly mentioned and properly respected. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: Training DiMoP3D encompasses separate phases for InterestNet, HOI-Estimator, and the selfprompted motion generator. This section delineates the specific training methodologies for each component. All training is conducted on a single NVIDIA RTX3090 GPU, with the complete pipeline converging in ∼ 8 hours.Training InterestNet. We adopt the ScanNet [21] pretrained SoftGroup model as our encoder and segmenter to enhance performance, and a transformer decoder for the interest net to achieve cross-modal analysis. The original datasets [4, 88] lack annotations for the interest map M ; hence, we enrich them with such annotations for each motion clip. InterestNet is employed to elucidate the relationship between the observed motion and the scene, facilitating the prediction of human interests, including likely destinations and objects of interaction. To ensure the interest map accurately represents these aspects, we annotate the human interest map M based on three critical factors: the contact area between humans and objects (M cont ), the proximity of humans to scene elements (M dist ), and the spatial relationship between each object O i and the target object O g (M obj ):EQUATIONEQUATIONEQUATIONEQUATIONHere, p cont denotes the human-object contact position, f N (µ,σ) (•) denotes the Gaussian function with mean µ and standard deviation σ. Smaller σ concentrates the interest map while larger σ distributes attention more broadly. We set σ 1 = 0.3 and σ 2 = σ 3 = 1.0 for balance. Dist3D(•) and Dist2D(•) are the 3D and X-Z 2D distance functions (Y-axis denotes height). Hyperparameters λ cont = 3, λ dist = 10, λ obj = 1 are adjusted to maintain a balance among the factors.Upon annotating the interest map M , InterestNet is trained using a KL divergence loss to minimize the discrepancy between the distributions of the predicted interest map and the annotated M :EQUATION)Training HOI-Estimator. The HOI-Estimator employs an autoregressive conditional variational autoencoder (cVAE) [34] architecture, designed to predict a range of feasible end-poses. It predicts interactive end-poses xend based on the target object's point cloud O g . Initially, we centralize the object on the X-Z plane (assume the Y-axis denotes height) by normalizing its position to the origin:EQUATIONwhere len(O g ) indicates the point count in O g and p xz is the X-Z coordinates of point p. The HOI-Estimator utilizes a conditional Variational Autoencoder (cVAE) [34] for encoding-decoding:EQUATIONEQUATION)The HOI-Estimator is trained to minimize the L2 loss between the reconstructed interactive pose xend and the ground-truth x end :EQUATION)Training motion generator. The motion generator is pre-trained on the HumanML3D dataset [25] .We predict the clean sample X0 directly instead of predicting noise following [63, 64, 83] , consisting of the basic diffusion loss: EQUATIONEQUATION)Description Template of Semantic Inspector. The semantic inspector incorporates language description to supervise the predicted motion to be consistent with target objects. To enable this supervision, we design a language description template for each class of objects in the format of:The person walks forward thenEQUATIONwhere [ACT ] and [OBJ] are placeholders for the action and object, respectively. There are a total of 18 classes of objects (excluding wall and floor) in the pre-trained ScanNet dataset [21] , and we design corresponding [ACT ] for each class, detailed in Table 4 .To facilitate diverse trajectory prediction, we enhance the conventional A* trajectory planner, allowing for the iterative generation of valid paths while penalizing previously traversed positions.Initially, we generate the scene height map S H as described in Alg. 1. To streamline height analysis and A* pathfinding, we voxelized the scene point cloud S along the X and Z axes (with the Y-axis representing height) into a grid of 0.02-meter resolution. We then identify the maximum height within each grid cell, omitting ceilings and tall cabinets, which, despite their height, do not impede movement and would otherwise be inaccurately marked as obstacles.Leveraging the generated height map S H , we outline our modified dynamic A* trajectory planner in Alg. 2. Our method transcends traditional one-hot encoding for marking obstacles by enabling navigation over lower barriers through a continuous cost function derived from S H . We adopt a power function to model the cost, with cell height acting as the exponent. To ensure smoother trajectories and reduce sudden changes, an L2 penalty on angular velocity is integrated into the cost framework. Additionally, to prevent the selection of repetitive paths, we increase the cost of cells once traversed. The modified A* algorithm then iteratively generates paths until the cumulative cost exceeds a predefined threshold or the maximum path count is attained, as in Figure 5 .Scene feature for baselines. To assess the influence of scene features on baseline methods, we conduct supplemental experiments integrating scene features into these methods. Despite the baseline models mentioned in the main paper [5, 72, 82] not inherently accommodating scene features, we 5 : Results of appending scene features for the baseline methods on GIMO [88] .scene APD↑ ADE↓ FDE↓ MMADE↓ MMFDE↓ ACPD↓ Dlow [82] 55. 12 augmented their input by appending the scene feature F s along the temporal dimension for [5, 82] , and along the joint dimension for [72] , to explore potential performance.The results in Table 5 reveal that Dlow failed to convergence, likely attributed to its recursive architecture, wherein the concatenated scene features diverge significantly from the distribution of motion features, making the network difficult to learn. Both SmoothDMP and BeLFusion exhibit marginal improvements, suggesting that the direct integration of scene features into the motion generator yields limited effectiveness. Conversely, these results underscore the efficacy of DiMoP3D's strategy of predicting the conditional factor, highlighting its superiority in harmonizing the scene and the observed motion.Ablation on the scene segmenter. To evaluate DiMoP3D's robustness across various point cloud instance segmentation methods, we performed additional experiments with different segmentators. Higher-quality segmenters yield more precise object delineations, enabling the selection of more accurate targets and reducing object boundary violations. Table 6 illustrates that DiMoP3D maintains stable performance even when the quality of the segmentation method declines, underscoring its robustness to variations in segmentation.Ablation on the trajectory planner. To validate the significance of our enhanced A* trajectory planner, we conducted an ablation study. Results presented in Table 7 show that the naive A* method underperforms due to its deterministic nature. Excluding the continuous cost model results in a binary scene representation, disregarding traversable lower obstacles and thus reducing path diversity. Additionally, omitting the angular velocity penalty leads to paths with abrupt turns, detracting from prediction accuracy. These outcomes significant the critical role of our modified A* trajectory planner in achieving nuanced and reliable path predictions.Instance segmentation and interest map. Exploring our multimodal scene parser, we present visualizations of 3D scene instance segmentation alongside the corresponding interest maps M for three distinct samples. Figure 6 (upper) showcases the segmented instances within the scene, while Figure 6 (lower) evidences our multimodal InterestNet's capacity to deduce potential human intentions.In the bedroom setting, with the person initially stationary and then beginning to move forward, the future action remains ambiguous. Consequently, almost all objects are highlighted as potential targets in M , except for the door situated behind the individual. In the living room scene, as the person navigates the narrow gap between a chair and a table, the sofa, an additional chair, and the table emerge as points of interest. Conversely, the proximate chair and a distant door garner lesser attention, aligning with the observed motion pattern. A similar observation is noted in a laboratory environment.These results underscore our multimodal scene parser's adeptness at inferring potential human intentions, and the generated interest map M can accurately reflect anticipated human interests and interactions.Multiple samples with single target. To further explore DiMoP3D's capacity for predicting diverse motions toward a deterministic target object, additional visualizations are presented.The left panel of Figure 7 , showcases predicted sequences where the person navigates different paths to reach the destination, performing varied actions such as looking down and reaching toward the target. The right panel illustrates sequences of the person adopting different positions for lying and sitting on the bed. These results affirm DiMoP3D's adeptness at generating varied and coherent human motions aimed at a specific target object, further ensuring diversity.Comparison with synthesis methods. To further explore the performance of synthesis methods in diverse scene-aware HMP tasks, we present visual comparisons between DiMoP3D and the SoTA synthesis method AffordMotion in Figure 8 . AffordMotion's predicted sequences tend to remain relatively static at the onset of prediction, which we attribute to its method of smoothly synthesizing The proposed DiMoP3D introduces a novel diverse scene-aware motion prediction framework, which may involve the following broader impacts: NeurIPS Paper Checklist 


Decision: This paper proposes the novel task of predicting diverse human motion in real-world 3D scenes, therefore abiding by stochastic predictions and 3D-scene-conditioning, to test with proposed metrics in the datasets of GIMO and CIRCLE and a new model for the task termed DiMoP3D.

Upon the rebuttal and the authors-reviewers discussion, the reviews were diverging, and one was borderline: 1x reject, 1x borderline reject, 1x weak accept, and 1x strong accept.

The AC summarizes these main aspects:

(1) The paper is clear and well-motivated, as agreed by most reviewers and confirmed by the AC

(2) The performance of the proposed model is effective, and the selected baselines are sensible, as agreed by all reviewers and confirmed by the AC.

(3) The proposed task is novel. Two reviewers are convinced about this aspect. Two reviewers are not convinced of this. The AC notes that the discussion with related work needs to be improved. The many raised questions clearly show that the authors need to make clearer points on the novelty and how this task differs from the existing ones. Effectively, the AC understands the two contrary reviewers' viewpoints, as this task appears incremental with respect to those tasks and benchmarks that are currently available. On the other hand, the AC stands on the positive side, arguing that proposing to bring together stochasticity and 3D real scenes is new and a great direction to proceed in this research field.

(4) The modules employed in DiMoP3D are mostly derived from best practices with limited novelty.

(5) As proposed, the new task is deemed helpful in progressing the relevant application fields. The proposed datasets and metrics appear to be an appropriate step forward. Progress on this task will likely reflect in progress across all the existing compound subtasks. This paper is the first to have proposed the task of diverse scene-aware motion prediction. The selected datasets are recent and large enough to ignite follow-up work on the task. The metrics are best practices in the field of scene-aware and stochastic human motion prediction, so the performance depicted in the paper is representative of modeling progress.

Further to the notes on novelty and performance, the motivation stems from the vast utility and timely need of the proposed task for application fields of significant potential impact, such as human-robot-scene navigation and interaction. As such, we recommend acceptance of this paper.

Reviews: review: paper_summary: paper_summary: This paper introduces a task that is scene-aware diverse human motion prediction. To be specific, given a 3D scene and history motion, this task aims to predict diverse future human motions that are consistent with the scene and history motion. This paper also proposed a model, DiMoP3D, to tackle this task. This model includes a Context-Aware Intermodal Interpreter to encode the scene and find the goal object, a  Behaviorally-Consistent Stochastic Planner to generate the end pose and an obstacle-free trajectory, and a diffusion model to generate the motions on the path.
strength_weakness: strength_weakness: 1. As my understanding, if the author follows the original data split of CIRCLE and GIMO, there are no multiple possible future motions for history motion. How to make sure the model can generate diverse future motions trained on such datasets?

2. Please compare the proposed task with [9, 29]. These two works have already tackled the task of scene-aware stochastic prediction. Why do the authors claim they proposed a new task?

3. The author claims that the predictions must adhere to deterministic constraints, including physical consistency. The proposed method only considers the obstacle-free trajectory, but there is no constraint for the generated poses, e.g., there could be penetration with the ground for the walking sequence. 

4. What is the insight and novelty of DiMoP3D to make it different from the previous paper? Most techniques are similar to previous work. The scene encoder is from [71], the prediction of diverse goals is similar to [9], the HOI estimation is similar to [1, 2], the trajectory planning is similar to [77, 82, 29], the motion generator is similar to [87].

5. Some parts are hard to read, e.g., the introduction.

6. For the 3D human motion prediction/generation tasks, I prefer to see a video rather than only some figures for the visualization part.

[1] Zhao, Kaifeng, et al. "Compositional human-scene interaction synthesis with semantic control." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

[2] Hassan, Mohamed, et al. "Populating 3D scenes by learning human-scene interaction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
questions: questions: The question is the same as Weaknesses. Overall, the paper seems to propose a valuable task and has good results. Addressing the limitations mentioned above would significantly strengthen the work and I am positive to change the score.
limitations: limitations: The authors addressed the limitations.
paper_summary: paper_summary: The paper introduces a novel task that incorporates real-world 3D scene information into the existing Human Motion Prediction task. The authors propose a model (DiMoP3D) that, starting from the observed motion sequence and the 3D scene, stochastically predicts the future poses and the interactions with the context; in particular, DiMoP3D infers the individual intention and the human-object interactions. Upon benchmarking on two datasets, GIMO and CIRCLE, the model surpasses all the baselines on most of the considered metrics.
strength_weakness: strength_weakness: W1. While it is clear how their proposed task diverges from human motion synthesis in 3D scenes, human-object and scene interaction prediction (e.g., [5]), and social navigation, it is not clear enough how it differs from scene-aware 3D human motion forecasting or synthesis. 

[5] Xu, Sirui, et al. "Interdiff: Generating 3d human-object interactions with physics-informed diffusion." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
questions: questions: I acknowledge that Sec. B.1 of the supplementary extends the literature review in Sec. 2 of the main manuscript, though the discussion could benefit from a tighter and more direct comparison with similar tasks in [1,2,3,4].

Typos and minor issues with the writing:

L99: forgot the full stop at the end of the sentence.

L119: “intermodal insights. utilizing…”

L181: “To tackle ress these”

L211: “When \alpha_T is approaches 0…”

L261: “...is introduces…”

L304: “...by the subject en route…”
limitations: limitations: Yes, they included the discussion about limitations and societal impact in Section G of the supplementary material.
paper_summary: paper_summary: This work studies human motion prediction in 3D scenes. The proposed DiMoP3D leverages context-aware intermodal interpreter, behaviorally-consistent stochastic planner, and self-prompted motion generator to solve the task. The authors conduct experiments on GIMO and CIRCLE to demonstrate a superior performance.
strength_weakness: strength_weakness: - I question the novelty of this work. Novelties in Context Awareness and Autonomous Intention Estimation are overclaimed. They have been studied in previous works but authors fail to provide a discussion:
[29] uses path planning to perform context-aware navigation and GoalNet to estimate the end-point state. 
[98] uses 3D scene point to encode the scene context and gaze to estimate the intention.
Still more work to supplement...
I believe the novelty is very overclaimed.


- The system estimates the motion of a fixed length $\Delta L$. Is it possible to not reach an end state? Taking Fig.2. as a case, how do you guarantee the human is sitting on the chair but not still walking to it after $\Delta L$? It may lead to unnature speed and motions.


- How do you evaluate the diversity and natureness of motions?
questions: questions: See weakness part.
limitations: limitations: As discussed in Appendix G, the system is not as efficient as baselines.
paper_summary: paper_summary: The paper present a method for human motion prediction that take into account the environment (3d point cloud) in which the person moves. The method work with several steps: first from the past motion an area of interest is estimated and several object of interest (bed, chair...) with which the person might interact. after selecting the most likely object the method will estimate an interaction with the object (e.g sitting on the bed or lying on the bed) and plan a trajectory to reach the object. Finally with the trajectory and the estimated interaction the model generate the 3d motion from the end of the past motion to the end of the interaction. Since this is a novel task the author compare to two type of method : motion prediction and environment aware motion synthesis and outperform the state of the art on both tasks.
strength_weakness: strength_weakness: At least some results of the scene aware motion synthesis from the appendix should be moved to the main paper. Especially since there is no qualitative comparison with other scene aware method in the main paper.

In figure 1 it is not clear that diffusion is being used.

A qualitative comparison with bifu would have been interesting since its results are the best after the proposed method.
questions: questions: Videos of the generated motion would have been appreciated

line 261 introduces => introduced
limitations: limitations: Limitations and impacts are discussed in the appendix.

Comments: title: Response to Reviewer wnWw by Authors
Dear Reviewer wnWw
Dear Reviewer nnKQ
Reply to Reviewer nnKQ from Authors
Reply to Reviewer p6AA from Authors
Comment from authors
Comment to Reviewer wnWw from authors
Reply to Reviewer nnKQ from Authors
rating after rebuttal
comment: I appreciate the author's rebuttal. After thoroughly reviewing the paper, supplementary materials, rebuttal, and comments from other reviewers, I have decided to maintain my rating as borderline reject based on the current quality of the work. While I recognize the contribution of the proposed new task, the contribution of the method remains limited, and the author's arguments have not convinced me. I concur with reviewer wnWw on the importance of faithfully reviewing related works with similar motivations, properly attributing their contributions and acknowledging their limitations. I recommend the author address these concerns in the main paper to improve the overall quality of the work.
Thank you for your thoughtful feedback on our rebuttal.  We aim to clarify why our work represents a significant contribution to the field.

**Novelty and Impact:**

Our work introduces a new perspective on human motion prediction that is both task-driven and methodologically innovative. While it is true that the proposed framework integrates existing techniques, the way in which these are combined and the novel problems addressed represent a substantial advancement:

- **Task-Oriented Approach:** Our work focuses on a specific challenge—predicting human motions in 3D scenes with real-world awareness, a problem that has not been adequately addressed in previous literature. We argue that *this novel task is a critical step towards enabling intelligent agents to navigate complex environments more safely and effectively*. By incorporating real-world scene information into the motion prediction process, our solution goes beyond merely predicting skeletal poses.

- **Pioneering Approach: Real-World Scene Awareness.** Our work, DiMoP3D, *pioneers the integration of 3D real-world scene awareness*into the inherently stochastic task of human motion prediction. This is a fundamental shift from traditional skeleton-based methods, enhancing the predictive capabilities of AI systems in complex environments.

- **Performance Gains: Enhanced Diversity and Naturalness.** Our framework achieves higher diversity and naturalness in motion predictions, as evidenced by our quantitative evaluations (APD, ACPD, and FID scores) and qualitative visualizations. These improvements are crucial for practical applications where the ability to predict a wide range of plausible human motions is paramount.
  
**Response to Incremental Concerns:**

We acknowledge that the combination of techniques may appear incremental at first glance, but we contend that the integration itself is non-trivial and leads to a significant leap forward in terms of performance and applicability (as commentator nnKQ notes). Moreover, the specific challenges addressed, **corner cases** (e.g., how to deal with a person arriving early in a prediction window, or not being able to arrive), and the **new challenges of this new task** (e.g., how to simultaneously balance the deterministic nature of the 3D scene given, as well as the stochastic nature of the human body's movements), require careful consideration and innovative solutions.


All these considerations suggest that the proposed DiMoP3D possesses enough novelty to go beyond simply incremental progress, especially for this newly proposed task of motion prediction in a 3D real-world point cloud.


/* **Remark:**  *We respectfully request that Reviewer wnWw consider the broader implications of our work within the context of human motion prediction in realistic environments. DiMoP3D represents a significant stride in advancing the field towards more human-like AI systems capable of understanding and anticipating behavior in complex, real-world scenarios. We believe this contribution represents a substantial leap forward for the community and the practical application of embodied AI.* */

We are grateful for the opportunity to refine our work based on the feedback provided. Addressing these concerns will not only strengthen our manuscript but also emphasize the importance of DiMoP3D in advancing the field of computer vision and AI towards better human-scene understanding and interaction capabilities.
I thank the authors' responses in their rebuttal. They have addressed my concerns about evaluation and efficiency. I have also read discussions with other reviewers and I share the same concerns about the novelty. I agree a mixture of existing techniques and the mixed task (historical conditions, scene awareness, intention estimation) can be considered as novelty, but somehow incremental. Besides, it is important to review related works with similar motivations faithfully and attribute their contributions and limitations. However, I am not very convinced by their texts. Plus the missing non-trivial details in tackling corner cases, this work is not very convincing to me and not ready for publication. Though I appreciate the authors' efforts in their rebuttal, I still suggest a major revision and can not recommend acceptance at this time.
Thank you for your insight to our work! We kindly remind you that the discussion deadline is within 24 hours. We believe that our rebuttal have addressed your concerns, and we are also prepared to answer any further questions you may have.

With this in mind, would you consider revising your rating, or if there are further questions or concerns, can we engage in more discussion!
Thank you for your interest in our work, your prompt responses, and your thorough engagement with us! We kindly remind you that the discussion deadline is within 24 hours. We trust that our comprehensive responses regarding synthesis tasks and methods have addressed your concerns. We are also prepared to answer any further questions you may have.

With this in mind, would you consider revising your rating, or if there are further questions or concerns, can we engage in more discussion!
**A7**: Thank you for your insights. While treating historical motion as a condition has proven effective in certain synthesis methods, it poses significant limitations in the context of diverse prediction:

**(1) Lack of crossmodal analysis between past motion and scene**. Even when incorporating and infilling the observed sequence into diffusion-based synthesis methods, they often lack an explicit crossmodal analysis between past motion and the scene. This analysis is crucial for inferring potential human intentions and goes beyond traditional scene-motion attention in synthesis methods.

**(2) Difficulty in reconstructing observation**. To ensure consistency between the prediction and observation, the predictor in [3] is forced to predict observation as well (train_diffusion_skeleton.py, line-190 in official github repo of InterDiff). However, accurately reconstructing observation is also challenging. During inference, the predicted observation part (0.33sec out of 1.07sec in total in InterDiff) is in fact apart from ground truth. In our long-term prediction task with a 3-second observation phase, this method would struggle to reconstruct long-term observations accurately, which negatively impacts performance.

**(3) Motion incoherence**. For methods that not infilling observation during inference, there may be issues in motion incoherence between observed and predicted sequences, as noticed by [4,5]. As a remedial measure, [4] proposes adversarial training to force the module to reconstruct the sequence (including observation) from the past motion embedding; [5] employs a post-refinement method to align the sequences better. These methods are more complex and introducing additional computational overhead.

**(4) Posterior collapse**. [5] finds that encoding past motion as a condition can cause posterior collapse during joint training, as strong decoders may ignore the learned latent variables. DiMoP3D overcomes this by decomposing the task into sub-tasks, making each sub-module easy to learn, ensuring both high fidelity and diversity even with limited data. 

**(5) Scene-awareness**. There are limited scene-aware synthesis methods, most of them (including [1,2]) lack scene awareness. The SoTA diffusion-based scene-aware synthesis method AffordMotion(CVPR24) [6], discussed in Appendix B, has limitations: **(a) past motion embedding** causes significant motion incoherence (see Appendix B.3), and **(b) observation infilling** may lead to significant posterior collapse.

We will incorporates more detailed dicussion and experimental comparison with synthesis methods in the next version!

**A4**: The primary novelty of our work lies in introducing the task of diverse motion prediction in real-world 3D scenarios, a challenge not addressed by current methods, as confirmed by Reviewer p6AA. Our approach deconstructs the challenge and adapts advanced methods from the community to fit this unique context. Significant enhancements have been made to existing technologies, including Scene Interpreter (crossmodal analysis), Path Planner (efficient stochastic navigation), and Motion Generator (self-prompted on the end-pose and trajectory). We kindly request that you reevaluate our contributions and novelty from this perspective, considering our comprehensive approach to addressing the complexities of diverse scene-aware motion prediction.

[1] Ye Yuan, Jiaming Song, Umar Iqbal, Arash Vahdat, and Jan Kautz. PhysDiff: Physics-guided human motion diffusion model. In ICCV, 2023.

[2] Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. MotionDiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208.15001, 2022. 3, 4

[3] Xu, Sirui, et al. "Interdiff: Generating 3d human-object interactions with physics-informed diffusion." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[4] Barquero, German, Sergio Escalera, and Cristina Palmero. "Belfusion: Latent diffusion for behavior-driven human motion prediction." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[5] Wei, Dong, et al. "Human joint kinematics diffusion-refinement for stochastic motion prediction." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 5. 2023.

[6] Wang, Zan, et al. "Move as You Say Interact as You Can: Language-guided Human Motion Generation with Scene Affordance." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
Thank you for your insightful feedback and for recognizing the core ideas and motivations of our work. We are honored to address your concerns and appreciate your alignment with our approach. This article introduces a novel and crucial task in the fields of robotic and autonomous navigation. DiMoP3D is proposed as a solution to this task accordingly, addressing challenges that existing methods cannot overcome. 

If there are any further issues, we are more than willing to discuss them!
My main concerns, specifically regarding Q4 and Q7, have not yet been addressed.

For Q4, the novelty and insight of the proposed method remain underwhelming. 'While the proposed framework is well-structured, it builds upon and integrates existing methodologies, leading to a limited novelty. The primary contribution seems to lie in the Scene Interpreter, which predicts potential target objects from 3D point clouds, as opposed to the previous work [9] that predicted targets from images.'

For Q7, although the results are good, it lack a compelling argument. Your reasoning is that "In time-series predictive tasks, history should not be treated as a conditional factor but as a foundational element that future sequences must adhere to. Treating historical information as a condition overlooks the temporal correlation between history and the future, whereas using it as a fundamental element ensures that the predicted results strictly maintain consistency with historical information." However, this appears to be just a different design choice for the diffusion process. Diffusion-based scene-aware synthesis methods could also adopt this approach, such as [1, 2], to predict future frames by infilling ground truth past motion into the denoised motion at each diffusion step. On the other hand, Interdiff [3] observes that encoding the historical motion as a condition leads to better performance. I do not see any limitation here. Please review my questions carefully.

I believe the key difference between scene-aware human motion synthesis methods and stochastic scene-aware human motion prediction lies in the condition. Please provide a robust argument regarding this distinction for your proposed methods. 

[1] Ye Yuan, Jiaming Song, Umar Iqbal, Arash Vahdat, and Jan Kautz. PhysDiff: Physics-guided human motion diffusion model. In ICCV, 2023.

[2] Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. MotionDiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208.15001, 2022. 3, 4

[3] Xu, Sirui, et al. "Interdiff: Generating 3d human-object interactions with physics-informed diffusion." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
I thank the authors and the other Reviewers for their insights.

I acknowledge that the authors' rebuttal answers my concerns and I appreciate the authors' commitment to further improve the clarity of the discussion of the submitted manuscript.

I understand the points that Reviewer nnKQ made: adding more comparisons with methods from similar tasks could clarify more the difference between the proposed task and model and existing works, and why the latter cannot be naively repurposed. However, I disagree with the comment regarding the limited novelty; to the best of my knowledge, the proposed task is indeed part of the paper's novelty and, even if it shares similarities with other tasks, it differs from them likewise the task of human motion forecasting differs from stochastic human motion forecasting or human motion forecasting with contact points. In my opinion, leveraging existing methodologies to tackle a novel problem shouldn't necessarily be regarded as a weakness, since the selection and the adaptation of such methods, if wisely devised, are already a contribution and a starting point to face the new task.
Happy to solve your concerns, and thank you for the time and effort!
Dear Reviewer wnWw:

Thank you for the time and effort you've invested in reviewing our manuscript. In our previous response, we addressed your concerns directly and comprehensively. We greatly appreciate your further feedback on our responses and look forward to discussing them with you!
**A1**: We appreciate the reviewer's point highlighting the challenging of our task as compared to traditional stochastic prediction. DiMoP3D overcomes this by decomposing the problem into target prediction, interactive pose estimation, path planning, and self-prompted motion generation that handle different aspects of the scene-aware diverse motion prediction. Thus, $ P(\hat{X}_{L:L+\Delta L} | X\_{1:L}, S) $ could be represented as the product of terms: $P(O_g | X\_{1:L}, S)$, $P(\hat{X}\_{L+\Delta L} | X\_{1:L}, S, O_g)$,  and $P(\hat{X}\_{L:L+\Delta L} | X\_{1:L}, \hat{X}\_{L+\Delta L}, \hat{\tau}^{plan}, S)$. **(1)** Our interpreter learns the distribution rules of potential targets within scenes $P(O_g | X\_{1:L}, S)$. For instance, a subject traversing an aisle is modeled to likely interact with objects along that path, while a subject facing a chair is likely to sit. This module capitalizes on patterns in the relationship between past motion and scene context to predict potential targets, effectively modeling the target distribution despite the limited frequency of specific motion histories. **(2)** Pose estimator models how interactive poses vary with different objects within the scene $P(\hat{X}\_{L+\Delta L} | X\_{1:L}, S, O_g)$. while similar motions in different scenes might lead to different interactions, the core interaction with objects (like sitting or picking) remains consistent across contexts. **(3)** Motion generator models coherent human movement based on learned stochastic factors $P(\hat{X}\_{L:L+\Delta L} | X\_{1:L}, \hat{X}\_{L+\Delta L}, \hat{\tau}^{plan}, S)$. The distribution of walking movements and the coherent switching of different poses is rich in our scene-aware data, making it easy to learn.

**A2**: **(1)** They ultilize pre-defined targets, meaning that the target object and location is fixed and known before the prediction begins, while we have to predict the distribution of potential targets dynamically by analyzing past motion and the scene context. **(2)** The scene-awareness of [29] is only based on the 3D voxel grid (8x8x8) of the pre-defined target and planning deterministic paths. In the contrary, DiMoP3D engages with the entire 3D scene, incorporating detailed, real-world environmental data. **(3)** They operate within a single, manually synthetic scene, where all objects' shapes, positions and features are known to the system, eliminating the need for the module to learn. In real-world applications, the ground truth of the scene and object is unaccessible, limiting its use. However, our DiMoP3D is designed to recoginize and analyze potential targets in a variety of real-world scenes.

**A3**: Incorporating the Signed Distance Field (SDF) of the human mesh and scene point cloud into the reinforcement learning module’s reward function may help avoiding issues like object penetration. While this approach is promising, the complexity of integrating it with real-world 3D scene data is substantial. Given that it is not the core focus of this work, we leave it in future research, which is a promising direction.

**A4**: For comparison to scene-aware synthesis, kindly refer to Appendix. B. We emphasise that DiMoP3D significantly adapts existing methods to the novel task of scene-aware diverse motion prediction. This is not merely an incremental improvement; it addresses a previously unexplored task. We believe our method represents a significant leap in the field, pioneering a new and promising track for motion prediction in real-world scenarios.

**A5**: Welcome for any additional feedback!

**A6**: We emphasize that our task diverges from scene-aware synthesis and deterministic prediction. Nonetheless, for the sake of completeness, we compare our approach with scene-aware synthesis in Appendix B. Additionally, we provide a supplemental visualization comparison with scene-aware BiFU in Section 5 on our anonymous page (line 588 in Appendix A).

**A7** Even with the last observed frame and embedding of past motion incorporated, synthesis methods still struggle to predict future motions coherently (especially switching between observation and prediction). In time-series predictive tasks, history should not be treated as a conditional factor but as a foundational element that future sequences must adhere to. Treating historical information as a condition ignores the temporal correlation between history and the future, while using it as a fundamental element means that the predicted results must strictly maintain consistency with historical information. This is the underlying reason why using historical information as a condition for LLM (or the synthesis methods) cannot achieve high fidelity prediction. Appendix B provides specific qualitative experiments, in which the discontinuity between historical and predicted actions in the comparative method (AffordMotion, CVPR'24) provides evidence for our statement.

**We are looking forward for further discussions!**
Thanks for the author's clarification! 

Q1: I understand the authors are trying to model the posterior distribution of potential motions. However, for the stochastic human motion prediction task, the posterior distribution is $P(X_{L:L+\Delta{L}}|X_{1:L})$ and there are many motions with similar history but different futures in the dataset, allowing them to learn a good distribution. But for the scene-aware stochastic human motion prediction task, the posterior distribution is $P(X_{L:L+\Delta{L}}|X_{1:L}, S)$ and there are very few motions with similar history but different futures in the same scene.

Q2: For [29], what is the key difference? And why does it lack such a capability?

Q3: Thank you for your experiments. However, the scene in this task is considerably more complex than in PhysDiff. The generated motion is not constrained solely by foot placement; other factors must also be considered. For example, there might be cases where your generated motion results in penetration with objects like tables or chairs when sitting down. Walking and penetrating the ground is just one scenario among many.

Q4: In my opinion, scene-aware stochastic motion prediction is distinctively conditioned on historical motion, setting it apart from other scene-aware synthesis methods. Therefore, it is equally important to compare it with these scene-aware synthesis methods rather than solely focusing on stochastic motion prediction. While the proposed framework is well-structured, it builds upon and integrates existing methodologies, leading to a limited novelty. The primary contribution seems to lie in the Scene Interpreter, which predicts potential target objects from 3D point clouds, as opposed to the previous work [9] that predicted targets from images.


Q5: Thanks for your clarification.

Q6: There are only comparisons with BelFusion. A comparison with scene-aware methods is also important.

Q7: I acknowledge that your results will likely surpass those of BelFusion since you incorporate scene information, whereas BelFusion does not. However, I'm more curious about the comparison between your method and other scene-aware approaches. For instance, you could easily adapt language-conditioned scene-aware synthesis methods by replacing the condition from language to history motion. What insights do you have that demonstrate your method's superiority over previous scene-aware methods? In what cases do previous methods fall short, and how does your approach address these limitations?
the rebuttal answered most of my and the other reviewers concerns, I keep my original rating.