Harmonizing Stochasticity and Determinism: Scene-responsive Diverse Human Motion Prediction
Authors: Zhenyu Lou
Qiongjie Cui
Tuo Wang
Zhenbo Song
Luoming Zhang
Cheng Cheng
Haofan Wang
Xu Tang
Huaxia Li
Hong Zhou
Venue: NeurIPS 2024 poster (2024)
Keywords: Human Motion Prediction
Diverse Motion Prediction
3D Scene-aware
Multimodal
Primary area: machine_vision

Abstract: Diverse human motion prediction (HMP) is a fundamental application in computer vision that has recently attracted considerable interest. Prior methods primarily focus on the stochastic nature of human motion, while neglecting the specific impact of external environment, leading to the pronounced artifacts in prediction when applied to real-world scenarios. To fill this gap, this work introduces a novel task: predicting diverse human motion within real-world 3D scenes. In contrast to prior works, it requires harmonizing the deterministic constraints imposed by the surrounding 3D scenes with the stochastic aspect of human motion. For this purpose, we propose DiMoP3D, a diverse motion prediction framework with 3D scene awareness, which leverages the 3D point cloud and observed sequence to generate diverse and high-fidelity predictions. DiMoP3D is able to comprehend the 3D scene, and determines the probable target objects and their desired interactive pose based on the historical motion. Then, it plans the obstacle-free trajectory towards these interested objects, and generates diverse and physically-consistent future motions. On top of that, DiMoP3D identifies deterministic factors in the scene and integrates them into the stochastic modeling, making the diverse HMP in realistic scenes become a controllable stochastic generation process. On two real-captured benchmarks, DiMoP3D has demonstrated significant improvements over state-of-the-art methods, showcasing its effectiveness in generating diverse and physically-consistent motion predictions within real-world 3D environments.

Harmonizing Stochasticity and Determinism:
Scene-responsive Diverse Human Motion Prediction
Zhenyu Lou'
Qiongjie Cui
Cheng Cheng² Hong Zhou' 'Zhejiang University, Nanjjngg University of Science and Technology, Xiaohongshu Inc, *University of Texas at Austin, *Concordia University, 119150440zju. edu. cuiqiongjie0126. com
Tuo Wang*
Zhenbo Song²
Luoming Zhang'
Haofan Wang?
Xu Tang* Huaxia Li
Object Penetration
Incebereace
Scene Inconsictency
Fesiiosoon (ICCV23)
DiMoP3D (Ours)
Comparison of our DiMoP3D with the SoTA baseline [[] Purple meshes represent obser- vations, and yellow meshes denote predictions. DiMoP3D produces high-fidelity, diverse sequences tailored real-world 3D scenes, while BeLFusion's 's inadequate scene context integration leads to issues such as object penetration, motion incoherence, and scene inconsistency.
Figure 1:
Abstract
Diverse human motion prediction (HMP) is fundamental application in computer vision that has recently attracted considerable interest. Prior methods primarily focus on the stochastic nature of human motion, while neglecting the specific impact of the external environment, leading to the pronounced artifacts in predic- tion when applied to real-world scenarios. To fill this gap, this work introduces novel task: predicting diverse human motion within real-world 3D scenes. In contrast to prior works, requires harmonizing the deterministic constraints im- posed by the surrounding 3D scenes with the stochastic aspect of human motion. For this purpose, we propose DiMoP3D, a diverse motion prediction framework with 3D scene awareness, which leverages the 3D point cloud and observed quence to generate diverse and high-fidelity predictions. DiMoP3D can compre- hend the 3D scene and determine the probable target objects and their desired interactive pose based on the historical motion. Then, it plans the obstacle-free trajectories toward these interested objects and generates diverse and physically consistent future motions. On top of that, DiMoP3D identifies deterministic fac- tors in the scene and integrates them into stochastic modeling, making the di- verse HMP in realistic scenes become a controllable stochastic generation pro- cess. On real-captured benchmarks, DiMoP3D has demonstrated significant improvements over state-of-the-an th-theerrt methods, showcasing its effectiveness in gen- erating diverse and physically consistent motion predictions within real-world 3D environments. More details and the video demo are available at the webpage https://sites: http:///ssitss.gglleeeooom http://teee...oooeoooooo/r/////e wtt.://./ooo/riee//////o./
"Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Human motion prediction (HMP), i.e., forecasting future human poses based on observation, is cru- cial for applications including autonomous vehicles and human-robot collaboration [115, 20, 39, A4 154, 60. 666]. Many existing works [MI 22, 2n 29 76, |y formulate HMP as a deterministic problem, aiming to generate single future sequence. However, it fails to capture the inherent stochasticity of human motion, where multiple plausible outcomes can arise from single observation. Recent research has shifted towards diverse or stochastic HMP, to achieve multiple plausible predictions [B, 6, GO, 59 Z22), which holds the potential in real-world applications and is the focus of our work.
Recent advances in diverse HMP primarily focus on stochastic predictions, where a random factor from the latent space conditions the diversity of predictions alongside observed motions [3. 12 Bi 59, ZTI 72, 82]. While these methods predict multiple plausible futures from a single past motion, they typically disregard the 3D environment, operating within idealized, context-free framework. This limitation becomes apparent in real-world applications, where motion must conform to physical and semantic scene constraints [29 67L 18, 78, 90], leading to issues like obstacle penetration and unrealistic interactions, as in Figure M This gap underscores the need for a new task that merges diverse HMP within real-world 3D scenes, enhancing both practicality and applicability.
Recognizing existing limitations, this work introduces a novel task, making diverse HMP within real-world 3D scenes. Its objective is to break the previous idcalized context-free setup towarding realistic and practical setting, which involves several key challenges: (1) Harmonizing Stochastic- ity and Determinism: This task necessitates delicate harmonization between the stochastic nature of human motion and the deterministic constraints from 3D scenes, thereby broadening the scope of traditional HMP; (2) Scene-Motion Intermodal Coordination: It requires analyzing coordination between human motion and scene dynamics to align predictions with contextual elements, which involves identifying human intentions and potential interactive objects; (3) Behaviorally Coherent Physical Consistency: The predictions must adhere to deterministic constraints, including physical consistency (e.g., avoid collision) and behavior coherence (e.g., sitting on chair, not lying).
To tackle these challenges, we introduce DiMoP3D (Diverse Motion Prediction in 3D Scenes), a framework for generating diverse, physically consistent, and plausible human motion predictions in real-world 3D scenes, which comprises three main components: (I) Context-aware Intermodal Interpreter analyzes potential areas of human interest and goals, essentially intentions within scene. Our method enhances traditional scene understanding by integrating 3D point clouds with observed motions for context-aware intermodal analysis. It first encodes the point cloud, segments object in- stances, and then pinpoints potential interaction targets, emphasizing intended factors while filtering out less probable ones. This strategy further transforms the task of diverse HMP into a control- lable stochastic generation process; (2) Behaviorallly behaviorally consistent action plans, representing stochastic conditional factors. Recognizing that human interaction with specific objects often follows deterministic patterns, we prioritize predicting the final human pose upon reaching each target, and generate obstacle- free trajectories toward it; (3) Self- Self-prompted Motion Generator harmonizes the stochastic nature of human motion with determin- istic constraints in a self- seff-prompted manner to produce varied predictions based on the conditional factor. To ensure coherence with this factor, it employs denoising diffusion model, guiding the motion denoising process toward deterministic, obstacle-fre free final state.
Stochastic Planner then constructs
Our contributions are threefold: (1) We introduce a novel and challenging task of predicting diverse human motions within real-world 3D scenes, advancing beyond the traditional scope of diverse HMP from an idealized context-free setting to a more realistic and practical one. (2) We propose DiMoP3D to tackle this task. It harmonizes the deterministic constraints of 3D scenes with the stochastic nature of human motion, enabling diverse and plausible motion predictions in real-world scenarios. (3) Evaluations on two real-captured benchmarks, GIMO and CIRCLE, show that DiMoP3D significantly outperforms existing SoTA methods, particularly in terms of physical consistency.
Related Work
2.1 Stochastic Human Motion Prediction
Human motion prediction diverges into deterministic and stochastic methods. Deterministic models
aim to predict singular sequence that closely aligns with future movements [
L9.
200
[6]

---

yet often encounter quality degradation over longer time spans (> 1-sec) due to the stochastic na- ture of human movement. In contrast, stochastic HMP [2 R. m 82] embraces this variability by generating a range of plausible future motions and modeling the distribution of human behaviors. Notably, also encompasses the most probable future sequence targeted by deterministic models, as likely scenario within its broader distribution. Such methodologies enhance applications across autonomous driving [Z2 33 25],, patient care [TM0 B2], and human flow prediction [62] [6], by embrac- ing a wider range of potential scenarios and have become a focal point of contemporary research.
Dominant approaches include VAEs, GANs, flow networks, and diffusion models [E] 12, 50, 6X. 59, 7T, 72, 82|. Despite promising progress in modeling stochastic human motions, a critical chal- lenge persists: human motion is not only stochastic but also heavily influenced by the external en- vironment. In real-world settings, human motion is intricately intertwined with surrounding scenes, necessitating that future trajectories comply with the scene's physical constraints (e.g., avoid object penetration). Additionally, predicted motions should be semantically consistent with the expected human-object interactions (e.g., sit for a chair, lie for a bed). Addressing these requirements calls for sophisticated approach to diverse HMP that balances the stochastic aspect of human motion with the deterministic factors imposed by the surrounding 3D scenes, which is the focus of our work.
2.2 3D Scene Encoding and Scene-aware Motion Prediction
3D scene understanding is essential in various applications, prompting extensive research on rep- resentations like RGB-D maps [9. 52, IIII,, scene graphs [αZЛ, 77 84), 3D voxels [23, 28, Z4], and notably, 3D point clouds [555, 56
65,,
87 Recognizing the importance of 3D scene information
in human motion prediction [IE 211 26. 557 X0], our work integrates 3D point clouds as the scene representation due to their direct derivation from sensing technologies.
To enhance fidelity of HMP in real-world scenarios, the connection between human actions and scene context has made scene-aware motion generation a major research focus [8] T7, 28, 88]. Early methods [S. I7, 6 relied on object bounding boxes, 2D images, and depth maps, which are insuffi- cient for capturing real 3D environments. Recent advances use 3D point clouds for scene represen- tation [2% 59, 88]: GIMO [S]] employs a bidirectional transformer fuse human motion and scene features, [50]] predicts future contact maps, and [[G] extracts global and local salient points.
However, these methods predict single sequence [EL, 50 SX whereas approach models dis- tribution of potential outcomes. Some stochastic methods add diversity, but [8] relies on 2D inputs, limiting 3D interactions, and [28] uses predefined objects without inferring targets. In contrast, our model parses the 3D scene through cross- object-specific human interest analysis and predicts scene-aware motions real 3D scenarios, greatly enhancing adaptability and authenticity.
2.3 Motion Synthesis in 3D Scenes
have sparked a new research direc- Specifically, SAMP [αअ] em- ploys a conditional variational autoencoder (cVAE) to generate one frame per forward pass within three-stage stochastic pipeline, COUCH [86] introduces human-chair dataset with auto- regressive, contact-satisfynng method, and DN-Synt [68]] proposes a hierarchical framework for ef- fective scene-aware human motion synthesis. With the rise of language models, methods utilizing natural language prompts have emerged [T6 69, 75, 85J. HUMANISE [62]] proposes an attention- based minguage-prompter synthesis method. Additionally, diffusion models have shown significant promise [Ird 29, 87, 63,, 54 ZO]. Notably, AffordMotion [XT]] employs scene affordance as an inter- mediate representation, achieving state-of-the of-the-art performance in object interaction synthesis.
Recent advancements in paired scene-motion data [4
27
XN
tion in synthesizing motions in 3D scenes [29, A.
1
53
69
[
Our novel task of scene-aware diverse HMP is partly inspired by advances in motion synthesis but is tailored for distinctly different applications and challenges. This task is further distinguished by key innovations: (1) gemporal-Dependeer
Prompting. Contrary to motion synthesis, which often lacks temporal context and follows static user instructions, our approach conditions predictions the unique prompt of historical human motion, enabling autonomous inference of human intentions. (2) Awwte Scene Parsing. Moving beyond traditional scene understanding, our method infers potential movement targets through the integration of context-aware intermodal insights. Utilizing scene determinism enhance the fidelity of diverse HMP predictions.
3

---

Context-Aware
Intercat Map M
Intermodal Interpreter
Selected Instance
Carbbal Sanmas
Cion-tredal Introothi
Motion Sequence
Segmectte
Point Clood
Excoding
3D Sicene Polnt Cloud
Overwiie
Planning
Eaimasor
Decoder
for each
Pose
Noise
Nirocback
Setected Instance
sp Soand
Planee
T tinees
Rehaviorally-Consiiee Stochastic Planner
Self-Prompped Generator
Predicted
Poler
Sequences
The architecture of DiMoP3D. DiMoP3D incorporates modalities of input, the past motion and the 3D scene point cloud. Initially, the Context-aware Intermodal Interpreter encodes the point cloud features Fa, identifies interactive objects 0, and uses cross-modal InterestNet to pinpoint potential interest areas, sampling target instance O according to interest map M. Following this, the Behaviorally-conniiten Stochastic Planner forecasts the interactive human end- pose & LL+AL and devises an obstacle-fre trajectory pplan towards this pose. The sampled end-pose pose and trajectory are incorporated as stochastic conditional factor to prompt the Self Self-prompted Motion Generator generate physically consistent future motions.
Figure 2:
Method
3.1
Problem Setup
Given L historical human poses ILL = {x1, .2.... xL} within 3D scenes represented by point clouds S E X6 goal is to predict K different sceme-consistent future motions
{X LLLAL Here, np denotes the number of points, each described by 3D spatial coordinates and RGB color information. L and AL denote the lengths the observed and predicted sequences, respectively. Each pose is described
the SMPL-X representation æ; = (t), O₁, Pt) [S8], where t, E R denotes the global translation, 0₁ E 50(3) denotes the orientation, and pt E 1R refers to
the body pose embedding. We set L 3-sec and AL 5-sce achieve long-term prediction H0 49]. Then, the task can be formulated as:
UIL I.L.
max L. S, I.L. S)d0. X.L.
The stochastic process is sampled jointly from the past motion X1,L and the scene S, and then utilized to condition the prediction motion X LL.++AL- We propose DiMoP3D to solve this novel task, which involves the following novelties: Context awareness: unlike traditional diverse HMP methods [122 5% [X]]] that focus solely on human motion, our task is more challenging as requires harmonizing the stochastic nature of human motion with the deterministic constraints of the rounding 3D scenes. Autonomous intention estimation: different from motion synthesis, our task requires independent intention estimation based past motion, prompt future motion prediction.
3.2
Context-Aware Intermodal Interpreter
Scene information plays a crucial role in predicting future motion X 67 Despite the stochastic manner of human motion, it is still feasible to deduce human interests and likely goals within scene. instance, Figure Z the door behind person is unlikely to be target based on their trajectory away from it, while the sofa, coffee table, and distant chair may emerge as potential points of interest. Diverging from traditional scene parsing methods, our approach integrates scene-motion intermodal coordination to better suppose human intentions in real-world settings.

---

In machine vision, 3D point clouds sourced directly from sensing devices have become fundamental for scene representation [55] ESS 87] and serve as the input for our scene interpreter. Acknowledging the significant influence of past motions and scene context on future human movements, we empha- size the need for a context-aware intermodal analysis that integrates historical motion with scene point clouds to infer potential human intentions. Furthermore, since human movements typically involve interactions with target objects [T 24,, KI], our approach identifies objects within the scene and computes human interest score for each, rather than analyzing isolated points. This helps determine specific interactive targets, making the motion prediction process more controllable and enabling diverse prediction by sampling different interactive targets.
To achieve this, our scene interpreter employs UNet-like encoder-decoder architecture [5X]] com- prising an instance segmenter for object recognition and an interest net for human interest inference. These two modules share the same point cloud encoder for efficiency but use different decoders, as illustrated in the yellow box in Figure E The shared encoder processes and downsamples the point cloud S into compact feature representation F, R with ni, << Rp (8 I nn I 50 in cases), and e represents the feature dimension. Subsequently, two decoders are employed to segment objects 0 and predict human interests M in the scene, respectively:
(2) Here, O = (0,, O2....... On,,) denotes the set of no segmented objects, each being a subset of the scene pointcloud (O; C S). M E T denotes the per-point interest map, with higher values indicating a greater likelihood of targeting specific scene elements. Once M is obtained, we compute the probability P. of each object O; E O being selected an interaction target:
F. = SceneFncoderr(S) S)
InstanceSSegmentt(((F)) M = InterestNet(F.. X XI..).
M. Σ M //) len(O.), D DEO,, [PI Pg...., Pa,,) = Softmax (MM1, (2,,, (n,
(3)
(4)
where (O,) denotes the number of points in O;, MM is the interest value of point p. and e 0.5 represents the temperature factor that controls randomness in sampling. During inference, we sam- ple the target object (g (1, ..., n,,)) based the probability distribution (Pi, P2,.... Pnn)
In cases where there is no human-object interaction or objects are beyond reach within AT, no explicit target object exists. To handle this, we include the ground as a potential target, voxelized into smaller patches to improve granularity. Each ground patch, treated as an individual object, has a side length of 0.5 meters to balance accuracy and efficiency. The interest scores and interaction probabilities for ground patches are then calculated similarly to other objects, as in Eq. and Eq.Il
Our scene interpreter aligns observed motions with scene context, filtering out less likely engage- ment areas and identifying potential targets. We note that this approach makes the diverse HMP con- trolled by those deterministic elements of the scene, thereby enhancing the physical consistency of predictions. Representing human intention through scene-motion intermodal analysis, the selected target object Og directs the subsequent planning process, as outlined below.
3.3 Behaviorally-Consister Stochastic Planner
Ensuring collision-free and scene-consistent human behavior is crucial but often overlooked, while learning these patterns directly requires impractically large datasets. To tackle these challenges, we employ an action planner to plan obstacle-free trajectories toward the target Og and deterministi- cally predict the interactive human end-pose Lend associated with Og. These intermediate predic- tions serve as stochastic conditional factors, guiding to craft future motions that respect physical constraints and typical
interactions while incorporating motion diversity.
To enable effective navigation and collision avoidance, we utilize scene height map S E = to delineate accessible areas and detect obstacles, inspired by [68, [5]] We first compute obstacle- free trajectories toward the target object Og using modified A algorithm (details on generating
diverse trajectories are in Appendix B) and then employ a single-layer transformer &i to predict per-frame human velocity, sampling discretize points from the planned trajectory:
plan A A X 1LL &end), (+plan .tend) = SSample(pples plan (/(X (x-))).
(5)
Here, plen denotes the continuous trajectory, pplan p sampled discretize trajectory points, and fend is the estimated timestamp for the end of the interactive
242) ( represents the
5

---

motion. Since the length of this trajectory varies across sequences, the human may not reach the target object exactly the prediction horizon AL 5-sec. In these cases, if the target object is too distant to reach within AT. (lend > A.), we truncate the trajectory 7 to fit within A. and adjust the target to the nearest ground patch. Conversely, if the target is reached too early (Lend < AT), we keep the subject relatively static after lend This adjustment ensures that the planned trajectory aligns with the prediction horizon. We then discretize the continuous trajectory to obtain the per- frame global translation palan to guide subsequent motion generation:
pplon =(t
splan 1 .... pplan A.
(6)
In addition to physical constraints, human interactions with specific objects often follow determinis- tic patterns despite potential action diversity. For instance, "set," and "wipe," are reasonable actions for a table, whereas "sit" and "lie" are not. Traditional diverse HMP methods typically overlook these Human-Object Interaction (HOI) patterns, leading motion and scene inconsistencies. Dif- fering from these methods, our approach predicts the target object Og in advance, enabling us to predict the interactive HOI end-pose Lend before the full-sequence prediction:
Bend HOI-Estimatoo((O))
This end-pose represents the final state of the prediction, secures appropriate human interaction with the scene. To this end, our planner constructs an obstacle-free trajectory, and the predicted end-pose (along with lend) stochastic conditional factor ( from icene-motion intermodal perspective:
(Aplon Lend; tend).
(8)
This factor further prompts the motion generator to predict behaviorally consistent motions.
3.4 Seff-Prompted Motion Generator
By constructing stochastic conditional factor @ in advance, DiMoP3D operates as a celf-prompted motion generator, which harmonizes the stochastic factor with deterministic motion generation rooted in 0. To generate diverse predictions that closely align with the predicted conditional fac- tor 0, we utilize a motion diffuser, taking advantage of the diffusion model's ability to effectively guide intermediate results [M22 5% ZTI. Additionally, to further maintain semantic coherence and physical consistency, we propose a semantic alignment inspector to supervise the denoising process.
For simplicity, we denote the sequence at noising step as X Diffusion is modeled as a Markov noising process ( X ) with X drawn from the data distribution, and:
q(x x = N(Vo, ,1 - T))
Here 0g E (0, 1). When or approaches O, we approximate x Xi.++AL ~ (O I), where 0 and represent the zero matrix and the identity matrix, respectively. To effectively integrate scene and motion features in our predictions, our diffusion model MD employs a transformer decoder to model distribution akin to the reversed diffusion process, leveraging its capacity for cross-modal attention. Instead of predicting noise, we predict the clean sample directly, following [63 5:l X3]:
X 0
MD( FF,tt.
(10)
where X represents the intermediate denoising result at each denoising step.
To align the predicted sequence with the observed XI.L. the planned trajectory pplan and the forecasted end-pose Lend at time tend, adjust the corresponding segments after each denoising step. To be sepecific, for each frame 2ạ (C) p!):
x
if S L, if L < L Lends for each frame i.
(Calan
-0 P.PP))
(11)
LLTAL
if i Lends
x!
if é > L + tends
This modified prediction is then noised back before the next denoising step:
(12)

---

where represents the posterior variance based on a at step t. Upon completing T denoising steps, the motion generator yields a cohesive sequence X which integrates smoothly with the observed sequence and aligns with the planned trajectory and goals.
To enhance the consistency between the predicted motion and the target object, we introduce semantic alignment inspector leveraging MotionCIIIP [62]]. It computes HOI semantic loss via natural language descriptions as follows:
L+AL
Seem
(1-cos( Σ
(X MCreae)
(13)
L AL
Here, Motooion MCTeat denote MotionCLIP's motion and text encoders, respectively, with D, signifying the interaction description template related to the class of the sampled target object Og. Acknowledging that HOI predominantly occurs later in motion sequences, our semantic loss formu- lation weights later motion frames more heavily to accurately capture these interactions.
4 Experiment
4.1 Experimental Setup
Dataset-1: GIMO [88], which records motion sequences represented by full-body SMPL-X poses with x 129K frames. consists of scenes with point clouds, each scene captured by a 3D LiDAR sensor, containing 10-20 objects with = 500K vertices. For fair comparison, we follow the official split divide the dataset into training and testing sets according to the scenes.
Dataset-2: CIRCLE [ comprises 10 hours of high-fidelity full-body motion sequences from
subjects across nine apartment scenes. Utilizing Vicon system with 12 cameras 120 FPS and the Al Habitat VR environment for virtual world simulation, CIRCLE achieves precise motion and scene capture. offers an integrated apartment mesh, designating each room as an individual scene. The dataset encompasses motion sequences for 128 tasks, totaling over 7,000 sequences.
We also notice other related datasets [27 28. 500, yet find limitations precluding their use (e.g.. jittering, sequence length, absence of human meshes).
Baselines. Our DiMoP3D is compared with four contemporary methods: DLow [82], SmoothDMP [72], Fus.ou [55], and BiFU [8S]. DLow [82] applies flow network, SmoothDMP [[2]] is VAE- based, and Bel.Fusion [5] is diffusion-basee
which achieves SoTT performance in diverse HMP. These three, however, do not focus on scene-aware diverse HMP, setting BiFU [88], a deterministic scene-aware method, apart as essential control for our analysis.
Metrics. To align with existing literature that evaluates human skeleton metrics, utilize the SMPL model [85] to convert body parameters X 1LL+AL into skeletons J1:L+AL {I1.J2 (i. ILLLL
where each j: Rayxt x3 represents skeleton with n, 22 joints, following [25]]
DiMoP3D is evaluated for diversity, accuracy, and physical consistency in scene-aware predictions. We begin with the well-established pipeline in [82]] Prediction diversity is quantified using the Average Pairwise Distance (APD) by computing the 1.2 distance across predicted sequences. The Average Displacement Error (ADE) measures the reconstruction accuracy among the whole pre- dicted sequence, while the Final Displacement Error (FDE) measures accuracy of the furthest frame, alongside their multimodal counterparts, MMADE and MMFDE, for diverse HMP scenarios.
To measure the physical consistency of the predicted motion within 3D scenes, additional metric, the Average Cumulated Penetration Depth (ACPD) is introduced, following [78, 83
L+LL
ACPD(J)
max ( (SDF o).
(14)
AL
1= L+1 ==1
where j[n] denotes the position of the nith joint in the skeleton, and SDF(-, S) refers to the signed distance function [5T]] of the scene point cloud S. For training details, please refer Appendix [X]
7

---

Table 1:
Comparison of DiMoP3D with baselines GIMO [8 and CIRCLE [ datasets. The best outcomes are highlighted bold. Given that BiFU &S
employs a deterministic prediction
approach, diversity metrics such as APD, MMADE, and MMFDE not applicable.
Method
APD 1
ADB
FDE 4
MMADE
MMFDE
ACPD 4
GIMO [SS]]
Dlow
[2]]
55.12
13.70
16.88
15.96
17.31
14.55
SmoothDMP
68.80
11.17
14.51
13.67
15.42
15.08
Bel.Fusion
[
38.04
9.69
11.19
11.28
12.02
13.73
BiFU |8
7.11
8.39
3.73
DiMoP3D
48.30
5,66
6.81
6,57
7.44
0.98
CIRCLE [
Dlow [SC]]
49.37
11.70
14.46
13.49
14.95
12.52
SmoothDMP
57.38
9.75
12.04
11.31
13.57
13.92
BcLFusion
39.46
7.91
9.30
9,56
10.04
14.06
BiFU 88
5.80
6.99
2.11
DiMoP3D
42.24
5.09
6.12
5.95
6.48
0.87
Table 2: Ablation of four main components in DiMoP3D over the sequences the GIMO [8
Ablation
APD1
ADE
FDE
MMADEJ
MMFDE
ACPD
w/o InterestNet
52.63
6.17
7.48
7.20
8.09
1.00
w/o HOI-Estimator
46.97
5.95
7.27
6.72
7.84
1.53
w/o TrajectoryPlanner
57.29
6.39
6.81
7.28
7.45
3.29
Semanticlnspector
47.79
5.82
6.82
6.75
7.46
1.06
DiMoP3D
48.30
5.66
6.81
6.57
7.44
0.98
4.2
Main Results
Table M demonstrates DiMoP3D's supriioor
the baseline methods across nearly all evaluation metrics on both datasets. The non-scene-aware methods (Dlow, SmoothDMP, Bel.Fusion) exhibit limited motion accuracy (ADE, FDE, MMADE, MMFDE) and physical scene consistency (ACPD), which we hypothesize is due to (I) lack of scene awareness, resulting in notable inconsistency in real-world applications, and (2) the absence of explicit motion goals, which hinders precise long- term (5-sec) motion forecasting. Despite their higher scores in diversity (APD), this is attributed to their erratic and unpredictable predictions, disregarding the scene context (detailed in Sec H.).
DiMoP3D's enhanced performance stems from three key factors: (1) Diversity. The stochastic conditional factor introduces diversity through multiple mechanisms: the intermodal interpreter sets broad motion objectives, the stochastic planner generates a variety of end-poses and trajectories, and the motion generator achieves diverse motion poses. This multi-faceted approach ensures breadth of plausible actions are considered, enabling DiMoP3D to achieve a considerable APD score. (2) Accuracy. DiMoP3D outperforms every baseline in ADE, FDE, MMADE, and MMFDE for large margin, even the deterministic BiFU. By estimating future human action based on a scene-motion intermodal analysis, DiMoP3D implicitly infers the subject's intent. This boosts the probability of accurately identifying the subject's genuine intent as the basis for prediction, thereby improving the prediction precision. The combination accurate intermodal scene interpreting and stochastic planning ensures precise motion prediction for each sequence. (3) Physical consistency. Our motion generator employs a diffusion model, prompted by the predicted stochastic factors. It also ensures motion coherence through priors overwrite at each denoising step. This dual focus on deterministic constraints enables DiMoP3D to achieve superior motion-scene consistency.
This superior performance demonstrates the efficacy of our DiMoP3D in predicting diverse human motion 3D scenes, further evidenced by subsequent ablation studies and visualizations.
4.3 Ablation Studies
In Table we dissect the impact of excluding four pivotal components from DiMoP3D. First, elim- inating InterestNet markedly decreases performance across ADE, FDE, MMADE, and MMFDE (first row). This decline stems from the process of selecting the target object O from O, which reverts to random sampling without scene-motion crossmodal analysis, impairing DiMoP3D's abil- ity to deduce human intentions. Consequently, the accuracy of predicting real actions diminishes, highlighting the significance of integrating multimodal scene-motion analysis for scene-aware HMP.

---

Bedroom
Seminur room
sample-1
sample-2 sample-3 BeLFusion (ICCV23)
sample-4
sample-1
sample-2
sample-3
sample-4
DiMoP3D (ours)
Figure 3: Visual comparisons between DiMoP3D and SoTA BeLFusion in bedroom and semi- nar room scenarios. BeLFusion's predictions, which rely solely on past human motion without considering 3D scene context, are shown the left. In contrast, DiMoP3D, displayed right, incorporates interactive goals and designs obstacle-fre trajectories for each sequence. Pur- ple meshes depict observed motions, while yellow ones signify predicted future motions. For clarity, distortions in BeLFusion's 's predictions are marked: red boxes for object penetration, green boxes for motion incoherence, and yellow boxes for scene inconsistency.
the
Addressing the role of stochastic planner, its absence undermines the planning of actions, including the prediction of end-poses by the HOI-Estimator and trajectory planning via the A Trajectory Plan- ner. Without these two components, the motion generator struggles to predict end-poses or future trajectories with scene consistency. Notably, omitting the HOI-Estimator results in imprecise in- teractive end-poses, often causing the subject to intersect with the target object in later frames, as evidenced by increased ACPD and reduced FDE and MMADE (second row). Similarly, excluding the TrajectoryPlanner significantly elevates ACPD (third row), indicating frequent subject penetra- tions into the scene context while approaching the end-pose. These findings underscore the vital role of coordinated end-pose and trajectory prediction in predicting motion within 3D scenes effectively.
Finally, the Semantic Inspectrr enhances the scene-motion alignment through natural language, with its omission resulting in higher ADE and MMADE. Please refer to Appendix D for further ablations.
4.4 Visualization
To delve deeper into DiMoP3D, in Figure EL we showcase DiMoP3D's predictions across two narios, contrasting them with the SoTA baseline Bel.ouioo [5]].
In bedroom scenario, the subject stands still behind the door. BeI.uuioon' predictions show no- table issues with object and wall penetrations. Furthermore, sample-1 and are marked by abrupt, illogical movements, and sample-3 and 4 display glaring scene inconsistencies: sample-3 has the subject sitting on the bare floor, and sample-4 involves the subject reaching for non-existent items. Conversely, DiMoP3D ensures physical consistency, directing each prediction towards specific movement goal: opening a window, lying on the bed, accessing a cabinet, and sitting on a chair.
In the seminar room scene, the subject is moving forward. BeLFusion struggles again, with sample- 1, 3. and 4 depicting the subject unrealistically exiting the room, and sample-2 showing an inconsis- tent motion of picking up a curtain. DiMoP3D, however, delivers high-fidelity predictions, depicting the subject walking through a door, grasping items, sitting, and pulling down curtains, respectively.
To emphasize DiMoP3D's predictive diversity, we further visualize various end-poses generated by the HOI-Estimator in Figure El Overall, DiMoP3D consistently delivers diverse, realistic, and thysically-consisee motion predictions with clear objectives, benefiting from our conditional factor prediction schema which models human-object interactions and navigates obstacle- trajectories.

---

(a) Chair
(e)
Table
Figure 4: Visualizations of diverse predicted
across five object point clouds. The HOI- Estimator can generate variety of human-object interactive poses tailored to specific scenarios.
Table 3: Comparison of DiMoP3D with scene-aware motion synthesis methods.
Method
APD
ADE
FDE + MMADE
MMFDE
FID 4
ACPD +
SAMP [28]
31,73
9.83
9.28
11.16
10.13
1.493
1.69
DN DN-Synt [68]
44.60
9.71
7.16
11.29
7.83
1.026
1.21
AffordMotion [+]]
52.54
8.96
8.14
10.38
8.95
0.687
1.26
DiMoP3D
48.30
5.66
6.81
6.57
7.44
0.769
0.98
4.5 Compared with Motion Synthesis Methods
In this section, we compare our DiMoP3D with three scene-aware motion synthesis approaches on GIMO [88]: SAMP [28]. DN-Synt [68], and AffordMotion [ZZ]]. SAMP and DN-Synt utilize VAE architectures, while AffordMotion employs diffusion-basse model.
To adapt these methods for our diverse HMP task, we initialize motion synthesis from the last observed frame æy and encode the complete observed sequence X 1:L into unified embedding for historical motion conditions using a 2-layer transformer encoder, similar to the embedding technique in [57]]- Additionally, introduce the FID metric, commonly used in motion synthesis, to evaluate the discrepancy between the distributions of generated and original dataset motions.
The results in Table D reveal an intriguing pattern: synthesis methods exhibit higher ADE than FDE. This occurs because, although these methods include explicit end-pose estimators yielding accurate final pose predictions, they struggle to condition on past motion. Consequently. they produce motion incoherence and significant prediction errors along the trajectory from the observation the final pose. Notably, AffordMotion achieves the best APD and FID scores, which attribute to its design that prioritizes fidelity over accuracy and allows a higher degree freedom. Meanwhile, DiMoP3D also demonstrates competitive performance in these metrics. These findings underscore DiMoP3D's capability to harmonize the stochastic nature of human motion and the deterministic constraints from the scene and the past motion, achieving superior performance in diverse scene-aware HMP, and maintaining competitive diversity and fidelity even when compared to SoTA synthesis method.
Conclusion and Limitation
This work introduces novel task of predicting diverse human motion in 3D scenes, along with novel framework, DiMoP3D, to address it. By incorporating multimodal motion-scene analysis, DiMoP3D identifies areas or objects the subject is likely to interact with, enabling diverse, accurate, and physically consistent human motion prediction. Evaluated on the GIMO and CIRCLE datasets, DiMoP3D reduces ADE and FDE by nearly half compared to the state-of-the of-the-art baseline BeLFu- sion, while maintaining high physical consistency. These results underscore the importance of scene awareness in diverse human motion prediction for real-world applications.
Despite its strong performance, DiMoP3D predicts motion in a fixed sequence length. When the actual sequence length differs, it either keeps the subject relatively static truncates the sequence. Future work could explore variable-length motion prediction or end-to-end prediction, where the motion generator predicts sequence length and generates motion simultaneously.
10

---

Acknowledgements
This work was supported in part by the National Key Research and Development Program of China (2022YFC3602011) in part by the Natural Science Foundation of Jiangsu Province (BK20220939), and in part by the National Natural Science Foundation of China (62306141).
References
[1] E. Aksan, M. Kaufmann, P. Cao, and O. Hilliges. A spatio-temporal transformer for 3d human motion prediction. In 3DV, pages 565-574. IEFF, 2021.
[2] S. Aliakbarian, S.S. Salch, M. Salzmann, L. Petersson, and S. Gould. A Stochastic Conditioning Scheme Diverse Human Motion Prediction. In CVPR, pages 5223 5232,, 2020.
[3] S. Aliakbarian, Saleh, Petersson, Gould, M. Salzmann. Contextually Plausible Diverse
3D Human Motion Prediction. ICCV, pages 11333-11342, 2021.
[4] Araújo, Li, K. Vetrivel, R. Agarwal, Wu, D. Gopinath, A. W. Clegg, and K. Liu, Circle: Capture in rich contextual environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21211-21221, 2023.
[5] G. Barquero, S. Escalera, and C. Palmero. Belfusion: Latent diffusion for behavior-driven human motion prediction. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 2317- 2327, 2023.
[6] E. Barsoum, J. Kender, and Z. Liu. HP-GAN: Probabilistic 3D Human Motion Prediction via GAN, In CVPR, pages 1418-1427 2018.
[7] D. E. Benrachou, S. Glaser, M. Elhenawy, and A. Rakotonirainy. Use of social interaction and intention to improve motion prediction within automated vehicle framework: A review. IEEE Transactions Intelligent Transportation Systems, 21(1))22807-22237 2022.
Z. Cao, H. Gao, K. Mangalam, Q.Z Cai, M. Vo, and Malik. Long-term human motion prediction with seene context. Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part I 16, pages 187-404. Springer, 2020.
[9] A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Niessner, M. Savva, S. Song, A. Zeng and Y. Zhang Matterportåd: Learning from rgh-d data in indoor environments. arXiv preprint o 1709.061555 158, 2017.
[10] P. Chang, J. Dang, Dai, and W. Sun. Real-time respiratory tumor motion prediction based temporal convolutional neural network: Prediction model development study. Journal of Medical Internet Research, 23(8):27235, 2021.
[11] Y.W.. Chao, Y. Liu, X. Liu, H. Zeng, and Deng. Learning to Detect Human-object object Interactions. In WACV, pages 381-389 IEFF, 2018.
[12] L.-H. Chen, J. Zhang, Y. Li, Y. Pang, X. Xia, and T. Liu. Humanmac: Masked motion complction for human motion prediction. arXiv preprint arXiv:2302.03660 2023.
[13] S. Chen, J. Fang, Q. Zhang, W. Liu, and X. Wang. Hierarchical aggregation for 3d instance segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15467-15478 15476. 2021.
[14] S. Chen, C. Ge, Tong, Wang. Song, Wang, and P. Luo. Adaptformer: Adapting vision transform- ers for scalable visual recognition. Advances in Neural Information Processing Systems, 35:16664-16677 2022.
[15] H.-K. Chiu, E. Adeli, B. Wang, D.-A. Huang, and J. C. Niebles. Action-Agnostic Human Pose Forecasting. WACV, pages 1423-1432, 2019.
[16] P. Cong, Z. W. Dou, Y. Ren, W. Yin, Cheng, Y. Sun, Long, X. Zhu, and Y. Ma. Laserhu- man: Language-guns guided scene-aware human motion generation in free environment. arXiv preprint arXiv:2403.13300 2024.
[17] E. Corona, A. Pumarola, G. Alenya, and F. Moreno-Noguet Context-aware Human Motion Prediction. In CVPR, pages 6992-7001, 2020. 2020.

---

[18] Q. Cui and H. Sun. Towards accurate 3d human motion prediction from incomplete observations. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4801- 4810, 2021.
[19] Q. Cui, H. Sun, and F. Yang. Learning Dynamic Relationships for 3D Human Motion Prediction. In CVPR, pages 6519-6527, 2020.
[20] Q. Cui, H. Sun, Y. Kong, X. Zhang, and Y. Li. Efficient human motion prediction using temporal convo- lutional generative adversarial network. Information Sciences, 545:427-447, 2021.
[21] A. Dai, A. X. Chang. M. Savva, M. Halber, T. Funkhouser, and M. Nicßner. Scannet: Hichly-annotate 3d reconstructions indoor scenes. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2017.
[22] L. Dang. Y. Nie. C. Long. Q. Zhang, and G. Li. MSR-GCN: Multi-Scale Residual Graph Convolution Networks for Human Motion Prediction. In ICCV, pages 11667-11776, 2021.
[23] J. Deng, S. Shi, P. Li, W. Zhou, Y. Zhang, and H. Li. Voxel r-cnn: Towards high performance voxel-based 3d object detection. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 1201-1209, 2021.
[24] G. Gkioxari, R. Girshick. Dollár, and K. He. Detecting and Recognizing Human-object Interactions. In CVPR, pages 8359 8067, 2018.
[25] C. Guo, S. Zou, X. Zuo, S. Wang. W. Ji, X. Li, and L. Cheng. Generating diverse and natural 3d hu- man motions from text. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5152-5161, 2022.
[26] M.-H. Guo, J.-X. Cai, 7.-N. Liu, T.JJ Mu, R. R. Martin, and S.-M. Hu. Pet: Point cloud transformer. Computational Visual Media, 7:187-199, 2021.
[27] M. Hassan, Choutas, D. Tzionas, and M. J. Black. Resolving 3d human pose ambiguitics with 3d scene constraints, In Proceedings of the IEEE/CVF international conference on computer vision, pages 2282-2292, 2019.
[28] M. Hassan, D. Ceylan, R. Villegas, J. Saito, J. Yang. Y. Zhou, and M. Black. Stochastic scene-aware motion prediction. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 11374-1 11384, 2021.
[29] S. Huang, Z. Wang, P. Li, B. Jia, T. Liu, Y. Zhu, W. Liang, and S.-C. Zhu. Diffusion- generation, optimization, and planning in 3d seenes. In Proceedings of the IEEE/CVF IEnE/CCFConfereecc on Computer Vision and Pattern Recognition, pages
2023.
[30] D. K. Jain, M. Zareapoor, R. Jain, A. Kathuria, and S. Bachhety. Gan-poser: improvised bidirectional gan model for human motion prediction. Neural Computing and Applications, 32(18):14579 14591, 2020.
[31] L. Jiang, H. Zhao, S. Shi, S. Liu, C.-W. Fu, and J. Jia. Pointgroup: Dual-set point grouping for 3d instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and Pattern recognition, pages 4867 4876, 2020.
[32] R. Jiang. Z. Cai. Z. Wang. C. Yang. Z. Fan, Q. Chen, K. Tsubouchi, X. Song, and R. Shibasaki. Deep- crowd: A deep model for large-scale citywide crowd density and flow prediction. IEEE Transactions on Knowledge and Data Engineering, 35(1):276-290, 2021.
[33] P. Karle, M. Geisslinger, J. Betz, and M. Lienkamp. Scenario understanding and motion prediction for autonomous vehiclesreview and comparison. IEEE Transactions Intelligent Transportation Systems, 23(10):16962 16982, 2022.
[34] D. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint 13Xiv:1312.6114, 2013.
[35] M. Kolodiazhnyi, A. Vorontsova, A. Konushin, and D. Rukhovich. Oneforme33d: One transformer for unified point cloud segmentation. arXiv preprint arXiv:2311. 2023.
[36] X. Kong, K. Wang, M. Hou, F. Xia, G. Karmakar, and J. Li. Exploring human mobility for multi- passenger prediction: A graph learning framework. IEEE Transactions on Intelligent Transportation Systems, 1319):66188-16 2022.
12

---

[37] N. Kulkarni, D. Rempe, K. Genova, A. Kundu, J. Johnson. D. Foubey, and L. Guibas. Nifty: Neural object interaction fields for guided human motion synthesis. arXiv preprint arXiv:2307.07511 2023.
[38] J. N. Kundu, M. Gor, and R. V. Babu. BiHMP-GAN: Bidirectional 3D Human Motion Prediction GAN.
In AAAI, volume 33, pages 8553-8560, 2019.
[39] M. Li. S. Chen, Y. Zhao, Y. Zhang. Y. Wang, and Q. Tian. Dynamic Multiscale Graph Neural Networks for 3D Skeleton Based Human Motion Prediction. In CVPR, pages 214-223, 2020.
[40] M. Li, S. Chen, Z. Zhang. L. Xie, Q. Tian, and Y. Zhang. Skeleton-Parde Parted Graph Scattering Networks for 3D Human Motion Prediction. In ECCV, pages 18 36. Springer, 2022.
[41] S. Li, K. Wu, C. Zhang, and Y. Zhu. On the learning mechanisms in physical reasoning. Advances Neural Information Processing Systems, 2222222222262 2022.
[42] H. Lin, W. Zou, T. Li, S. Feigenberg, B.-K. K. Teo, and L. Dong. A super-learner model for tumor motion prediction and management in radiation therapy: development and fcasibility evaluation. Scientific reports, 91): 1466,, 2019.
[43] Y. Liu. J. Zhang, L. Fang. Q. Jiang, and B. Zhou. Multimodal motion prediction with stacked transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7577- 7586, 2021.
[44] Z. Liu, K. Lyu, S. Wu, H. Chen, Y. Hao, and S. Ji. Aggregated multi-gans for controlled 3d human motion prediction. In AAAI, volume 35, pages 2225 2232,22222222222 2021.
[45] M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black. Smpl: A skinned multi-persee linear model. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pages 851-866. 2023.
[46] Z. Lou, Q. Cui, Wang, X. Tang, and H. Zhou. Multimodal sense-informed forecasting of 3d human motions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2144-2154, 2024.
[47] S. Lu, L.-H. Chen, A. Zeng. J. Lin, R. Zhang, L. Zhang, and H.-Y. Shum. Humantomato: Text-aligned whole- bode--bdyydde motion generation. arXiv preprint arX1::2310.12978, 2023.
[48] J. Mao, Y. Xue, M. Niu, H. Bai. J. Feng. X. Liang. H. Xu. and C. Xu. Voxel transformer for 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3164- 3173, 2021.
[49] W. Mao, M. Liu, M. Salzmann, and H. Li. Learning Trajectory Dependencies for Human Motion Predic- tion. In ICCV, pages 9489-9497, 2019.
[50] W. Mao, R. I. Hartley, M. Salzmann, et al. Contact-aware human motion forecasting. Advances in Neural Information Processing Systems, 35:7356-7367, 2022.
[51] Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages (65-1744 2019.
[52] H. Peng, B. Li, W. Xiong, W. Hu, and R. Ji. Rgbd salient object detection: A benchmark and algorithms. In Computer EssionEECVV 2014 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part III 13, pages 92-109. Springer, 2014.
[53] X. Peng. Y. Xie, Z. Wu, V. Jampani, Sun, and H. Jiang. Hoi-diff: Text-driven synthesis 3d human- object interactions using diffuusio models. arXiv preprint arXiv:2312.06553, 2023.
[54] A. Piergiovanni, A. Angelova, A. Toshev, and M. S. Ryoo. Adversarial Generative Grammars for Human Activity Prediction. In E.CVV, pages 507-523. Springer, 2020.
[55] C. R. Qi, H. Su, K. Mo, and L. J. Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660,72 2017.
[56] G. Qian, Y. Li, H. Peng. J. Mai, H. Hammoud, M. Elhoseiny, and B. Ghanem. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. Advances in Neural Information Processing Systems, 35:23 192-22444,,44 2022.
13

---

[57] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin. J. Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, PMLR, 2021.
[58] O. Ronneherger, Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmenta- tion. In Medical image computing and computer-assisted intervention-MICCA. 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings. part III 18, pages 234 231-241. Springer. 2015.
[59] S. Saadatnejad, A. Rasekh, M. Mofayezi. Y. Medghalchi, S. Rajabzadeh, T. Mordan, and A. Alahi. generic diffusion-based approach for 3d human pose prediction in the wild. 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 8246-8253. IEEE, 2023.
[60] T. Sofianos, A. Sampieri, L. Franco, and F. Galasso. Opace-Time-Seprrbble Graph Convolutional Network for Pose Forecasting. In ICCV, pages 1 1209-11218, 2021.
[61] J. Sung. C. Ponce, B. Selman, and A. Saxena. Human activity detection from rghd images, plan, activity, and intent recognition, 64, 2011.
[62] G. Tevet, Gordon, Hertz, A. Bermano, and Cohen-Or. Motionclip: Exposing human motion generation to clip space. In European Conference on Computer Vision, pages 358 374. Springer, 2022.
[63] G. Tevet, S. Raab, B. Gordon, Y. Shafir, D. Cohen-Or, and A. H. Bermano. Human motion diffusion model. arXiv preprint arXiv:2209, 14916, 2022.
[64] J. Tseng, R. Castellon, and K. Liu. Edge: Editable dance generation from music. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 448-458, 2023.
[65] T. Vu, K. Kim, T. M. Luu, T. Nguyen, and C. D. Yoo. Softgroup for 3d instance segmentation on point clouds. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2708-2717, 2022.
[66] H. Wang. J. Dong, B. Cheng, and J. Feng. PVRED: A Position- Velocity Recurrent Encoder- Decoder for Human Motion Prediction. IEEE Transactions on Image Processing, 30:6096-6106, 2021.
[67] J. Wang. S. Yan, B. Dai, and D. Lin. Scene-aware generative network for human motion synthesis. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 12206- 12215, 2021.
[68] J. Wang, Y. Rong, J. Liu, S. Yan, D. Lin, and B. Dai. Towards diverse and natural scene-aware 3d human motion synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20460-20469, 2022.
[69] Z. Wang, Y. Chen, T. Liu, Y. Zhu, W. Liang, and S. Huang. Humanise: Language-conditione human motion generation in scenes. Advances Neural Information Processing Systems, 1111499999999 2022.
[70] Z. Wang. Y. Chen, B. Jia, P. Li, J. Zhang. J. Zhang, T. Liu, Y. Zhu, W. Liang, and S. Huang. Move as you say, interact as you can: sagguage-guided human motion generation with scene affordance, arXiv preprint 240: 100..88030 18036, 2024.
[71] D. Wei, H. Sun, B. Li. J. Lu, W. X. Sun, and S. Hu. Human joint kinematics riffussion-refinmment for stochastic motion prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 61 6110-6118, 2023.
[72] M. S. Wei Mao, Miaomiao Liu. Generating Smooth Pose Sequences for Diverse Human Motion Predic- tion. In ICCV, pages 489, 2021.
[73] J. Won, D. Gopinath, and J. Hodgins. Physics-hased character controllers using conditional vaes. ACM Transactions on Graphics (TOG), 41(4):1-12, 2022.
[74] Z. Wu, Song, A. Khosla, Yu, L. Zhang, X. Tang, and J. Xiao. 3d shapenets: deep representation for volumetric shapes. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912-1920, 2015.
[75] Z. Xiao, T. Wang, J. Wang, Cao, W. Zhang, B. Dai, D. Lin, and J. Pang. Unified human-scene interaction arXiv preprint 20X,99230,.07918, 2023.
via prompted
14

---

[76] C. Xu, R. T. Tan. Y. Tan, S. Chen, X. Wang, and Y. Wang. Auxiliary tasks benefit 3d skeleton-based human motion prediction. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 9509-9520, 2023.
[77] D. Xu, Y. Zhu, C. B. Choy, and L. Fei-Fei. Scene graph generation by iterative message passing. In Proceedings of IEEE conference on computer vision and pattern recognition, pages 5410-5419, 2017.
[78] S. Xu, Z. Li, Y.-.. Wang, and L.-Y. Gui. Interdiff: Generating 3d human-object interactions with physics- informed diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14928- 14940, 2023.
[79] Yang. J. Lu, Lee, D. Batra, and D. Parikh. Graph r-enn scene graph generation. In Proceedings of the European conference on computer vision (ECCV), pages 670-685, 2018.
[80] X. Yu. L. Tang, Rao. Huang. Zhou, Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19313-19322, 2022.
[81] Z. Yu, S. Kim, R. Mallipeddi, and M. Lee. Human intention understanding based on object affordance and action classification. In 2015 International Joint Conference on Neural Networks (IJCNN), pages 1 1-6. IEEE, 2015.
[82] Y. Yuan and K. Kitani. Dlow: Diversifying Latent Flows for Diverse Human Motion Prediction. In ECCV, pages 346-364, 2020.
[83] Y. Yuan, J. Song, U. lqbal, A. Vahdat, and J. Kautz. Physdiff: Physics-guided human motion diffusion model. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 16010- 16021, 2023.
[84] R. Zellers, M. Yatskar, S. Thomson, and Y. Choi. Neural motifs: Scene graph parsing with global context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5831 5840, 2018.
[85] Zhang. Cai, L. Pan, F. Hong, X. Guo, L. Yang, and Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[86] X. Zhang, B. L. Bhatnagar, S. Starke, V. Guzov, and G. Pons- Poll-Moll. Couch: Towards controllable human- chair interactions. In European Conference on Computer Vision, pages 518-535. Springer, 2022.
[87] H. Zhao, L. Jiang, J. Jia, H. Tom, and Koltun. Point transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages
2021.
[88] Y. Zheng, Y. Yang, K. Mo, J. Li, T. Yu, Y. Liu, C. K. Liu, and L. J. Guibas. Gimo: Gaze-informed human motion prediction in context. In European Conference on Computer Vision, pages 676-694. Springer, 2022.
[89] C. Zhong, L. Hu, Z. Zhang, Y. Ye, and S. Xia. prediction. In CVPR, pages 6447-644 6456, 2022.
gating-adjacenny gen for human motion
[90] W. Zhu, X. Ma, D. Ro, H. Ci, J. Zhang, J. Shi, F. Gao, Q. Tian, and Y. Wang. Human motion generation:
A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
15

---

A
Traning Details
Training DiMoP3D encompasses separate phases for InterestNet, HOI-Estimator, and the self- prompted motion generator. This section delineates the specific training methodologies for each component. All training is conducted on a single NVIDIA RTX3090 GPU, with the complete pipeline converging in 8 hours.
Training InterestNet. We adopt the ScanNet [2a|] pretrained SoftGroup model as our encoder and segmenter to enhance performance, and transformer decoder for the interest net to achieve cross- modal analysis. The original datasets [ 88] lack annotations for the interest map M; hence, we enrich them with such annotations for each motion clip. InterestNet is employed to elucidate the relationship between the observed motion and the scene, facilitating the prediction of human inter- ests, including likely destinations and objects of interaction. To ensure the interest map accurately represents these aspects, we annotate the human interest map M based on three critical factors: the contact area between humans and objects (M cone), the proximity of humans to scene elements ( (M dist), and the spatial relationship between each object O, and the target object Og g obj))):
M con't = fN((0.0)) (Dist3D(S, Peont)),
(15)
M Mdist
1
L+AL
Σ
(1- 1.)
(N(0,0.) (DistID(S, &)),
(16)
AL
(AL)2
1=++1
M obj
Σ
(Moo.,)DiisD(((,,, 0g)) O.,
(17)
O,EO
M Xeont M cont + Adist dist + MobjMabb)
(18)
Here, Poont denotes the human-object contact position, [N(,..) denotes the Gaussian function with mean and standard deviation a. Smaller a concentrates the interest map while larger a distributes attention more broadly. We set 0 = 0.3 and 02 dg = 1.0 for balance. D-st3D(-) and Dist2D(.) are the 3D and X-Z 2D distance functions (Y-axis denotes height). Hyperparameters Aoont = 3, Adist 10, Aobj = 1 are adjusted to maintain a balance among the factors.
Upon annotating the interest map M. InterestNet is trained using a KL divergence loss to minimize the discrepancy between the distributions of the predicted interest map and the annotated M:
Liserees KL InterestNet( 111)))
(19)
Training HOl-Estimator. The HOI-Estimator employs an autoregressive conditional variational autoencoder (cVAE) [[44] architecture, designed to predict a range of feasible end-poses. It predicts interactive end-poses &end based on the target object's point cloud Og. Initially, we centralize the object on the X-Z plane (assume the Y-axis denotes height) by normalizing its position to the origin:
9
Norm =0,
(On
1
Σ pEO,
Pox:
(20)
g)
where len(O,) indicates the point count in O, and Pes is the X-Z coordinates of point p. The HOI-Estimator utilizes a conditional Variational Autoencoder (cVAE) [G2]] for encoding-decoding
μ1, Encoder( N Norm X1.L),
(21)
9
Band Decoder O Norm μ. o).
(22)
The HOI-Estimator is trained to minimize the L2 loss between the reconstructed interactive pose Lend and the ground-truth Vend:.
CHOI @[@end Fendllé 1
(23)
Training motion generator. The motion generator is pre-trained on the HumanML3D dataset [25]]
We predict the clean sample x directly instead of predicting noise following 63 nel Se
consist-
ing of the basic diffusion loss:
Lrase |
X 2
(24)
16

---

Table 4: Description template of semantic inspector among 18 objects in the ScanNet dataset [2]]]
[OBJ] ground cabinet bed sits
[ACT]
walks on opens
stands on searches lies
reaches hands to
picks things from
chair sits on sofa sits on table sets door opens opens
wipes
reaches hands to reaches hands to reaches hands reaches hands to reaches hands to reaches hands to reaches hands to reaches hands reaches hands to reaches hands to reaches hands to reaches hands reaches hands reaches hands to
picks things from passes
closes
window
closes
bookshelf
arranges
leans on takes down leans on organizes closes
picks things from
picture hangs
counter
wipes
desk
wipes
picks things from
curtain
opens
refrigerator
opens
closes
picks things from
shower curtain
opens
closes
toilet
sits on washes flushes
flushes
sink
picks things from takes shower in interacts with
bathtub
other furniture
uses
checks
supplemented by the semantic alignment loss (detailed in Section 3.4 the main paper), forming the total training loss for motion generator:
Laitt
Coase + Lsem-
(25)
Description Template of Semantic Inspector.
The semantic inspector incorporates language de- scription to supervise the predicted motion to be consistent with target objects. To enable this super- vision, design language description template for each class of objects in the format of:
The person walks forward then [ACT] the [OBJ],
(26)
where [ACT] and [OBJ] placeholders for the action and object, respectively. There total of 18 classes of objects (excluding wall and floor) in the pre-trained ScanNet dataset [2I]], and we design corresponding [ACT] for each class, detailed in Table !!
B Modified A Trajectory Planner
To facilitate diverse trajectory prediction, we enhance the conventional A trajectory planner, allow- ing for the iterative generation of valid paths while penalizing previously traversed positions.
Initially, generate the scene height map S described in Alg. M To streamline height analysis and A pathfinding, we voxelized the scene point cloud along the X and Z axes (with the Y-axis representing height) into grid of 0.02-meter resolution. We then identify the maximum height within each grid cell, omitting ceilings and tall cabinets, which, despite their height, do not impede movement and would otherwise be inaccurately marked obstacles.
Leveraging the generated height map SH, we outline our modified dynamic A trajectory planner in Alg. C. Our method transcends traditional one-hot encoding for marking obstacles by enabling navigation over lower barriers through a continuous cost function derived from S We adopt power function to model the cost, with cell height acting as the exponent. To ensure smoother trajectories and reduce sudden changes, an L2 penalty on angular velocity is integrated into the cost framework. Additionally, to prevent the selection of repetitive paths, increase the cost of cells once traversed. The modified A algorithm then iteratively generates paths until the cumulative cost exceeds predefined threshold or the maximum path count is attained, in Figure 5L
C Additional Ablation Studies
Scene feature for baselines. To assess the influence of scene features baseline methods, we con- duct supplemental experiments integrating scene features into these methods. Despite the baseline models mentioned in the main paper M g not inherently accommodating scene features,
17

---

Algorithm/ set_hhightmap(S)::
S: Scene point cloud with shape (Np, 3), where Np is the point count
invalid -10000, grid 0.02, h 1.2
Sg VoxeIGridX2(s, grid)
# x.Z coordinates are gridded, while is not (Sg.maxz Sg.minz/grid) * invalid
H (aa.assg.max-S.mmmx//eaee
For p in Sg:
If p-y h_tt::
# Excluding the ceiling and high cabinets
x, 7= Sg.grid(p..) p.z)
return H[x, z] = max z). p.y)
Algorithm 2 dynamic aStar(X, d, H):
X: Observed motion trajectory with shape (I., 2), where is the input length
d: Destination position with shape (2)
H: Scene height map with shape (lenX, lenZ.
def cost_ffnncion (assfffuccioonlasttoss,,, p. path): return lastcost C[p] angular angular_vel(path) (pa(h))h)
invalid 10000, grid 0.02, h th 1.2, noise 1.0 costbase 1000, costth 1000, costinc = 10 H[H==invalid] h th costbase H
paths = []
For npath in range(MAX PATHS): X[-1], d, cost_ ffun_funn
path, cost = astar(C
If cost > mincost costth: break
C[path] += costinc
paths attppen/eteseesmmottnnng//////oo
return paths
Table 5: Results of appending scene features for the baseline methods on GIMO [
Method
APD1
ADE
FDEL
MMADEJ
MMFDEJ
ACPD
Dlow [S2]
55.12
13.70
16.88
15.96
17.31
14.55
SmoothDMP TA
68.80
11.17
14.51
13.67
15.42
15.08
BeLFusion
38.04
9.69
11.19
11.28
12.02
13.73
Dlow [X1]
SmoothDMP [[]]]
66.36
11.02
14.49
13.43
15.37
14.71
BeLFusion [S]
36,89
9.55
11.04
11.12
11.76
13.52
DiMoP3D
48.30
5.66
6.81
6.57
7.44
0.98
augmented their input by appending the feature F, along the temporal dimension for [ 821. and along the joint dimension for [722], to explore potential performance.
The results in Table E reveal that Dlow failed to convergence, likely attributed to its recursive ar- chitecture, wherein the concatenated scene features diverge significantly from the distribution of motion features, making the network difficult to learn. Both SmoothDMP and BeLFusion exhibit marginal improvements, suggesting that the direct integration of scene features into the motion gen- erator yields limited effectiveness. Conversely, these results underscore the efficacy of DiMoP3D's strategy of predicting the conditional factor, highlighting its superiority harmonizing the scene and the observed motion.
Ablation on the scene segmenter. To evaluate DiMoP3D's robustness across various point cloud instance segmentation methods, we performed additional experiments with different segmentators. Higher-quality segmenters yield more precise object delineations, enabling the selection of more accurate targets and reducing object boundary violations. Table R illustrates that DiMoP3D main-
18

---

Table 6: Results of DiMoP3D with various segmenters. "mAP50" denotes the mean average precision at 50 IoU threshold for each segmenter on the ScanNetv2 dataset [21]. Higher "mAP50" represents better segmenter performance.
Segmentator
mAP50
APD1
ADE
FDEL
MMADE
MMFDEL
ACPD
PointGroup [[1
63.6
46.22
5.70
6.93
6.63
7.56
1.05
HAIS [033]]
69.9
46.97
5.67
6.87
6.61
7.50
1.02
SoftGroup [65]
76.1
48.30
5.66
6.81
6.57
7.44
0.98
Table 7: Results of DiMoP3D with various trajectory planners. Trajectory planner has no effect on FDE and MMFDE they are end-pose errors, so omit them in this table.
Planner
APD1
ADE
MMADE
ACPD1
Naive A
39.26
5.93
6.82
0.98
ours continuous cost
45.59
5.77
6.72
0.98
ours w/o angular penalty
50.17
5.91
6.88
0.97
ours
48.30
5.66
6.57
0.98
tains stable performance even when the quality of the segmentation method declines, underscoring its robustness to variations in segmentation.
Ablation on the trajectory planner. To validate the significance of our enhanced A* trajectory plan- ner, we conducted an ablation study. Results presented Table D show that the naive A* method underperforms due to its deterministic nature. Excluding the continuous cost model results in binary scene representation, disregarding traversable lower obstacles and thus reducing path diver- sity. Additionally, omitting the angular velocity penalty leads to paths with abrupt turns, detracting from prediction accuracy. These outcomes significant the critical role of our modified A* trajectory planner in achieving nuanced and reliable path predictions.
D More Visualizations
Instance segmentation and interest map. Exploring multimodal scene parser, we present visualizations of 3D scene instance segmentation alongside the corresponding interest maps M for three distinct samples. Figure E (upper) showcases the segmented instances within the scene, while Figure (lower) evidences our multimodal InterestNet's capacity to deduce potential human intentions.
In the bedroom setting, with the person initially stationary and then beginning to move forward, the future action remains ambiguous. Consequently, almost all objects are highlighted as potential targets in M, except for the door situated behind the individual. In the living room scene, as person navigates the narrow gap between a chair and table, the sofa, an additional chair, and the table emerge as points of interest. Conversely, the proximate chair and a distant door garner lesser attention, aligning with the observed motion pattern. A similar observation is noted in laboratory environment.
These results underscore our multimodal scene parser's adeptness at inferring potential human in- tentions, and the generated interest map M can accurately reflect anticipated human interests and interactions.
Multiple samples with single target. To further explore DiMoP3D's capacity for predicting diverse motions toward a deterministic target object, additional visualizations are presented.
The left panel of Figure @ showcases predicted sequences where the person navigates different paths to reach the destination, performing varied actions such as looking down and reaching toward the target. The right panel illustrates sequences of the person adopting different positions for lying and sitting on the bed. These results affirm DiMoP3D's adeptness at generating varied and coherent human motions aimed at specific target object, further ensuring diversity.
Comparison with synthesis methods. To further explore the performance synthesis methods in diverse scene-aware HMP tasks, we present visual comparisons between DiMoP3D and the SoTA synthesis method AffordMotion Figure AffordMotion's predicted sequences tend to remain relatively static the onset of prediction, which attribute to its method of smoothly synthesiz-
19

---

destination point
starting point
starting point
starting point
destination point
destination point
Visualization samples of the modified A* trajectory planner. Black lines denote the Figure 5:
observed trajectory, while colored lines represent the generated paths.
Instance
Segmentation
Interest
Map
bedroom
living room
laboratory
Figure 6: Visualization of 3D scene instance segmentation (upper) and the corresponding interest map (lower). Red points in the interest map denote higher interest, while blue points denote lower interest. Leveraging the insight provided by the predicted interest map enables the exclusion of im- probable or illogical targets, thereby enhancing the reliability and scene congruency of predictions.
ing human motion irrespective of prior motion states, resulting in significant errors along the paths. Particularly in samples and 4, AffordMotion shows notable motion incoherence, characterized by abrupt changes the transition between prediction and observation. In contrast, DiMoP3D con- siders past motion, predicting coherent and plausible sequences, thereby demonstrating its superior performance in diverse scene-aware HMP tasks. While motion synthesis methods show promise in handling diverse scene-aware HMP, they face the critical challenge of integrating deterministic cues from past motions effectively.
E Potential Broader Impacts
The proposed DiMoP3D introduces novel diverse scene-aware motion prediction framework, which may involve the following broader impacts:
20

---

side sieee
laboratory, otherfurniture
bedroom, bed
Figure 7:
Visualization of multiple samples with fixed target object. DiMoP3D able predict motions with diverse trajectories and actions (or end poses) toward a deterministic target object, while maintaining each motion sequence to be consistent with the observation and the scene.
AffordMotion (CVPR24)
DiMoP3D (Ours)
sample-1
sample-2
sample-3
sample-4
Figure 8: Visualization comparisons between DiMoP3D and SoTA synthesis method AffordMotion.
The results from AffordMotion demonstrate significant motion incoherence.
Enhanced Safety in Robotics and Automation. DiMoP3D improve the interaction between humans and robots. By predicting human motions accurately, robots can avoid col- lisions and unsafe interactions, making environments safer for both humans and machines.
Improved VR and Gaming Experiences. In VR and video games, this framework can lead more realistic and responsive interactions with virtual characters and environments. By understanding and predicting how human might move within a scene, VR systems can offer more immersive and natural experiences, enhancing user engagement and satisfaction.
Advancements in Assistive Technologies. For people with disabilities the elderly, sistive technologies equipped with human motion prediction can anticipate needs or ac- tions (like falling reaching for object) and provide timely assistance or interventions, thereby enhancing independence and quality of life.
Applications Autonomous Vehicles. Integrating this framework into autonomous vehi- cle systems can improve pedestrian safety and traffic management. By predicting human movements, autonomous vehicles can better navigate complex urban environments where interactions with pedestrians frequent and unpredictable.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Section B tell the method, Section P demonstrates results, proving the effec- tiveness the method.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Secon & analyzes the efficiency limitation of the proposed DiMoP3D.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
22

---

Justification:
Section E explains the assumptions, theorems, and formulas in detail.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theo- rems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Appendix A tells the training details and the supplemental material provides our source code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
23

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Appendix A tells the training details and the supplemental material provides our source code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://miis.cc/)
abblc//guddes/Cddeelliiisss occio for more details.
While we encourage the release of code and data, understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips.ccppin Sttiii//guidee//oodeeeee www.aaammissioooooooooooo asionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Appendix A tells the training details and the supplemental material provides our source code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropri- ate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: Appendix A tells the training details and the supplemental material provides our source code.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
24

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emror rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix A conduct all the experiments on single NVIDIA RTX3090 GPU within 8 hours.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't t make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have reviewed the NeurlPS Code of Ethics and ensured that our research conforms to
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The proposed DiMoP3D may have broader impacts on robotics, autonomous vehicles, video games, assistive technologies, etc., detailed in Appendix El
Guidelines:
25

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: There is high risk for misuse in the data models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The licenses and terms of use are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
26

---

If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswithcode com/datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification:
There are no new assets introduced in the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: There is no crowdsourcing experiments and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection. cura- tion, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper deseribe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: There is no research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
27

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "vW6UCNoFbb", "reviewer": "Reviewer_nnKQ", "summary": "This paper introduces a task that is scene-aware diverse human motion prediction. To be specific, given a 3D scene and history motion, this task aims to predict diverse future human motions that are consistent with the scene and history motion. This paper also proposed a model, DiMoP3D, to tackle this task. This model includes a Context-Aware Intermodal Interpreter to encode the scene and find the goal object, a  Behaviorally-Consistent Stochastic Planner to generate the end pose and an obstacle-free trajectory, and a diffusion model to generate the motions on the path.", "questions": "The question is the same as Weaknesses. Overall, the paper seems to propose a valuable task and has good results. Addressing the limitations mentioned above would significantly strengthen the work and I am positive to change the score.", "limitations": "The authors addressed the limitations.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. This paper tackles an interesting task.\n2. The numerical results are good.", "weaknesses": "1. As my understanding, if the author follows the original data split of CIRCLE and GIMO, there are no multiple possible future motions for history motion. How to make sure the model can generate diverse future motions trained on such datasets?\n\n2. Please compare the proposed task with [9, 29]. These two works have already tackled the task of scene-aware stochastic prediction. Why do the authors claim they proposed a new task?\n\n3. The author claims that the predictions must adhere to deterministic constraints, including physical consistency. The proposed method only considers the obstacle-free trajectory, but there is no constraint for the generated poses, e.g., there could be penetration with the ground for the walking sequence. \n\n4. What is the insight and novelty of DiMoP3D to make it different from the previous paper? Most techniques are similar to previous work. The scene encoder is from [71], the prediction of diverse goals is similar to [9], the HOI estimation is similar to [1, 2], the trajectory planning is similar to [77, 82, 29], the motion generator is similar to [87].\n\n5. Some parts are hard to read, e.g., the introduction.\n\n6. For the 3D human motion prediction/generation tasks, I prefer to see a video rather than only some figures for the visualization part.\n\n[1] Zhao, Kaifeng, et al. \"Compositional human-scene interaction synthesis with semantic control.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n\n[2] Hassan, Mohamed, et al. \"Populating 3D scenes by learning human-scene interaction.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: How to generate diverse motions training on such datasets?\n\n**A1**: Similar to [1,2] utilizing single\\_history-to-single\\_future data, DiMoP3D is designed to model the posterior distribution of potential motions $P(\\hat{X}_{L:L+\\Delta L} | X\\_{1:L}, S)$, rather than a deterministic mapping function $F(\\textbf{X}\\_{1:L},S)=\\textbf{X}\\_{L+1:L+\\Delta L}$ during training. This approach enables DiMoP3D to predict multiple future sequences by repeatedly sampling from this distribution, facilitating a single-to-multiple mapping.\n\n**Q2**: Other works in scene-aware stochastic prediction.\n\n**A2**: We argue that [9, 29] focus on simplistic, non-realistic scene representations, while our task utilizes real-captured, unstructured, and complex 3D scene point clouds. The methods in [9, 29] cannot adequately address the challenges presented by our configuration, making a direct comparison infeasible.\n**Compare with [9]**: This work processes the scene using a single 2D image and represents observed sequences with 2D skeletons, which significantly limits interaction with real-world 3D scenes. **Compare to task in [29]**: This is a synthesis method rather than prediction. It employs a predefined target object and does not infer intended targets accordingly. Furthermore, while it considers scene features, it lacks the capability for comprehensive perception and understanding necessary for real-world scene-aware motion predictions.\n\n**Q3**: Physical consistency.\n\n**A3**: Our emphasis on physical consistency prioritizes the crossmodal rationality and realism of motions in real-world scenes. While foot penetration is a recognized challenge, it has traditionally been addressed using various techniques [3,4]. After integrating PhysDiff [4] into DiMoP3D, we observed a significant reduction in Average Contact Penetration Depth (ACPD) from 0.98 to 0.56 and a decrease in foot penetration rate from 13.8% to 1.2% in the GIMO dataset. However, this integration relies on external libraries like IsaacGym, which substantially increase computational overhead. Although these methods are complementary to our approach, focusing on effectively avoiding foot penetration remains a promising direction for future research.\n\n**Q4**: What is the insight and novelty of DiMoP3D?\n\n**A4**: The proposed task of real-world scene-aware stochastic motion prediction is novel yet important to the field of autonomious vechiles and robotics. Stochastic prediction is essential for navigation applications to prevent collisions, yet existing methods fail to account for real-world scenes, yielding suboptimal results. To the best of our knowledge, we are the first to propose this task and develop DiMoP3D to address it. The components in DiMoP3D significantly diverges from existing approaches in several key ways:\n\n**Scene interpreter**: Our interpreter innovatively proposes to explicitly predict potential target objects through crossmodal analysis of past motion and the scene. The scene encoder [71] processes the point cloud to map it into the motion feature space, with target estimation remaining the focus of our interpreter. Besides, although [9] predicts diverse goals, it only identifies pixel points from simple 2D images, failing to recognize interactive objects with low accuracy, and thus does not effectively capture motion intentions in real 3D scenes.\n\n**Trajectory planning**: We plan stochastic obstacle-free trajectories towards targets in real-world scenes, addressing shortcomings of previous methods: **(1)** [29] uses traditional A* that generates deterministic trajectories, contrasting with our diverse prediction strategy. **(2)** [77] treats the scene as a binary map, where any position or point with a height greater than zero is considered an obstacle and thus blocked. In contrast, we recognize that some obstacles in real-world scenes are partially traversable and develope a numerically-continuous scene map (refer to Table 7). **(3)** [82] applies a module-based controller to navigate, which struggles with direct adaptation to real-world 3D point clouds and is challenging to fine-tune due to data limitations.\n\n**Human estimator**: While our estimator is inspired by previous research, its application is distinct. In our model, it serves as a deterministic semantic constraint, guiding the motion generator to comply with the scene context. This integration is part of our approach to tackling the challenge of scene-aware diverse motion prediction.\n\n**Motion generator**: The main insight of our motion generator is to harmonize the stochasticity of human movements with the deterministic constraints of the real-world scenes. Unlike [87], which only considers deterministic constraints between the contact joint (e.g., hand) and a single object, our approach accounts for whole-body constraints within the entire real-world 3D scene, guided by a self-prompted stochastic factor.\n\n**Q5**: Some parts are hard to read.\n\n**A5**: The introduction comprises five paragraphs: (1) the importance of stochastic motion prediction; (2) the limited consideration of real-world scenes in existing stochastic prediction methods; (3) the key challenges associated with scene-aware stochastic motion prediction; (4) the structure of our DiMoP3D, designed to address these challenges; (5) our contributions. We will refine the structure of the article to enhance readability and clarity in the next version. Welcome for any additional feedback!\n\n**Q6**: Video samples.\n\n**A6**: Kindly refer to the anonymous page in Appendix. A (line-588).\n\n[1] Barquero, et al. \"Belfusion: Latent diffusion for behavior-driven human motion prediction.\" ICCV2023.\n\n[2] Yuan, et al. \"Dlow: Diversifying latent flows for diverse human motion prediction.\" ECCV2020.\n\n[3] Liu, et al. \"Learning basketball dribbling skills using trajectory optimization and deep reinforcement learning.\" TOG2018.\n\n[4] Ye, et al. \"Physdiff: Physics-guided human motion diffusion model.\" ICCV2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xBAZkBTFXM", "reviewer": "Reviewer_p6AA", "summary": "The paper introduces a novel task that incorporates real-world 3D scene information into the existing Human Motion Prediction task. The authors propose a model (DiMoP3D) that, starting from the observed motion sequence and the 3D scene, stochastically predicts the future poses and the interactions with the context; in particular, DiMoP3D infers the individual intention and the human-object interactions. Upon benchmarking on two datasets, GIMO and CIRCLE, the model surpasses all the baselines on most of the considered metrics.", "questions": "I acknowledge that Sec. B.1 of the supplementary extends the literature review in Sec. 2 of the main manuscript, though the discussion could benefit from a tighter and more direct comparison with similar tasks in [1,2,3,4].\n\nTypos and minor issues with the writing:\n\nL99: forgot the full stop at the end of the sentence.\n\nL119: “intermodal insights. utilizing…”\n\nL181: “To tackle ress these”\n\nL211: “When \\alpha_T is approaches 0…”\n\nL261: “...is introduces…”\n\nL304: “...by the subject en route…”", "limitations": "Yes, they included the discussion about limitations and societal impact in Section G of the supplementary material.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed task lies in the intersection of existing ones while capturing a problem setting that hasn’t been explored yet. Similar tasks either considered only the contact points within the scene but with no stochasticity in the prediction ([1,2,3]) or focused on stochastically generating human motion conditioning on the observed pose sequence and the 3D scene but fixing the goal (e.g., [4]). As such, the discussion is technically sound and original. The proposed model is convincingly designed as the role of its submodules is properly motivated and analyzed both in the method’s section and in the ablation studies. The results on CIRCLE and GIMO confirm the effectiveness of the proposal.\n\n[1] Mao, Wei, Richard I. Hartley, and Mathieu Salzmann. \"Contact-aware human motion forecasting.\" Advances in Neural Information Processing Systems 35 (2022): 7356-7367.\n\n[2] Luca Scofano, Alessio Sampieri, Elisabeth Schiele, Edoardo De Matteis, Laura Leal-Taixé, Fabio Galasso. “Staged Contact-Aware Global Human Motion Forecasting.” BMVC 2023: 589-594\n\n[3]  Xing, Chaoyue, Wei Mao, and Miaomiao Liu. \"Scene-aware Human Motion Forecasting via Mutual Distance Prediction.\" arXiv preprint arXiv:2310.00615 (2023).\n\n[4] Hassan, Mohamed, et al. \"Stochastic scene-aware motion prediction.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.", "weaknesses": "W1. While it is clear how their proposed task diverges from human motion synthesis in 3D scenes, human-object and scene interaction prediction (e.g., [5]), and social navigation, it is not clear enough how it differs from scene-aware 3D human motion forecasting or synthesis. \n\n[5] Xu, Sirui, et al. \"Interdiff: Generating 3d human-object interactions with physics-informed diffusion.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: How it differs from scene-aware 3D human motion forecasting or synthesis, and comparisons with similar tasks.\n\n**A1**: **For scene-aware motion forecasting**: Traditional scene-aware 3D human motion forecasting [5,6,7] typically predicts a single sequence based on observation. In contrast, our task models a distribution of potential movements, enabling diverse future scenarios to emerge from a single historical motion. This capability is essential in autonomous vehicles and robotics to effectively anticipate and avoid potential collisions.\n**For scene-aware motion synthesis**: Synthesis methods [8,9,10] generate motions from arbitrary starting positions without considering historical context, while our task requires each generated sequence to be consistent with observed dynamics. Our method contributes to the robotics and navigation, while synthesis methods are often geared towards multimedia and gaming with less constraints.\n\nThe mentioned [1,2,3] focus on deterministic prediction, while [4] deals with motion synthesis. We will enhance the discussion of these similar tasks in the next version of our paper.\n\n**Q2**: Typos.\n\n**A2**: We will correct these typos in the next version.\n\n[1] Mao, et al. \"Contact-aware human motion forecasting.\" NIPS2022.\n\n[2] Luca, et al. “Staged Contact-Aware Global Human Motion Forecasting.” BMVC 2023.\n\n[3] Xing, et al. \"Scene-aware Human Motion Forecasting via Mutual Distance Prediction.\" arXiv:2310.00615.\n\n[4] Hassan, Mohamed, et al. \"Stochastic scene-aware motion prediction.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n\n[5] Zheng, et al. \"Gimo: Gaze-informed human motion prediction in context.\" ECCV2022.\n\n[6] Araújo, et al. \"Circle: Capture in rich contextual environments.\" CVPR2023.\n\n[7] Mao, et al. \"Contact-aware human motion forecasting.\" NIPS2022.\n\n[8] Wang, et al. \"Humanise: Language-conditioned human motion generation in 3d scenes.\" NIPS2022.\n\n[9] Hassan, et al. \"Stochastic scene-aware motion prediction.\" ICCV2021.\n\n[10] Huang, et al. \"Diffusion-based generation, optimization, and planning in 3d scenes.\" CVPR2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xjxJdUVyse", "reviewer": "Reviewer_wnWw", "summary": "This work studies human motion prediction in 3D scenes. The proposed DiMoP3D leverages context-aware intermodal interpreter, behaviorally-consistent stochastic planner, and self-prompted motion generator to solve the task. The authors conduct experiments on GIMO and CIRCLE to demonstrate a superior performance.", "questions": "See weakness part.", "limitations": "As discussed in Appendix G, the system is not as efficient as baselines.", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper is well-organized and easy to follow.\n- In extensive experiments,  DiMoP3D demonstrates a superior performance over the baselines with analysis.", "weaknesses": "- I question the novelty of this work. Novelties in Context Awareness and Autonomous Intention Estimation are overclaimed. They have been studied in previous works but authors fail to provide a discussion:\n[29] uses path planning to perform context-aware navigation and GoalNet to estimate the end-point state. \n[98] uses 3D scene point to encode the scene context and gaze to estimate the intention.\nStill more work to supplement...\nI believe the novelty is very overclaimed.\n\n\n- The system estimates the motion of a fixed length $\\Delta L$. Is it possible to not reach an end state? Taking Fig.2. as a case, how do you guarantee the human is sitting on the chair but not still walking to it after $\\Delta L$? It may lead to unnature speed and motions.\n\n\n- How do you evaluate the diversity and natureness of motions?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: The novelty of this work.\n\n**A1**: The task of predicting stochastic human motions with real-world scene awareness is crucial for embodied applications like robotics and autonomous vehicles, enhancing their navigational systems to effectively avoid collisions by considering dynamic real-world conditions. This aspect is usually overlooked by current tasks, and we believe we are the pioneers in introducing scene perception in diversity prediction to achieve better performance in real-world applications. Besides, a novel framework DiMoP3D is proposed to tackle this challenge. Please refer to **A4 to reviewer nnKQ** for details due to the limit of space. \n\nMoreover, compared with BiFU [98]: **(1)** DiMoP3D’s scene interpreter explicitly identifies potential targets as conditional factors for motion generation, in contrast to BiFU’s basic use of PointNet which lacks crossmodal analysis of past motions and intention analysis; **(2)** Human gaze is frequently unavailable in real-world settings, yet it forms the sole basis for inferring human intentions in BiFU. This heavy reliance  significently limits its practicality; **(3)** Different from human gaze which generally indicates deterministic intentions, DiMoP3D analyzes the distribution of human intentions through integrated scene-motion analysis, making it better suited for diverse prediction tasks.\n\nOverall, we will provide a more thorough discussion of our novelty and the differences between the components of DiMoP3D and existing methods in the next version.\n\n**Q2**: The system estimates the motion of a fixed length.\n\n**A2**: For scenarios where the subject reaches the goal within 5 seconds, DiMoP3D sets the subject to remain relatively static upon arrival. If targets are too far to be reached within 5 seconds, our scene interpreter would prevent their selection. In cases where the target is beyond reach within this timeframe, DiMoP3D predicts the initial 5 seconds of motion towards the goal, designating the 'floor' midway as the intended target and 'walk through' as the action of end-pose. Although this issue rarely occurs in the relatively small indoor scenes, it worths further discussion.\n\nThe 'floor' is categorized as a special class of object and is segmented by the point segmentator; points from the 'floor' also contribute to the interest calculation. In Equation 4, we calculate the average interest for each object to facilitate sampling of target objects. However, this approach is not suitable for the 'floor' object, as the 'floor' object spread a large area, and the interest levels across different areas of the floor can vary significantly. To address this, we specifically discuss the selection of the 'floor' as a target. We start by calculating the probability of choosing the 'floor' with $\\dfrac{\\sum_{p \\in \\text{floor}} \\exp(M[p])}{\\sum_{p \\in S} \\exp(M[p])}$. If the 'floor' is selected, a precise target point is identified using GumbelSoftmax within the floor's point cloud. In our experiments in small indoor scenes, the probability of selecting the 'floor' as a target is below 1%; therefore, we have not detailed this in the current version but will include a comprehensive discussion in the next update.\n\n**Q3**: How do you evaluate the diversity and natureness of motions?\n\n**A3**: **For diversity**, we adopt the APD metric (Average Pairwise Distance) following [1,2,3]. It reportes the L2 distances between poses and trajectories of the generated sequences. **For natureness**, we employ the ACPD (Average Cumulated Penetration Depth) to assess scene-motion consistency and physical naturalness, along with the FID (Fréchet Inception Distance) following [4,5,6], which quantifies the discrepancy between the latent distributions of generated motions and ground truth in synthesis. Given the complexity of accurately gauging naturalness, visualization serves as a valuable tool for a more intuitive evaluation.\n\n**Q4**: System efficiency.\n\n**A4**: We argue that the baseline methods are not scene-aware as they do not process 3D scene features, which explains their faster performance. However, DiMoP3D significantly outperforms these baselines, and the trade-off in speed is justifiable given its enhanced capabilities. Additionally, integrating efficient diffusion modules [7,8] and advanced vision modules [9] holds promise for further accelerating DiMoP3D's performance.\n\n[1] Barquero, et al. \"Belfusion: Latent diffusion for behavior-driven human motion prediction.\" ICCV2023.\n\n[2] Mao, et al. \"Generating smooth pose sequences for diverse human motion prediction.\" ICCV2021.\n\n[3] Yuan, et all. \"Dlow: Diversifying latent flows for diverse human motion prediction.\" ECCV2020.\n\n[4] Wang, et al. \"Move as You Say Interact as You Can: Language-guided Human Motion Generation with Scene Affordance.\" CVPR2024.\n\n[5] Tevet, et al. \"Human motion diffusion model.\" ICLR2023.\n\n[6] Wang, et al. \"Towards diverse and natural scene-aware 3d human motion synthesis.\" CVPR2022.\n\n[7] Wang, et al. \"Patch diffusion: Faster and more data-efficient training of diffusion models.\" NIPS2024.\n\n[8] Baranchuk, et al. \"Label-efficient semantic segmentation with diffusion models.\" arXiv preprint arXiv:2112.03126.\n\n[9] Zhu, et al. \"Vision mamba: Efficient visual representation learning with bidirectional state space model.\" arXiv preprint arXiv:2401.09417.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2GJgfsUYno", "reviewer": "Reviewer_Q59f", "summary": "The paper present a method for human motion prediction that take into account the environment (3d point cloud) in which the person moves. The method work with several steps: first from the past motion an area of interest is estimated and several object of interest (bed, chair...) with which the person might interact. after selecting the most likely object the method will estimate an interaction with the object (e.g sitting on the bed or lying on the bed) and plan a trajectory to reach the object. Finally with the trajectory and the estimated interaction the model generate the 3d motion from the end of the past motion to the end of the interaction. Since this is a novel task the author compare to two type of method : motion prediction and environment aware motion synthesis and outperform the state of the art on both tasks.", "questions": "Videos of the generated motion would have been appreciated\n\nline 261 introduces => introduced", "limitations": "Limitations and impacts are discussed in the appendix.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper introduce a new task of environment aware motion prediction where the environment is a static 3d point cloud.\n\nInteresting approach where the models first find the most likely item to be used then create a trajectory and finally generate the future motion based on that trajectory.\n\nThe paper provides extensive and detailed experiments and ablation study on two datasets\n\nThe qualitative results look very good.\n\nDespite tackling a new task the authors compare with sota methods for two related tasks.\n\nThe paper is clear,  well written and well detailed.", "weaknesses": "At least some results of the scene aware motion synthesis from the appendix should be moved to the main paper. Especially since there is no qualitative comparison with other scene aware method in the main paper.\n\nIn figure 1 it is not clear that diffusion is being used.\n\nA qualitative comparison with bifu would have been interesting since its results are the best after the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: Synthesis methods move to the main paper.\n\n**A1**: We will incorporate the discussion of the synthesis method into the main body of the paper in the next revision. Specifically, Section B.1 will be moved to Section 2.3, B.2 will be positioned in a new subsection between Sections 4.2 and 4.3, and B.3 will be relocated to Section 4.4. Additionally, we will expand our discussion on the differences between tasks in the introduction and in Section 3.1, and we will elaborate on the experimental results of the synthesis method in the newly added subsection of Section 4.\n\n**Q2**: In figure 1 it is not clear that diffusion is being used.\n\n**A2**: Figure 1 showcases the diverse predictions of DiMoP3D. Diffusion model adapts better to our diverse prediction task and outperforms both cVAE and GAN in terms of performance. Its stochastic nature also enhances the stochastic factors in DiMoP3D. If you are referring to Figure 2, we detail the structure of DiMoP3D there, illustrating how diffusion is implemented within the Self-Prompted Motion Generator. We will enhance the clarity of the use of diffusion model in Figure 2 in the next version.\n\n**Q3**: Qualitative results with BiFU.\n\n**A3**: We have included a qualitative comparison with BiFU in Section-5 of the anonymous page (see Appendix. A, line-588). Notably, DiMoP3D not only generates diverse predictions but also achieves closer alignment with the ground truth than BiFU in sequences with the same targets. This accuracy is facilitated by the guidance of predicted interactive poses and trajectories.\n\n**Q4**: Videos.\n\n**A4**: Kindly refer to the anonymous page in Appendix. A (line-588).\n\n**Q5**: Typos in line 261.\n\n**A5**: We will correct the typo in the next version.\n\n[1] Zheng, et al. \"Gimo: Gaze-informed human motion prediction in context.\" ECCV2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes the novel task of predicting diverse human motion in real-world 3D scenes, therefore abiding by stochastic predictions and 3D-scene-conditioning, to test with proposed metrics in the datasets of GIMO and CIRCLE and a new model for the task termed DiMoP3D.

Upon the rebuttal and the authors-reviewers discussion, the reviews were diverging, and one was borderline: 1x reject, 1x borderline reject, 1x weak accept, and 1x strong accept.

The AC summarizes these main aspects:

(1) The paper is clear and well-motivated, as agreed by most reviewers and confirmed by the AC

(2) The performance of the proposed model is effective, and the selected baselines are sensible, as agreed by all reviewers and confirmed by the AC.

(3) The proposed task is novel. Two reviewers are convinced about this aspect. Two reviewers are not convinced of this. The AC notes that the discussion with related work needs to be improved. The many raised questions clearly show that the authors need to make clearer points on the novelty and how this task differs from the existing ones. Effectively, the AC understands the two contrary reviewers' viewpoints, as this task appears incremental with respect to those tasks and benchmarks that are currently available. On the other hand, the AC stands on the positive side, arguing that proposing to bring together stochasticity and 3D real scenes is new and a great direction to proceed in this research field.

(4) The modules employed in DiMoP3D are mostly derived from best practices with limited novelty.

(5) As proposed, the new task is deemed helpful in progressing the relevant application fields. The proposed datasets and metrics appear to be an appropriate step forward. Progress on this task will likely reflect in progress across all the existing compound subtasks. This paper is the first to have proposed the task of diverse scene-aware motion prediction. The selected datasets are recent and large enough to ignite follow-up work on the task. The metrics are best practices in the field of scene-aware and stochastic human motion prediction, so the performance depicted in the paper is representative of modeling progress.

Further to the notes on novelty and performance, the motivation stems from the vast utility and timely need of the proposed task for application fields of significant potential impact, such as human-robot-scene navigation and interaction. As such, we recommend acceptance of this paper.
Decision: Accept (poster)