Contact-aware Human Motion Forecasting
Authors: Wei Mao
miaomiao Liu
Richard Hartley
Mathieu Salzmann
Venue: NeurIPS 2022 Accept (2022)
Keywords: scene-aware human motion prediction
human-scene contact

Abstract: In this paper, we tackle the task of scene-aware 3D human motion forecasting, which consists of predicting future human poses given a 3D scene and a past human motion. A key challenge of this task is to ensure consistency between the human and the scene, accounting for human-scene interactions. Previous attempts to do so model such interactions only implicitly, and thus tend to produce artifacts such as ``ghost motion" because of the lack of explicit constraints between the local poses and the global motion. Here, by contrast, we propose to explicitly model the human-scene contacts. To this end, we introduce distance-based contact maps that capture the contact relationships between every joint and every 3D scene point at each time instant. We then develop a two-stage pipeline that first predicts the future contact maps from the past ones and the scene point cloud, and then forecasts the future human poses by conditioning them on the predicted contact maps. During training, we explicitly encourage consistency between the global motion and the local poses via a prior defined using the contact maps and future poses. Our approach outperforms the state-of-the-art human motion forecasting and human synthesis methods on both synthetic and real datasets. Our code is available at https://github.com/wei-mao-2019/ContAwareMotionPred.

Contact-aware Human Motion Forecasting
Wei Mao', Miaomiao Liu', Richard Hartley', Mathieu Salzmann" 2.3 Australian National University: PCVLab, EPFL; "ClearSpace, Switzerland {wei mao, miaomiao.. liu, richard, hartley) edu. au, mathieu. salzmannSepfl.ch
Abstract
In this paper, we tackle the task of scene-aware 3D human motion forecasting, which consists of predicting future human poses given 3D scene and a past human motion. A key challenge of this task is to ensure consistency between the human and the scene, accounting for human-scene interactions. Previous attempts do model such interactions only implicitly, and thus tend to produce artifacts such as "ghost motion" " because of the lack of explicit constraints between the local poses and the global motion. Here, by contrast, we propose to explicitly model the human-scene contacts. To this end, we introduce distance-based contact maps that capture the contact relationships between every joint and every 3D scene point each time instant. We then develop a two-stage pipeline that first predicts the future contact maps from the past ones and the scene point cloud, and then forecasts the future human poses by conditioning them on the predicted contact maps. During training, we explicitly encourage consistency between the global motion and the local poses via prior defined using the contact maps and future poses. Our approach outperforms the state-of- of-the-arr art human motion forecasting and human synthesis methods on both synthetic and real datasets. Our code is available /tttps://githhi cwm/we////eeeoeoeeeeeeeee wei mao 2019/ContAwaane ConttAnareMtiioorrr
Introduction
Human motion prediction has a broad application potential covering human robot interaction [17].
autonomous driving 23), virtual/augmented reality (AR/VR) (28 and animation (3T]. As such, has been an active research topic for decades 4 27 30 B5). Nevertheless, most methods [19] [[]
22
21 disregard the fact that humans evolve in 3D environments, thus ignoring the human-scene interactions. By contrast, in this work, we tackle the task of scene-aware human motion forecasting, which aims to incorporate the scene context to predict future 3D human motions.
While some recent works in human motion prediction 66 8 and human synthesis [33] have started explore the use of scene context, they do implicitly, by taking an embedding of 2D scene image [6]. a 3D scene [33], or a specific object [8] as an additional input to their model. While such embeddings encode valuable information, they do provide precise cues help placing the human in the scene. In the context of synthesizing a static human body in scene, some efforts have nonetheless been made to incorporate more precise information, such as the distance between every scene point and the closest vertex on body surface (36), or semantic scene label, e.g., floor, sofa, at every vertex on the 3D body mesh [12]. However, while such representations allow one to place human in the scene, they remain under-constrained for motion prediction. Specifically, they impose neither temporal consistency, nor consistency between local pose changes and global motion. As such, they yield artifacts such as "ghost motion".
In this paper, we address this by explicitly modeling the contact between the human body joints and the scene. To this end, we introduce per-joint contact maps that encode the distance between each joint and every 3D scene point. Such contact maps constrain both the global motion and the local human pose, thus avoiding the "ghost motion" issue. As illustrated in Fig [ our human motion
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(a) predict the future contact maps
(b) forecast the future human poses
Figure I: Our approach. Given the historical motion, depicted as gray skeletons, and the 3D scene, (a) our model first predicts per-joint contact maps in the future frames, shown with color maps. Here, for visualization purpose, we only show the contact maps of the left foot (blue) and right foot (red). (b) Conditioned on the predicted contact maps, then forecast future poses, shown orange skeletons. The per-joint contact maps provide strong cues about the 3D human joint locations.
prediction framework therefore consists of a two-stage pipeline that first predicts the future contact maps and then forecasts the future human poses by conditioning them on the predicted maps.
Specifically, we obtain a compact temporal representation of the historical contact maps using a similar Discrete Cosine Transform (DCT) encoding strategy to that of [22] for human motion, and use a feed-forward 3D scene encoding network, i.e., PVCNN [20], to predict the future contact maps from the scene point cloud, past human motion and the historical contact maps. To then forecast human motion, we retain the closest contact points for every human joint according to the predicted contact maps and forecast the global translations and then the local poses given these points and the past human motion. Consistency between the global translations and the local poses is achieved via prior defined using the predicted human poses and the contact points.
Our contributions can summarized as follows. (i) We introduce a distance-basee based per-joint contact map that captures fine- -raingr human-scene interactions avoid generating unrealistic human mo- tions. (ii) We further propose a two-stage pipeline whose first stage models the temporal dependencies of past contact maps and predicts the future ones, and whose second stage forecasts the future human motion conditioned on these contact maps. Our experiments on both synthetic and real datasets demonstrate the benefits of approach over the state-of-the-art human motion forecasting and human motion synthesis methods.
2 Related Work
Human motion prediction. Modeling 3D human motion has been long-standing research goal 4
27 30 35). While traditional methods 4 [5]. relying on either Hidden Markov Models [4] or the
Gaussian process latent variable model [35], can tackle periodic and simple non non-periodic motions, such as walking and golf swing, more complex motions have been shown to be better modeled via deep learning frameworks 19
5 which can roughly categorized into feed-forward models [19, 22 21 and recurrent networks I
18
21
5 according to their temporal- spatial encoding strategies. Despite the success of these methods at forecasting complex motions, they typically only predict local poses, disregarding global motion and any scene information. In this paper, we seek to predict future human motions that are consistent with the 3D scenes they are performed in.
32 9 [8]
Recently, few works 8 have started incorporate scene context in motion forecasting. In particular, Corona et al. g introduced a semantic- aaaah model that extracts a joint embedding of the human pose and an object of interest, such as a cup. This method, however, ill-suited to model interactions with the whole scene itself, for example the floor or stairs that the person touches while walking. In [6), Cao al. proposed a multi-stage pipeline that breaks down the motion forecasting into three sub-tasks: predicting 2D goal, planning 2D and 3D path, forecasting the 3D poses

---

following the path. To this end, they extracted scene representation using 2D images and let their model learn the scene constraints implicitly. This strategy, however, cannot handle scene occlusions and does not enforce consistency between the local and glohal motion. More importantly, both these methods aim to learn interactions implicitly. By contrast, our contact maps explicitly encode the human-scene interactions, leading to direct constraints at the human joint level.
Scene-aware human synthesis. Synthesizing realistic human in 3D scene has recently gained 34
an increasing popularity (36 37
10] A cornerstone in the success of these methods the modeling of human-scens scene interactions. To achieve this, many of these works [7
12
33
follow similar approach to [6], modeling the human-scene interactions implicitly. via either Gen- erative Adversarial Networks (GANs) [4], or Variational Autoencoders (VAEs) [37 33. Three methods [36 12 [0] nonetheless exploit explicit representations of human-scoos interaction. In
particular, POSA 12 uses body-centric representation of the human-scene interaction where a semantic scene label, e.g., floor, sofa, is assigned to every human mesh vertex. This label encodes the contact probability to the scene surface and the corresponding semantic scene label. However, this semantic representation does not provide any information about the 3D location of the human body, and is thus ill-suited to the motion forecasting task. In PLACE [36]. Zhang et al. introduce a contact representation based on Basic Point Sets (BSP) [26]. Specifically, given set of basic 3D scene points, they represent the human-scens interaction using the minimum distance from every such point to the human body surface. As in the semantic- bassnttcc---edddddeee case, this strategy only gives weak prior on the human pose, as it does not explicitly defines which joint should be in contact with which scene point. In SAMP 10), their approach only models coarse interaction of the human with given object in the final frame by predieting the final root location and orientation. Such coarse interaction however cannot constrain the poses at intermediate frames. By contrast, our per-joint contact maps provide a more detailed contact information for every human joint at each future frame.
Hand-objed interaction Although hand-object contact relationships have already been studied for the task of grasping 2 29 14, existing methods cannot be naively applied to human-scene interactions because their object-centric contact relationships tend to be static across time. For example, when we are using a hammer, we will grasp the handle tightly, and thus the contact region between palms and the hammer does not change across time. By contrast, our human-scen contact maps change across the frames for almost all human activities. This motivates us to propose a blaace-based per-joint contact map each frame.
3
Approach
Let us now introduce our approach to scene-aware 3D human motion forecasting. Following previous work (22), sequence of P past human poses is represented as X [x1, X2, xx]] E 3x33 where Xp JR33 encodes the 3D locations all joints at time p in a global reference frame. The 3D scene is represented as a set of 3D points S E TR. NRNx3 Given the historical motion X and the 3D scene S. our goal is then to forecast T future human poses Y [XP+1,X442) ... XP+7] E xJxx To this end, we introduce a two-stage pipeline that first predicts future human-scene contact maps, and then forecasts the future poses by conditioning them on these contact maps. An overview of our pipeline is shown in Fig. 2 Below, we present our contact representation, and our pipeline to predict future contact maps and future motion.
3.1 Per-joint Scene Contact Map
We represent human-scene contact using the distances between the human joints and the scene points. Specifically, given a human pose x and a 3D scene S, we first compute the per-joint distance map d J where each entry dyn encodes the lg distance between the j-th human joint and n-th scene point, i.c.,
djn = |x, S,ll2
(1)
where j E (1, and (1,2....,.)) .N.
In the distance map, the scene points that are less relevant to a human joint, because they are far away from it, have higher value. This will tend to give them more influence when used in a deep neural network and may in turn cause issues when training our contact prediction module. To address this, normalize the distance map to obtain continuous contact map R JxN whose elements are
3

---

X&S
X
GRU
M
C
H
DCT
PVCNN
IDCT
X
GRU
root
Yroot
Y local
MLP
GRU
Figure 2: Network architecture. Our contact prediction network (top row) takes as input the scene points S, the past contact maps C and the past human motion X to predict the future contact maps M. From these contact maps, then extract the per-joint contact points Q for future time instants. Conditioned on these contact points, our motion forecasting network (bottom row) forecasts the global translations Frooot and then the local poses Ylocal
defined as
Cjn
(2)
where the constant is the normalizing factor. In such contact map, closer scene points have higher values than far-away ones, whose values will be very close to zero.
Given the 3D scene S and past human poses X, compute the sequence of past contact maps C= [c,, C2. ... Cyl E xPXxxN as described above. Our goal then is to predict the future contact maps M = [cp41, Cp42, Cp+T] R TxXNN In the next section, introduce contact map prediction module to do so.
3.2 Contact Prediction Network
Since contact maps are based on distances, they are smooth over time. Therefore, similar to 22 for human motion, we adopt a temporal encoding strategy based on the Discrete Cosine Transform (DCT). With the DCT, a sequence is represented a linear combination bases. By discarding the high-frequent components, the DCT provides a compact representation, which nicely captures the smoothness of the sequence.
of pre-defined cosine
More formally, let us denote a P-frame sequence of contact values between the j-th human joint and the n-th scene point čjn [cijn C2jn: Cpja]. This sequence can be fully represented using P DCT coefficients, with the 1-th one given by
2 E -VP
1 cos (2p 1)(1 ()(1-1)) -1)
hijn =
Cpjn VI+
(3)
where E (1,2, PP and dij denotes the Kronecker delta function, i.e.,
8,j [1 if
(4)
ij
Given these DCT coefficients, the sequence the original space can be obtained via the Inverse Discrete Cosine Transform (IDCT)
I.
Cpjn
hija
/I + dr
1 cos
2p
(2p 1)(1 -1)
(S)
1=1

---

where L is the number of DCT coefficients used. For lossless reconstruction, L = P, but when the sequence is smooth, one can ignore the coefficients the high-frequency DCT bases and thus use
L < P at a negligible loss.
Recall that our goal is to predict the future T contact maps M [cr+1, Cp42, cp44) from the past P ones C = [c1, C2 cpp. To leverage the DCT representation, we then reformulate this problem as learning mapping from the DCT coefficients of the past contact maps to those of the future ones. Specifically, following the padding strategy of [22]. we first repeat the last contact
map cp T times to create a sequence of P + T contact maps C' = [c1, C2, CP, Cp, [2(+7))JJXN We then compute the DCT coefficients of this sequence and define it as H E _LXJXNN Note that for each joint we only retain the first L DCT coefficients. Our goal then is to learn a residual between these DCT coefficients H and those of the real sequence [c1, C2. .
Cpl E
We formulate this a point-cloud encoding problem. That is, for each scene point, regard the corresponding DCT coefficients of all joints as its feature. Thus, every scene point has an initial feature vector of size LJ. Our contact prediction network then takes input the 3D scene points S, the DCT features H and the past human poses X, and outputs the future contact maps C = [c1, C22 -.. ,Cprrl as
Č IDCT F(S G.))))
(6)
where FF represents the trainable point-cloud processing model and G2 is the GRU encoder.
Specifically, as shown in Fig. 2 we use the Point-Voxel CNN (PVCNN) 20 to encode the 3D scene with its DCT feature vectors. The PVCNN was designed to process a 3D point cloud. incorporates voxel-based convolutions and point- based representations, leading to memory- and computation-efficic structure for 3D data. We adapt the PVCNN to also take as input the past human poses X, encoded by a Gated Recurrent Unit (GRU) 7, predict residual of the DCT coefficients. We then obtain the predicted contacted maps Č [é1, é2, ,Cp4T] via the IDCT.
Note that, here, we seek not only to predict the future contact maps but also to recover the past ones. To this end, we use the average &g loss between the ground-truti truth contact maps and the predicted ones. Formally, this loss is defined as
P+T
mmpp (P (P+T)JN
11ppjn - Epjn
3.3 Motion Forecasting Network
Given the future contact maps M and the past human motion X, human motion forecasting module aims to predict the future human poses Y. To this end, we first extract the closest contact scene point each joint from the contact maps. Given these contact points, we first use a simple neural network to predict the future path (global translations) and then forecast the future local poses with an RNN-based model. Let us discuss these steps in more detail.
Contact points. Since our focus now to predict the 3D location of human joints, we propose to retain the most relevant scene point i.e., the closest one each joint. Specifically, given the contact map at time step p Cp E J and the 3D scene points S E R Nxx we would like to find the scene point that is closest to each human joint. Let us denote the resulting contact points as qp E IR Jx4 each of which stores the 3D location of the scene point closest the corresponding human joint together with a binary value indicating whether the joint truly is in contact with the scene or not. More formally, the contact point for joint j computed
SSa, 1], where k argmax Cj, if Cj.k >
(1,2.....N)
00,0,0.0
otherwise,
where Sa is the 3D location of the k-th scene point, and e is a threshold to determine whether the joint is in contact with the scene not. We compute such contact points for the entire future sequence, which yields a sequence of contact points Q = [qr+1, 9P+2.' [94T]] RTxJx1 where qp Rd xx is the contact points at the p-th time step.
5

---

Motion forecasting. As shown in Fig 2 each frame and then the local motion given the global translations. Specifically, given the past motion X and the contact points Q, we use a simple multilayer perceptron (MIP) to predict the future global
our forecasting model first predicts the global translation in
translations Yroo E Tx3
Y (X), Q)
where M represents the MLP and G. the GRU to encode the past motion.
The global translations are then fed into a GRU to predict the local pose at each future time step. Assuming that the past and future local pose sequences represented by X [xloca, x pcal Px(--1x) and Y/ocaa [x P42 x E RTx(J-1)x3
local
respectively, this can expressed as
xp = xp=1 + ( 11
geer Ap. 91 (X)).
where 9 denotes the GRU to predict the furture poses, E + 1, ... , T), and &local E ((J-1)x3 &pm & R and dp E R are the local human pose, global translation and contact points time p, respectively.
The global translation and local motion prediction modules are trained jointly. To this end, we use loss terms. The first one is a global translation loss defined as
Coor
Σ p=
X
roor
root
(10)
T
where x E R33 and Xppar E R33 are the ground-truth and predicted global translations at time p.
The second loss accounts for the local human pose prediction and is expressed
1 P+T
Gocal
Σ
x
local
local
(11)
T(J -)
p=P+1
where logal E R33 and pj E R are the ground-truth and predicted local positions of the j-th joint time p.
Finally, our third loss term encodes a contact prior based on the contact points. We define it as
P+T
Lcontact
Σ
Apjal1*pp
(12)
TJ
P+1
where Xpj Rd the predicted location of joint j at time p. obtained by adding the predicted global translation time p to the corresponding local pose. 4py[1:3] E R and 4pj4 {0, 1) the 3D coordinates the contact scene point and the indicator value, respectively.
The overall loss is then expressed as
motion X1 Eroot + Apfiocal + Aghermtact
(13)
We use stage-wiss training strategy where the contact prediction network and motion forecasting network are trained separately. During training, the motion forecasting network is given the ground- truth contact points input. At test time, first compute the contact points from the predicted contact maps and then use these contact points to forecast the future human motion.
Experiments
4.1
Datasets
We evaluate our method on two datasets, GTA-IM 6 and PROX [T].
GTA-IM. The GTA Indoor Motion dataset 6 is large-scale synthetic dataset that captures human- scene interactions. consists of 50 different characters performing various activities in different
scenes. Each scene is building, and each building has several rooms more floors. The

---

Path Error (mm)
Pose Error (mm)
method
0.5s
1.0s
1.5s
2.0s
mean
0.5s
1.0s
1.5s
2.0s
mean
LTD [22]
67.0
119.3
207.6
375.6
147.4 4
67.5
93.8
98.9
103.5
80.5
DMGNN 18
82.7
158.0
227.8
286.9
156.2
47.5
69.1
85.6
95.3
64.9
SLT' 33
45.9
117.0
186.7
267.1
121.8
70.8
181.4
150.2
196.0
112.6
Ours
58.0
103.2
154.9
221.7
108.4
50.8
67.5
75.5
86.9
61.4
Ours w/o contact
61.1
111.7
171.0
249.0
118.8
57.8
74.8
82.4
98.1
68.2
Ours w/ GT contact
52.4
77.8
95.8
129.5
74.1
49.8
64.8
70.4
78.3
58.2
Table 1: Quantitative results on GTA-IM [6]. We report the MPJPE in millimeter for both the
global translations (path error) and the local poses (pose error). The "mean" error was obtained by averaging over the 60 future time steps. The indicates that we adapted the official SLT (33) code, designed for motion synthesis, to our task. We also show ablation results of our model without contact maps ("Ours w/o contact") and with ground-truth contact maps ("Ours w/ GT contact").
Path Error (mm)
Pose Error (mm)
method
0.5s
1.0s
1.5s
2.0s
mean
0.5s
1.0s
1.5s
2.0s
mean
LTD [22]
117.8
232.0
346.9
461.2
236.3
156.0
273.9
387.7
497.9
273.3
DMGNN [8
119.1
242.7
360.2
462.4
243.7
91.0
141.3
171.8
187.8
129.1
SLT 33
105.8
227.2
384.1
453.5
255.0
112.1
230.6
233.7
269.6
175.5
Ours
93.3
187.2
284,4
381.2
192.2
89.9
127.5
149.3
167.5
116.8
Ours w/o contact
104.9
196.5
290.0
385.5
200.1
90.3
135.4
160.5
184.1
122.4
Ours w/ GT contact
73.9
106.7
104.6
117.4
88.0
83.7
112.9
125.2
132.9
101.1
Table 2: Quantitative results on PROX [11]. Our model outperforms baseline models by large margin across all time steps.
dataset contains around million RGB-D frames together with the corresponding 3D human poses. We use 4 of the scenes as our training set ("r001", ""r001",rr002",, "r003", ""03""""""" and the remaining 3 as our test set ("r010", "T01 1"0,1""" "r'0113"" To obtain the 3D point clouds of the different scenes, we register their depth maps from different videos sequences with the ground-truth camera extrinsic matrices. Following [6]. we use 21 out of the 98 human joints provided by the dataset. The videos run at 30Hz. We train our models to observe the past 30 time steps (1 second) and predict the future 60 time steps (2 seconds).
PROX. Proximal Relationships with Object eXclusion (PROX) 11 is real dataset captured using Kinect-One sensor. It comprises 12 different scenes with 20 subjects interacting with the scenes, The dataset also provides SMPL-X parameters (25 as the ground-truth human pose and shape in each frame. Since these parameters were obtained by frame-wise fitting algorithm, the motion sequences are jittery and thus ill-suited to our task. We therefore refine the dataset via a simple temporal optimization process to generate smooth motions. More details are provided in the supplementary material. Following [33, we use 8 scenes for training ("N3Library", "MPHI12", "MPH11", "MPH8" "Basemen&SittngBoooth "NOSofa", "N3Office", "Werkraum") "') and 4 scenes for testing ("MPH16", "MPH1 Librarrr "NOSittinBBooth" "NeOpenArea"". We use the 22 body joints of SMPL-X model. As for GTA-IM, the frame-rate rate of this dataset is 30 Hz, and we train our models to take the past 30 time steps as input and predict the future 60 steps.
4.2 Metrics, Baselines & Implementation
Metrics. We use the Mean Per Joint Position Error (MPJPE) 13 to evaluate both the global translations (path error) and the local motion (pose error).
Baselines, We compare our method with two human motion prediction models (LTD (2 and
DMGNN (18) and one scene-aware human motion synthesis method (SLT (3) LTD 22 repre-
sentative feed-forward method based on Graph Convolutional Networks (GCNs) 16). DMGNN 18
Note that the dataset does not provide an official training-testing split. We use this split to balance the number of motion sequences in training and testing.
7

---

GTA-IM
PROX
GT contact
GT poses
Ours contact
Ours poses
Figure 3: Predicted contact maps on GTA-IM 66 (top) and PROX [1] (bottom). We show the contact maps of four joints: left foot (blue points), left elbow (purple points), right foot (red points) and right elbow (green points). To show the influence of our contact maps, we also show the past motion (grey skeleton), ground-truth trutt future motion (orange) and our predictions (blue).
is iate-of-the-ar RNN- approach for human motion prediction. SLT 33 is a stage-wise approach to synthesize long-term human motions. We used the official implementations of LTD 22 and DMGNN [18] to train them our datasets. For SLT (3), we adapted the official code to our task e.g., we modified their model so as to take the past motion as input. The detail these changes are provided in the supplementary material.
Implementation details. Our models are implemented in Pytorch 24 and trained using the ADAM [15 optimizer. Both our contact prediction network and motion forecasting one are trained for 50 epochs with learning rates of 0.0005 and 0.001. respectively. The training of each network takes about 12 hours on a 24GB NVIDIA RTX3090Ti GPU and the evaluation of one sample takes around 90 ms during testing. For both datasets, the normalizing factor a, the number of DCT coefficients L and the contact threshold are set to 0.2, 20 and 0.32, respectively. For the motion forecasting network, the loss weights (A1, X2 As) are set to (1.0, 1.0, 0.1) for both datasets. For each motion sequence, we randomly sample 5000 scene points that are within 2.5 meters away from the root joint of the last observed pose. Additional implementation details are given supplementary material.
4.3 Results
Quantitative results. We provide quantitative results on GTA-IM and PROX in Table and 2 respectively. Our approach outperforms the baselines for 3D paths and poses both datasets across almost all time steps by a large margin. Specifically, the baseline models either perform well for the 3D path but comparatively poorly for the local poses, e.g., SLT 331 with average path error of 121. 2m.8mm but the highest pose error on GTA-IM, or the reverse, e.g., DMGNN [8]. By contrast, our models produce more accurate 3D paths and poses than those of the baselines.
As an ablation study, we trained our human motion forecasting network without contact maps ("Ours w/o contact") and observe increase of up-to 10mm in the pose error. By contrast, using the ground-trute contact maps ("Ours w/ GT contact") yields a further performance boost, especially the mean path error with decrease of up-to 104mm. This indicates the effectiveness of conditioning the motion predictions on per-joint contact maps.
path error and 7mm in the mean
Qualitative results. We show our contact maps of four joints (left foot, left elbow, right foot and right elbow) Fig. 3 Our model predicts accurate contact maps for diverse motions, such as "walking" (top) and "sitting down" (bottom). Note that, for the sample from GTA-IM 6 where the subject is about walk around corner, our contact maps precisely capture the contacts between the left elbow and the wall (shown purple points on the wall), Icading to accurate human motion predictions.

---

Side view
Top view
Side view
Top view
GT
DMGNN
SLT
Ours
Figure 4: Qualitative comparison. We show the results of DMGNN I8] SLT 33 and model
on GTA-IM [6] (top) and PROX (I]) (bottom). The past, ground-truth future and predicted future motions are shown in grey, orange and blue, respectively, and the shade of each color indicates the time step. The baseline methods, which either do not consider scene context [18) or implicitly model the numan-scene interactions (highlighted with a red box) or walking through sofa (highlighted with green boxes).
[33] tend to produce unrealistic human movements, e.g, ghost motion
We further compare our results with those of the baselines in Fig. We restrict this comparison to DMGNN [18) and SLT (33). which quantitatively more accurate than LTD (22) The complete comparison is included
the supplementary material. Due to the lack of explicit constraints on
global motion and local movements, the baseline methods tend to produce unrealistic human motions, such as motions with almost no local movements but large global translations, i.e., ghost motion (highlighted with red box), walking through sofa (highlighted with green boxes). Thanks to per-joint contact maps, our results are more plausible and closer to the ground truth.
Conclusion
In this paper, we have introduced framework for scene-aware human motion forecasting that encourages consistency between global motion and local poses by exploiting human-scene contacts. To this end, we have proposed a per-joint contact map representation that captures the contact relationships between every human joint and the scene points. Our model consists of two stages. We first predict the per-joint contact maps given the motion history, and then forecast the future global

---

translations and local poses given the estimated future contact maps. Thanks to the explicit constraints provided by our per-joint contact maps, our method yields more plausible and more accurate future human motions than the state-of-the-art motion prediction or scene-aware human synthesis strategies.
Limitations & Societal Impacts One limitation of our work is that the quality of our predicted motions depends on that of the contact maps. As evidenced by our results with ground-truth contact maps, improving the contact map predictions translates in better motion predictions. This will therefore be one of our future research directions. Additionally, in many applications where the human shape matters, our joint-based contact map may not be enough to regularize the human surface. In our future work, we would like to extend our contact map to human surface. Furthermore, in real application, a potential risk of our method is that it may predict future motions that do not obey real physical rules, e.g,, motion with imbalanced forces. For example, in the scenario of human-robe robot interaction where the agent i.e., a robot, needs to plan its actions according to the future human motion, such physically unrealistic future motion may lead to unsafe situations such as collision.
Acknowledgements
This research was supported in part by the Australia Research Council DECRA Fellowship (DE18010022)) and ARC Discovery Grant (DP200102274), The authors would like to thank NVIDIA for the donated GPU (Titan V).
References
[1] Emre Aksan, Manuel Kaufmnnn,, and Otmar Hilliges. Structured prediction helps 3d human motion modelling. In ICCV, pages 7144-7153, 2019.
[2] Samarth Brahmbhatt, Cusuh Ham, Charles C Kemp, and James Hays. Contactdl:: Analyzing and predicting grasp contact via thermal imaging. In CVPR, pages 8709-8718 8719, 2019.
[3] Samarth Brahmbhatr, Chengcheng Tang. Christopher D Twigg. Charles C Kemp, and James Hays. Con- tactpose: A dataset of grasps with object contact and hand pose. In ECCV, pages 361-378. Springer, 2020.
[4] Matthew Brand and Aaron Hertzmann. Style machines. Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 183-192. ACM Press/Addison- Wesley Publishing Co., 2000.
[5] Yujun Cai, Lin Huang, Yiwei Wang, Tat-Jen Cham, Jianfei Cai, Junsong Yuan, Jun Liu, Xu Yang, Yiheng Zhu, Xiaohui Shen, et al. Learning progressive joint propagation for human motion prediction. In ECCV, pages 226-242. Springer, 2020.
[6] Zhe Cao, Hang Gao, Karttikeya Mangalam, Qi-Zhi Cai, Minh Vo, and Jitendra Malik. Long- Long-term term human motion prediction with scene context. In ECCV, pages 387-404. Springer. 2020.
[7] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using mnn encoder-decoder for statistical machine translation. EMNLP, 2014.
[8] Enric Corona, Albert Pumarola, Guillem Alenya, and Francesc Moreno-Noguer. Context-aware human motion prediction. In CVPR, pages 6992-7001, 2020.
[9] Anand Gopalakrishnan, Ankur Mali, Dan Kifer, Lee Giles, and Alexander Ororbia. A neural temporal model for human motion prediction. In CVPR, pages 12116-12125, 2019.
[10] Mohamed Hassan, Duygu Ceylan, Ruben Villegas, Jun Saito, Jimei Yang, Yi Zhou, and Michael J Black. Stochastic scene-aware motion prediction. In ICCV, pages 1.374-11344, 2021.
[11] Mohamed Hassan, Vasileios Choutas, Dimitrios Tzionas, and Michael J Black. Resolving 3d human posc
ambiguities with scene constraints. In ICCV, pages 2282 2292,22222222222222222 2019.
[12] Mohamed Hassan, Partha Ghosh, Joachim Tesch, Dimitrios Tzionas, and Michael Black. Populating 3d scenes by learning human-scene interaction. In CVPR, pages 14708-14718, 2021.
[13] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.66 6:6: Large scale datasets and predictive methods for 3d human sensing in natural environments. TPAMI, (6()):1325-1339 jul 2014.
[14] Hanwen Jiang, Shaowei Liu, Jiashun Wang. and Xiaolong Wang- Hand-object contact consistency reasoning for human grasps generation. In ICCV, pages 11107-11116 2021.
[15] Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. ICLR, 2015.
[16] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In
ICLR, 2017.
[17] Hema Swetha Koppula and Ashutosh Saxena. Anticipating human activities for reactive robotic response. In IROS, page 2071. Tokyo, 2013.
[18] Maosen Li, Siheng Chen, Yangheng Zhao, Ya Zhang, Yanfeng Wang, and Qi Tian. Dynamic multiscale graph neural networks for 3d skeleton based human motion prediction. In CVPR, pages 214-223, 2020.
10

---

[19] Xiu Li, Hongdong Li, Hanbyul Joo, Yebin Liu, and Yaser Sheikh. Structure from recurrent motion: From rigidity recurrency. In CVPR, June 2018.
[20] Zhijian Liu, Haotian Tang, Yujun Lin, and Song Han. Point-voxel can for efficient 3d deep learning.
NeurIPS, 32, 2019.
[21] Wei Mao, Miaomiao Liu, and Mathieu Salzmann. History repeats itself: Human motion prediction via motion attention. In ECCV, pages 474-489. Springer, 2020.
[22] Wei Mao, Miaomiao Liu, Mathieu Salzmann, and Hongdong Li. Learning trajectory dependencies for human motion prediction. In ICCV, pages 9489-9497, 2019.
[23] Brian Paden, Michal Cáp, Sze Zheng Yong, Dmitry Yershov, and Emilio Frazzoli. A survey of motion planning and control techniques for self-driving urban vehicles. IEEE Transactions intelligent vehicles, 1(1):33-55, 2016.
[24] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Vetioo, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NeurlPS- W, 2017.
[25] Georgios Pavlakos, Vasilcios Choutas, Nima Ghorbami, Timo Bolkart, Ahmed A. Osman, Dimitrios Tzionas, and Michacl J. Black. Expressive body capture: 3D hands, face, and body from single image. In CVPR, pages 10975-10985, 2019.
[26] Sergey Prokudin, Christoph Lassner, and Javier Romero, Efficient learning on point clouds with basis point sets. In ICCV, pages 1332-4341,,011 2019.
[27] Hedvig Sidenbladh, Michael Black, and Leonid Sigal. Implicit probabilistic models of human motion for synthesis and tracking. In ECCV. pages 784-800. Springer, 2002.
[28] Sebastian Starke, He Zhang. Taku Komura, and Jun Saito. Neural state machine character-scene interactions. ACM Trans. Graph., (8)6)99-- 2019.
[29] Omid Taheri, Nima Ghorbani, Michael J Black, and Dimitrios Tzionas. Grab: dataset of whole-body human grasping objects. ECCV, pages 581-600. Springer, 2020.
[30] Graham W Taylor, Geoffrey E Hinton, and Sam Roweis. Modeling human motion using binary latent
variables. In NeurlPS, pages 1345 1352, 2006.
[31] Herwin Van Welbergen, Ben JH Van Basten, Arjan Egges, Zs M Ruttkay, and Mark H Overmars. Real time animation of virtual humans: a trade-off between naturalness and control. In Computer Graphics Forum, volume 29, pages 2530-2554, 2554. Wiley Online Library, 2010.
[32] Borui Wang, Ehsan Adeli, Hsu-kuang Chiu, De-An Huang, and Juan Carlos Niebles. Imitation learning for human pose prediction. In ICCV, pages 7124-7133, 2019.
[33] Jiashun Wang, Huazhe Xu, Jingwei Xu, Sifei Liu, and Xiaolong Wang. Synthesizing long-term 3d human motion and interaction in scenes. In CVPR, pages 9401-9411, 2021.
[34] Jingbo Wang. Sijie Yan, Bo Dai, and Dahua Lin. Scene-aware generative network for human motion synthesis. CVPR, pages 12206-1215,, 2021.
[35] Jack M Wang. David J Fleet, and Aaron Hertzmann. Gaussian process dynamical models for human motion. TPAMI, 30(2):283-298 2008.
[36] Siwei Zhang, Yan Zhang, Qianli Ma, Michael J Black, and Siyu Tang. Place: Proximity learning of articulation and contact in 3d environments. In 3DV, pages 642-651. IEBE, 2020.
[37] Yan Zhang, Mohamed Hassan, Heiko Neumann, Michael Black, and Siyu Tang. Generating 3d people in scenes without people. In CVPR, pages 6194-6204, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] see Section [
(b) Did you describe the limitations of your work? [Yes] see Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] see Section S
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
11

---

(a) Did you include the code. data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or URL)? [Yes] Data and implementation details are in Section 4 and the supplementary material. Our code is available at tattps://githin
com wei- http///////ottaaaaaoooooooaaaaaa
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] see Section [ and the supplementary material
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See the supplementary material
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] see the supplementary material
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "ms6dAWB_Cg4", "reviewer": "Reviewer_cHVU", "summary": "This paper promotes explicit contacts modeling when handling the challenging scene-aware human motion forecasting problem,\n\nTo achieve that, a dense scene-joint distance maps are utilized to densely model human dynamics when interacting with the static scenes,\nfollowed by a novel discrete pre-processing with DCT to get sparse principal features, while also enabling residual motion prediction of\nthe contacts point in frequency domain. \n\nA two-stage pipeline is assembled together to get better future motion predictions for all 3D body joints, with stage 1 combining GRU-based dynamics modeling and PVCNN-based 3D scene encoding for sequential contact distance maps prediction, and stage 2 carrying out contacts-guided sequential motion forecasting in stage 2.\n", "questions": "1. Related to Weakness 2, is this single closest contact priors\nwould be enough to guide the motion forecasting network? The single contact point looks like a special form of\nsigned distance function to me.\n\n2. The motion forecasting network is still a very implicit design when leveraging contact points,\nlike for the global translation, the author seems to want the MLP module to learn root motion implicitly from all the closest points,\nvery similar to a learnable center point. If this is possible, I am wondering how this implicit design would be better than\na simple xyz averaging over all closest points belonging to the feet-nearby joints (maybe plus a certain offset).\nPlus, it is not clear that in Figure 2, how does GRU take all three inputs to update the local joints offsets prediction?\n\n3. How does number of scene points affect the prediction performance? Do we need to select all the scene points in order to train the stage 1 model? The author does not mention the computation efficiency clearly in the implementation section.", "limitations": "See above\nMinor: Line 179, should be '...are then fed into...' ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n[Novelty]\nAs mentioned above, dynamic contacts modeling and the way to use it are the shining points in this paper, in which the authors leverage an effective point-joint distance field followed by a novel frequency transformation(DCT) to better capture sparse smooth motion patterns. Per-joint closest scene point is used to better guide the motion prediction. This design \n\n[Completeness]\nThree baselines and their proposed methods are validated in two common datasets, including both real and synthetic ones. They also provide both quantitative and qualitative results, including a video demo in the supp. \n\n[Effectiveness]\nThe author gets consistently better motion forecasting results(global and local) on long-term motion predictions on two benchmarks compared to all the baselines.\n\n**Weaknesses**\n1. Firstly, even though the whole method seems to be novel, I do not think that line 49-54 is carefully written to\nwell capture the whole work, in (ii) the two-stage pipeline itself should not be considered as a key contribution,\nit has overlaps with (i), also I did not see any unique technical contribution statements when tackling this conditional\nmotion synthesis task. The author needs a better clarification on the contributions.\n\n2. Even though so many efforts(DCT/IDCT transformation, GRU models, PVCNN) have been conducted to get better\ncontact maps forecasting, it seems that only one nearest scene point is selected and used per joint point, I am\nwondering whether such a heavy pipeline is really necessary. \n\n3. The result in table 1 seems to show that the proposed method does not perform well enough on short-term predictions, even though it performs consistently better on predictions >= 1s\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_507hdQuFVu", "reviewer": "Reviewer_QN5C", "summary": "This paper proposes to tackle scene-aware 3D human motion forecasting by explicitly modeling the human-scene interactions, i.e., representing the contact between human body joints and scene points with a distance-based contact map. They also introduce a two-stage pipeline that first predicts the future contact map with the given motion history; then forecasts the future global translation and local poses. The proposed method can predict more physically plausible motions and avoid artifacts such as “ghost motion”.\n\n**Main Contributions:** This paper proposes a contact map representation explicit modeling human-scene interactions and propose a two-stage framework to forecast human motions with given motion histories and 3D scenes.", "questions": "1. How do you use the motion features extracted by GRU in PVCNN? The PVCNN processes the 3D point cloud at point level, but the motion feature is the feature of the given motion history. Additionally, the notation $\\mathbf{X}$ in Eq.6 is inaccurate, as $\\mathcal{F}$ takes as input the latent feature of motion history instead of past human poses $\\mathbf{X}$. Similar problems in Eq.9 and Eq.10.\n\n2. The partial problem setting follows GTA-IM, which forecasts future human motions with a multi-stage pipeline. The quantitative evaluation should include the comparison between the proposed method and the variant of the method in GTA-IM.\n\n3. How about the model's ability of forecasting long-term motions (more than 5 seconds)? The current setting only predicts the future motions in 2 seconds which is too short for humans to have distinguishable movements.", "limitations": "When considering the contact between human and scene, I think the shape-based human body representation, e.g., SMPL and marked-based representation, is more reasonable to model the contact between body surface and scene. This representation can produce a more fine-grained contact map, which thus can model more realistic details of human-scene interactions.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths:\n\n1.\tThis paper explicitly models the human-scene interactions with a contact map, which measures the distance between the human joints and the scene points. The contact map enables more physically plausible and realistic human-scene interaction generation.\n2.\tThe proposed two-stage prediction pipeline disentangles contact prediction and human pose forecasting, thus capable of explicitly encouraging consistency between human motions and contact points in given 3D scenes.\n\n- Weaknesses:\n\n1. The contact map computed between joints and scene points is too coarse. It is more plausible to compute the contact map between body surface vertices and scene points because the human body surface rather than joints contact with the environment.\n2. The contact map has been widely used in grasp generation tasks[ Jiang et al., ICCV2021; Brahmbhatt et al., CVPR2019 ]. I think you should discuss the contact map in the literature review. And the idea of using the contact map to model human-scene interactions is not very appealing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SdPix4XY7Np", "reviewer": "Reviewer_EMAJ", "summary": "This work proposes a two-stage human motion forecasting framework that explicitly models human-scene contact. It proposes to decouple the problem into two stages: past pose conditioned contact forecasting, and contact-conditioned pose forecasting. Specifically, it proposes to use a Discrete Cosine Transform (DCT) based network to predict contacts based on past contact, human pose, and scene point clouds. After the future contact is predicted, a series of networks are used to predict the human’s global translation, rotation, and body joint positions. ", "questions": "It would be great if the authors could compare more closely with the motion generation literature (such as SAMP) and discuss differences. ", "limitations": "The authors have discussed limitations adequately. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n\n**Explicit Contact Modelling**\n\n- The proposed two-stage pipeline is intuitive and performs well in the context of human motion prediction. Contact and physical constraints play an important role in governing human motion and based on contact human motion are a lot less ambiguous. The idea of explicitly predicting the future contact of humans in a known scene to guide motion prediction is interesting.\n\n**Performance compared to State-of-the-art**\n\n- The proposed method outperforms SOTA methods in the motion prediction task.\n\n## Weakness\n\n**Novelty in lieu of motion generation methods**\n\n- Given the existence of methods such as SAMP [1], where human motion is generated based on path and scene context, the proposed framework has limited novelty. While the settings are slightly different (motion and interaction generation vs forecasting), the methodology is largely similar. The two-stage modeling has been largely explored (first generate goals or subgoals, then generate local motion), and this work mainly excels at better modality (explicit contact).\n\n**Lack of generative modeling** \n\n- While human motion is multi-modal, the lack of generative and stochastic modeling means that the estimated human motion could be memorizing past observed interactions (especially in PROX and GTA-IM datasets, where the motion are largely similar).\n\n**Lacking Qualitative Results**\n\n- Since motion is better seen in videos, it would be better if more qualitative samples are provided.\n\n[1] Hassan, Mohamed et al. “Stochastic Scene-Aware Motion Prediction.” 2021 IEEE/CVF International Conference on Computer Vision (ICCV) (2021): 11354-11364.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept