HUMANISE: Language-conditioned Human Motion Generation in 3D Scenes
Authors: Zan Wang
Yixin Chen
Tengyu Liu
Yixin Zhu
Wei Liang
Siyuan Huang
Venue: NeurIPS 2022 Accept (2022)
Keywords: language and 3D scene
motion generation
language-conditioned generation
human-scene interaction

Abstract: Learning to generate diverse scene-aware and goal-oriented human motions in 3D scenes remains challenging due to the mediocre characters of the existing datasets on Human-Scene Interaction (HSI); they only have limited scale/quality and lack semantics. To fill in the gap, we propose a large-scale and semantic-rich synthetic HSI dataset, denoted as HUMANISE, by aligning the captured human motion sequences with various 3D indoor scenes. We automatically annotate the aligned motions with language descriptions that depict the action and the individual interacting objects; e.g., sit on the armchair near the desk. HUMANIZE thus enables a new generation task, language-conditioned human motion generation in 3D scenes. The proposed task is challenging as it requires joint modeling of the 3D scene, human motion, and natural language. To tackle this task, we present a novel scene-and-language conditioned generative model that can produce 3D human motions of the desirable action interacting with the specified objects. Our experiments demonstrate that our model generates diverse and semantically consistent human motions in 3D scenes. 


HUMANISE: Language-conditioned Human Motion Generation in 3D Scenes
Zan Wang 1,2, Yixin Chen?, Tengyu Liu?
Yixin Zhu 3644 Wei Liang" 14663 Siyuan Huang 25
X indicates corresponding authors
School of Computer Science Technology, Beijing Institute of Technology
Beijing Institute for General Artificial Intelligence (BIGAI) Institute for Artificial Intelligence, Peking University Yangtze Delta Region Academy of Beijing Institute of Technology. Jiaxing
https: https://siLverrstrrrg github. 1 HUMANISE/
Sit the couch
Walk the refrigerator
Lie down on the bed
Stand up from the table far away from the door
Sit the toilet
Figure Examples from our proposed Human-Scene Interaction (HSI) dataset HUMANISE. contains 19.6k high-quality motion sequences in 643 indoor scenes. Each motion segment contains rich semantic informa- tion about the ground-truth action type and objects being interacted with, specified by language descriptions.
Abstract
Learning to generate diverse scene-aware and goal-oriented human motions in 3D scenes remains challenging due to the mediocre characteristics of the existing datasets on Human--Scene Interaction (HSI); they only have limited scale/quality and lack semantics. To fill in the gap, we propose large-scale and semantic-rich synthetic HSI dataset, denoted as HUMANISE, by aligning the captured human motion sequences with various 3D indoor scenes. We automatically annotate the aligned motions with language descriptions that depict the action and the unique interacting objects in the scene; e.g., sit on the armchair near the desk. HUMANISE thus enables a new generation task, language language-condition conditioned human motion generation in 3D scenes. The proposed task is challenging it requires joint modeling of the 3D scene, human motion, and natural language. To tackle this task, present novel scene-and- and---gaada conditioned generative model that can produce 3D human motions of the desirable action interacting with the specified objects. Our experiments demonstrate that model generates diverse and semantically consistent human motions in 3D seenes.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Introduction
Imagine instructing an agent (e.g., ., a virtual human or humanoid robot) to "sit on the armchair near the desk." Upon accepting the instruction, the agent needs comprehend its semantic meaning (i.e., sit) and understand the surrounding environment (i.e., the armchair near the desk) to generate desired interactions with the 3D scene. Despite recent progress in human motion synthesis, generating 3D human motion conditioned on the instructions with proper scene context is still challenging.
Specifically, existing works generate plausible human poses iZhang et al., 2020b;a] or motions [Starke et al., 2019; Cao et al., 2020; Wang et al., 202 Hassan et al., 2021a; Yi et al., 2022] by learning from existing HSI datasets (e.g., PROX [Hassan et al., 2019], GTA-IM [Cao et al., 2020]). However, these datasets possess two fundamental issues that prevent existing methods from learning instruction-aware and generalizable 3D human motions.
Limited scale and quality. PROX [Hassan et al., 2019] only contains 12 indoor scenes with jittering human motions. The synthetic GTA-IM [Cao et al., 2020] only has 49 scenes and lacks 3D human shapes and diverse movement styles.
Absence of scene and interaction semanties. PROX [Hassan et al., 2019] and GTA-IM [Cao et al., 2020] both suffer from incomplete 3D semantic segmentation annotations. Particularly, the lack of human action labels/descriptions hinders its ability to generate instruction-aware human motions.
To tackles the above issues, we propose a large-scale and semantic-rich synthetic HSI dataset, HUMANISE (see Fig. 1). by aligning the captured human motion sequences [Mahmood et al., 2019] with the scanned indoor scenes [Dai et al., 2017]. Specifically, for an action-specific motion sequence (e.g., (it) we first sample an interacting target object (e.g., armchair) and the possible interacting positions on the object surface. Next, we sample the valid translation and orientation parameters by employing the collision and contact constraints, such that the interactions between the transformed agent and the scene are physically plausible and visually natural. Finally, we automatically annotate the synthesized motion sequences with template-based language descriptions; for instance, sit on the armchuir near the desk. To specify the interacting objects, we adopt the object referential utterances in Sr3D [Achlioptas et al., 2020J. In total, HUMANISE contains 19.6k high-quality motion sequences in 643 indoor scenes. Each motion segment has rich semantics about the action type and the corresponding interacting objects, specified by the language description.
Based on HUMANISE, we propose new task: dunguage-conditoned conditioned human motion generation in 3D scenes. It requires the model to generate diverse and physically plausible human motions with designated action types and interacting 3D objects specified by the language descriptions. We argue that this task is more challenging than previous motion generation tasks in the following aspects: (i) The motion generation is conditioned on the multi-modal information including both the 3D scene and the language description. (ii) The generated human motions should perform the correct action and be precisely grounded near the target location according to the language descriptions. (iii) The generated human motions should be realistic and physically plausible within the 3D scenes. Therefore, ideal model should equip with capabilities of multi-modal understanding, language grounding, and physically plausible generation.
Computationally, we leverage the conditional variational unto-encoder (cVAE) [Sohn et al., 2015] framework to generate human motions conditioned on the given scene and language description. The scene and language conditions are learned with separate encoders and fused with the self-attention mechanism to generate conditional embedding. The input motion is encoded with a recurrent module and decoded with a transformer decoder to generate human motions. We further design two auxiliary losses for object grounding and setion-specific generation. The qualitative and quantitative results show that our model generates diverse and semantically consistent human motions in 3D scenes; it outperforms the baselines on various evaluation metrics. By evaluating on downstream tasks, further experiments demonstrate that HUMANISE alleviates the limits of existing HSI datasets.
This paper makes three primary contributions: (i) We propose large-scus scale and semantic-rice -ich synthetic HSI dataset. HUMANISE, that contains human motions aligned with 3D scenes and corresponding language descriptions. (ii) We introduce a new task of languuge-conditoon
human motion generation in 3D scenes that requires holistic and joint understanding of scenes, human motions, and language. (iii) We develop a generative model that produces diverse and semantically consistent human motions conditioned on the 3D scene and language description.

---

2
Related work
Conditional human motion synthesis Human motion synthesis can be conditioned on past motion
[Li al., 2018; Barsoum ct al., 2018; Yuan and Kitani, 2020; Cao et al., 2020; Xie ct al., 202 3D scenes [Starke et al., 2019; Wang et al., 2021a;b; Hassan et al., 2021 aa], objects [Chao et al., 2019; Taheri et al., 2020), action labels [Guo et al., 2020; Petrovich et al., 2021] and language descriptions [Ahuja and Morency, 2019; Ghosh et al., 2021; Lin and Amer, 2018]. Given a furnished 3D indoor scene, algorithms [Zhang et al., 2020b;a; Hassan et al., 2021 Wang et al., 202 can generate physically plausible single-frame human pose or long-term human motions. However, these methods either generate random/uncontrolllan
human motions or require target positions due to the lack of semantics constraints. Provided textual descriptions, existing motion generation algorithms [Lin and Amer, 2018; Ahuja and Morency, 2019; Ghosh et al., 2021] focus only on motions alone and neglect the scene context. In this work, we incorporate the context of both scenes and language descriptions, generating human motion sequences consistent with the specified action and location.
Human-Scene Interaction (HSI) datasets Data captured in both the physical and the virtual worlds have been adopted to study HSI related topics [Wang et al., 2019; Chen et al., 2019; Liang et al., 2019; Jia et al., 2020; Wang et al., 2020; Huang et al., 2022]. PiGraphs [Savva et al., 2016], captured with RGB-D sensors, suffers from inaccurately reconstructed scenes, noisy human poses, and low frame rate. Even with new pipelines to reconstruct human motions from monocular RGB-D sequences, PROX's s pose fitting results [Hassan et al., 2019] are still visibly worse compared to high-quality, large-scale MoCap data (e.g., AMASS [Mahmood et al., 2019I). GTA-IM [Cao et al., 2020] exploits the game engine to collect a synthetic dataset of human skeletons, but with limited HSIs and scene diversities. In comparison, our HUMANISE dataset synthesizes large-scale HSIs by aligning high-quality motions in AMASS [Mahmood et al., 2019] with real-world 3D scenes in ScanNet [Dai et al., 2017], resulting in high-quality and realistic human-scene interactions with high diversity (i.e., 643 scenes, 19.6k motions) and rich semantics provided by language descriptions.
Language grounding in 3D scenes Grounding 3D objects w.r.t. the language input has been explored in 3D reference |Achlioptas et al., 2020; Chen et al., 2021; Hong et al., 2021; Li et al., 2022; Thomason et al., 2022], 3D question answering (QA) [Das et al., 2018; Ye et al., 2021; Azuma et al., 2021]. and embodied navigation [Anderson et al., 2018; Gordon et al., 2019; Gan et al., 2020]. To date, locating the interacting objects in 3D scenes with referential descriptions is still challenging [Roh et al., 2022; Chen et al., 2020; 2021] due to the noisy scanned scenes, diverse expressions, and complex spatial relation reasoning. To better learn conditional embedding that facilitates locating and grounding according to the provided language descriptions with proper scene context, we introduce a multi-mode conditional module that learns a joint embedding from textual descriptions and 3D point clouds. We further enhance the model's spatial grounding and action grounding capabilities by introducing two auxiliary tasks of locating interacting 3D objects and action comprehension.
3 The HUMANISE dataset
A large-scale and high-qualit dataset with rich semantic information is in need to facilitate the research in human motion synthesis and HSI. In this work, we propose a synthetic dataset by leveraging the existing datasets in human motion (i.e., AMASS [Mahmood et al., 2019I). 3D indoor scene (i.e., ScanNet [Dai et al., 2017]), and additional annotations of these datasets (i.e., motion action label in BABEL [Punnakkal et al., 2021] and 3D referential descriptions in Sr3D [Achlioptas et al., 2020]). Our novel automatic synthesis pipeline avoids expensive hardware and time-consuming labor for realistic capture. Fig. 2 shows more samples in HUMANISE Of note, our synthesis pipeline can generalize to other 3D scene datasets and actions; please refer to Appendix C for more details.
Below, we start by describing how to align the motion with the 3D scene in See. 3. followed by automatic language description generation in Sec. 3.2. Sec. 3.3 provides dataset statistics.
3.1 Motion alignment
The idea of synthesizing HUMANISE is straightforward: Automatically aligning captured human motions with 3D indoor scenes. Given human motion sequence and its action label, we first sample an interacting target object by the predefined category labels and possible interacting positions on the object surface in the 3D scene. Next, we sample the valid global translation 'I and rotation R. parameters by employing the collision and contact constraints; we align the human motion with the 3D scene through transformation with T and R, while all the local poses remain unchanged.
3

---

While the sable
couch
couch
to
Sit the armchair that
Sit the conch the armrchai
Sir on the couch
Sit on the chair
Sit the milet
Stand up
Stand
futthest Boon laptop
Stand the
Stand from the bed
Lie down table close to the cabinet
Lie Cown on cleser to the backpock
table
down on
Samples from the HUMANISE dataset. HUMANISE includes representative interactive actions (e.g., sit, stand-up, lie-down, walk) in diverse indoor scenes (e.g., office, bedroom, living room, dining room).
Figure 2:
Collision constraint We treat the point cloud of the 3D scene a 3-dimensional k-d tree [Bentley, 1975]. Collision avoidance of human motions is equivalent to finding the closest point to the queried human vertex in the scene by searching through the k-d tree. Collision happens as the distance between the human vertex and the closest point the scene is less than a threshold. A valid motion should avoid collisions with the objects or walls for all the poses along the trajectory.
Contact constraint We further decompose this constraint into the foot-ground support constraint and the proximity between the human and the interacting object. The support constraint is distance threshold between the foot and the ground plane. The human-object proximity differs among actions; we design the following action-specift proximity constraints for realistic interaction:
Sit We sample a sittable object (e.g., a chair) interacting objects and uniformly sample 3D points on the object surface as potential contact points. We assume the final pose of the sitting motion should contact the sittable object on the hip area, such that the distances between the hip vertices and the contact point should be less than threshold.
Stand-up Since the stand-up motion interacts with the sittable object at the starting frame, we enforce the similar constraints in sit the starting pose instead of the final pose.
Walk The language description in HUMANISE provides instructions to walk to a target location, usually to object; e.g., walk to the refrigerator. We sample dense positions near the specified object on the floor as the potential targets for final pose of walking motion. along the walking trajectory should satisfy the feet-ground support and avoid collisions.
the body poses
Lie-down We select objects with a large flat surface (e.g., bed) as the interacting objects. The translation T and rotation R are sampled such that the final pose of the lie-down motion is in contact with the object's top surface. Apart from the non-collision and contact constraint, require the human body to lie within the object area.
Diverse affordance We sample motions representing diverse 3D affordance with the motion alignment algorithm, e.g., stand up from the coffee table, lie down on the office table, and on the toilet. ITUMANISE contains these Human-Object Interactions (HOIs) that are not frequently captured in daily activity or previous HSI datasets. They encode meaningful and versatile object/scene affordances, which introduce both potentials and challenges to the model building.
3.2 Language description generation
The motion descriptions HUMANISE depict the action type and the objects/locations being interacted with. To automatically generate these language descriptions, design compositional

---

template following Sr3D [Achlioptas et al., 2020]:
< action ><
target-class > [< spatial >< anchor-class(es) >].
For instance, sit on the armchair near the desk. The template contains four placeholders, of which spatial-relation and anchor are optional placeholders. The action is taken from the motion labels in BABEL [Punnakkal et al., 2021], whereas the target represents the interacting object. To uniquely refer to a target object, a spatial-relation is defined between the target and a surrounding object (anchor). We utilize the referential utterances in Sr3D [Achlioptas et al., 2020], which defines five types of spatial relations: horizontal proximity, vertical proximity, between, allocentric, and support.
3.3 Dataset statisties
HUMANISE includes 19.6k human motion sequences in 643 3D scenes. The four most representative actions are sit (5578), stand up (3463), lie down (2343), and walk (8264). Tab. compares HUMANISE with existing HSI datasets; HUMANISE shows advantages in scene variety, clip number, frame number, and human pose quality. Of note, HUMANISE is the only dataset with rich semantic information of HSIs. Fig. 2 shows more examples from HUMANISE. Please refer to Appendices A and B for more statistics and quantitative comparison between HUMANISE and PROX, respectively.
Table 1: Comparison between HUMANISE and existing HSI datasets.
Dataset
#Scenes
#Clips
#Frames
Human Representation
Pose Jittering
Semantics
PiGraph [Savva et al., 2016]
30
63
0.IM
Skeleton
PROX-Q |Hassun et al., 2019]
12
60
0.1M
Shape
GTA-IM [Cao al,, 2020]
1.0M
Skeleton
HUMANISE
643
19.6k
1.2M
Shape
4 Language-conditioned human motion generation in 3D scenes
4.1 Problem definition and notations
Our goal is to generate a human motion sequence that is both semantically consistent with the language description and physically plausible in interacting with the scene. We denote the given scene as S E R representing an RGB-colored point cloud N points. The language description is tokenized word sequence of length D, denoted LI.D [w,,--. wp) We represent the human pose and shape in the motion sequence using the SMPL-X [Pavlakos al., 2019] body model. Specifically, the SMPL-X body mesh M E R10475x3 is parameterized by M = F(t., r, B.P), p). where t R3 is the global translation, r E R a continuous representation [Zhou et al., 2019] of the global orientation, 3 E RI0 the body shape parameters, p E RJx3 J joint rotations in axis-angle, and I' differentiable linear blend skinning function. To model the effect of body shape in motion generation, we treat 3 as an additional condition and denote the parameters of interest as θ = (1,,,p]
4.2 Method
We propose a novel generative model to generate human motions conditioned on the given scene and language description. The overall architecture is illustrated in Fig. 3. Specifically, employ conditional variational auto-encoder (cVAE) framework to model the probability p(O1.T) S, LT.D), where T' is the length of the motion sequence. Below, we describe the details of each module.
Condition module The condition module takes input from two modalities (i.e., the given scene S and the language description L) and outputs a joint conditional embedding. To process the 3D scene, employ the Point Transformer [Zhao et al., 2021] to produce point-level features [u,,". uxd]. where the gransition-dowe module reduces the cardinality of the scene point cloud from N points to N' points. For the language description LI.D, we use the pre-trained BERT [Kenton and Toutanova, 2019] to extract the word-level embedding
The point-level features and the word-level features are concatenated and fed into the feature fusion module, a single-layer self-attention module. As result, both the point-level and the word-level features attend to the features from two modalities by self-attention. The updated point features are concatenated and forwarded to fully-connected (FC) layer to obtain the scene feature f.. We take the updated [CLS] feature as the language feature fw. The scene and language features are finally concatenated and mapped to conditional latent embedding 2e with FC layers.
5

---

Condition Module
Auxiliary Tasks
2
FC Layer
Motion Encoder
fw
fa
μ. Σ
Self-attention Layer
Sampling from N(u, 2)
Mxx"
BERT
Point Transformer
Motion Decoder
Wi
W2
WT
[CLS]
sit
chair
Figure 3: Illustration of the proposed generative model. It adopts the cVAE framework and incorporates motion encoder and decoder to generate human motions. The condition learned from the joint embedding the 3D scene and language description with auxiliary tasks 3D location grounding and action recognition.
Motion encoder We first use bidirectional GRU to obtain a sequence-level feature of the input motion O Next, the output is concatenated with the conditional embedding že, followed by MLP layer to predict the Gaussian distribution parameters (i.e., and E). Finally, we sample a latent vector z from the distribution using the reparameterization trick [Kingma and Welling, 2013].
Motion decoder Following Petrovich et al. [2021]. we use a transformer decoder to generate a sequence of parameters θ for a given duration T. Specifically, we use T sinusoidal positional embeddings query the concatenation of the sampled latent z and the conditional embedding že- The outputs of the transformer decoder are mapped into body meshes with the differentiable SMPL-X model. Of note, although the duration T equals the length of the input motion sequence in the training phase, it can be arbitrary values during inference.
4.3 Training loss
We devise the training loss with three components: reconstruction loss, KI. divergence loss, and two auxiliary losses for object grounding and action-specine generation. Formally, the training loss is:
C Lree + agg Cut a. + Oka Car
where a denotes the loss weight of each term. We detail each component below.
Reconstruction loss
Our reconstruction loss is formulated as
Lrec - a, + Cp + ag
(2)
where Li, Cr, and Lp are & distance between the predicted body parameters and the ground truth (i.e., global translation t, global rotation and joint rotations p). L. is an additional term for regressing the positions of the vertices on the SMPL-X body mesh. In practice, the vertex set can be all 10475 vertices on the SMPL-X body mesh or selected sparse markers [Zhang et al., 2021].
KL divergence loss Denoting the learned distribution of latent = v (r, E), regularize & to be similar to the normal distribution & = (O, I), ie., Lur = (x
Auxiliary loss Learning the conditional latent embedding 2, is crucial to generative model, as it represents compound information the action, the interacting object, and the 3D scene context from two different modalities. Rather than simply maximizing the negative evidenced lower bound (ELBO), we introduce two auxiliary tasks for locating the interacting target object and generating action-specific motions.
For the first auxiliary task, we directly use an FC layer on top of the latent =, to regress the target object's s center. For the second auxiliary task, we use another FC layer followed by softmax to map the latent 2, to the probability of each action. Specifically, we use the MSE loss C. in the target object center regression and the cross-entropy loss term C, in the action category classification.

---

4.4 Implementation details
We train our generative model on HUMANISE for 150 epochs using Adam [Kingma and Ba, 2014] and a fixed learning rate of 0.0001. For hypee-parametery
we empirically set Ordl Q, = 0.1. au = 0.5, a, 1.0, and ap = 0g 10.0. We set the dimension of global condition latent 2e 512 and latent to 256. The hidden state size is set to 256 in the single-layer bidirectional GRU motion encoder. The transformer motion decoder contains two standard layers with the 512 hidden state size. We train our model with batch size of 32 on V100 GPU. It takes about 50 hours to converge when training the action-agnostic model on the entire HUMANISE dataset.
Point transformer Our point transformer module that processes the scene point cloud adopts from the original architecture [Zhao et al., 2021]. For each scene, we down-sample N 32768 points from the original scanned scene and obtain N' = 128 points with 512-D feature for each point after applying the point transformer. We pre-train the point transformer with a semantic segmentation task on ScanNet [Dai et al., 2017] dataset, whose train-test split is the same as HUMANISE.
BERT We employ the official pre-trained BERT [Kenton and Toutanova, 2019] model process the language descriptions with maximum length of 32. We map the obtained 768-D word-level features into 512-D before concatenating them with the point features.
5
Experiments
We benchmark our proposed task tos---auuuggge-oddineeeeee human motion generation 3D scenes on HUMANISE and describe the detailed settings, baselines, analyses, and ablative studies.
5.1 Experimental setting
To evaluate the model performance, conduct experiments in both the action speccin and action- agnostic settings. In the action-specific setting, we train and evaluate our model on HUMANISE subsets of each action without the auxiliary task of classifying the action category. In the action- agnostic setting, we train our full model on the entire dataset with auxiliary tasks.
Dataset splitting We split motions in HUMANISE according to the original scene IDs and split in ScanNet [Dai et al., 2017], resulting in 16.5k motions in 543 scenes for training and 3. lk motions in 100 scenes for testing.
Ablative baselines We compare our model with three types of ablative baselines to verify the significance of the auxiliary tasks, the feature fusing module, and the scene encoder, respectively:
We remove the auxiliary tasks for locating the target interacting object and recognizing the action category, denoted as w/o L. (only removing L.), w/o La (only removing L.),, and w/o aux. loss (removing both L., and L.). We train these models on the entire HUMANISE dataset.
Instead of feeding the scene and the language feature into the self-attention layer, we directly concatenate them as the global conditional feature. We denote this model as w/o self-att. We train and test on the walk subset.
We replace the point transformer with PointNet++ [Qi et al., 2017] (i.e., PointNer++ Enc.), also pertained with the semantic segmentation task. We train and test on the walk subset.
5.2 Evaluation metrics
Following Wang et al. [2021a], we introduce the following metrics to evaluate the model's reconstruction capability. We report the l error (x (x100) between the predicted SMPL-X parameters and the ground truth, including global translation t, global orientation and body pose p. We also compute the Mean Per Joint Position Error (MPJPE) [Lonescu et al., 2013] and Mean Per Vertex Position Error (MPVPE) in millimeters between the reconstructed SMPL-X body mesh and the ground truth to evaluate the reconstruction in a more intuitive and fine- ane-ggaieed manner. We report these metrics by averaging over the sequence due different motion duration.
Reconstruction metrics
As we hope our generated motion interacts with the correct object specified by the language description, we introduce body-to-goal distance metric (goal dist.). It is the shortest distance (in meters) from the target object to the interacting human body, computed as Max(Min(W (M (O)),, 0), where v vt( is the positive signed distance field of body mesh M, and
Generation metrics
7

---

Sit the sofa chair that is close to the tv
Stand up from the coffee table close to table
Walk to the bed
Lie on the bed
Reference
Generation
Generation
Generation
Attention
Figure 4: Qualitative generation results of action-specific models on the HUMANISE dataset. We visualize one reference motion and three generation motions for each scenario. The attention map visualizes the attention weights the [CI.S] token attended to the scene point cloud in the self-attention layer. Red denotes higher weight.
O C. S the target object point set. Of note, we use the human body in the first frame to compute the body-to-goal distance for stand up action and use the last frame for other actions. We generate ten randomly sampled motion sequences for each case in the test split and report the average distance. To measure the diversity the generated motion, we report the Average Pairwise Distance (APD) [Yuan and Kitani, 2020; Zhang et al., 2021]: The average Eg distance between all pairs of motion samples within scenarios. APD is computed K[K-1J7 ExEE Σ | -x,,ell- where K is sample size (we set K- 20), and Xi,t the sampled marker-based representation [Zhang al., 2021].
Perceptual study We perform human perceptual studies to evaluate the generation results in terms of the overall quality and cetion-semantie accuracy. Given the rendered motion and the language description, a worker is asked to score from 1 to 5 for (i) the overall generation quality, and (ii) whether the generated motions are performing the action specified by the description. A higher value indicates that the generated result is more plausible with the given scene and language description. We randomly generate samples in 20 scenarios for each model, and three workers score each sample.
5.3 Results and discussion
Tab. 2 tabulates the guantitative results of both reconstruction and generation. Fig. 4 and 5 show some qualitative results. Below, discuss some significant observations.
As seen in Tab. 2, our model performs well in the action-specific setting, e.g., walk, sit, and stand up, Fig. 4 also shows our model generates human motions that are visually appealing and semantically consistent with the language description. Furthermore, the attention visualizations show that the [CLS] token attends to the point cloud features around the target interacting objects. Meanwhile, we also notice that the goal dist. lie down much smaller than other actions. We suppose the model finds it easier to ground the target object in the lie down action, as the interacting objects are mostly large furniture with a flat surface such as "bed" and "table." We provide more clarifications with ablative experiments the grounding loss weight and data efficiency in Appendix F.
8

---

Table Quantitative results of reconstruction and generation HUMANISE dataset.
Reconstruction
Generation
Model
trans)..
orient..
pose.
МP]РЕ]
MPVPЕ,
goal dist...
APD1
quality scoret
action scoret
sit
5.17
3.19
1.77
113.28
112.43
0.903
10.12
2.37
3.19+1111
stand up
5.63
3.43
1.69
126.05
124.84
0.802
9.57
2.83=11223
4.20:1::94
lie down
6.46
3.09
0.76
136.87
136.20
0.196
9.18
1.31=1008
2.85+:1.31
walk
5.84
2.80
1.85
125.05
123.88
1,370
12.83
2.91
3.88
wlo self-att.
5.72
1.85
122.19
120.81
1,500
13.28
2.88=1.144
3.80:11.09
PointNet++ Eice
5.81
2.64
1.81
124.67
123.69
1.444
12.61
2.80=1.35
3.75+1.27
all actions
4.20
2.91
1.96
98.01
96.53
1,008
11.83
2.57=11.20
3.59+11.38
w/o Co
4.20
2.89
1.93
98.15
96.69
1.383
15.09
2.42=1.211
3,57
4.23
1.95
98.67
97.11
1.135
12.66
2.17=1004
2.29
w/o aux. loss
4.28
2.99
1.92
99.30
97.80
1.361
15.18
1.97=0.98
2.44:11.38
(a) Reference
(b) w/o aux.
(c) w/o aux. loss
(d) Ours
Figure 5: Ablation results of action-agnostic models. For the description sil on the coffee table, the model wo aux. loss struggles in (b) generating the action specified by the description or (c) locating the interacting object. (d) In comparison, full model generates motions semantically consistent with the language description.
The action-agnostic setting is more complex and difficult than the action-specific setting, but the auxiliary tasks improve the model's capability in spatial and action grounding. As shown Tab. removing L. leads to a higher goal dist., which means L, can help our generative model to locate the target 3D object. La will help the model generate motions with the correct action label as can be seen from the action score. The significant differences in goal dist., quality score, and action score indicate that the two proposed auxiliary tasks help provide better generation results while maintaining the reconstruction quality. Qualitative results in Fig. 5 further verify this observation.
The ablative experiments on the feature fusing module and scene encoder validate our model designs. For the action walk, our full model outperforms the model w/o self-atf. in all generation metrics. We conclude that the transformer layers learn better conditional embedding by attending to features across the scene and language modalities. The difference goal dist. compared with +ointNet+ Enc. shows our better grounding performance through finer-grained scene understanding.
Discussion of failure cases We provide
typical examples of failure cases in Appendix E.
Our model sometimes fails to ground the correct object for interaction as locating objects the 3D scene with referential descriptions is still challenging [Roh et al., 2022; Chen et al., 2020]. This calls for more efforts joint learning of different tasks and modalities in language modeling, scene understanding, and motion generation.
is the incorrect HOI relation with physical implausibility and collision. We suspect this is primarily due to the lack of explicit HOI modeling. Our current architecture is an end-to-end generative model; the scene information is represented by the point-level and scene-level features rather than object-level geometry and semantics. These point out that goal-oriented motion generation may benefit from future work on 4D HOI modeling and object affordance understanding.
Another typical failure
Qualitative results of different duration T Following Petrovich et al. [2021], I], we use a transformer decoder as the motion decoder to generate a sequence of body parameters θ1.r in given duration T. Fig. 6 shows some generated motions of different duration (i.e., 30 frames. 60 frames, 90 frames, and 120 frames). We render a human body every 15 frames for fair comparisons.
Please refer Appendix F for additional experiments and Appendix D for qualitative results.
5.4 HUMANISE for downstream tasks
We further demonstrate the quality of HUMANISE by another existing downstream task proposed in Wang et al. [2021a]: humun motion synthesis in 3D scenes. This task only considers the scene- conditioned generation, and the language annotations HUMANISE are not used this setting. More specifically, this task takes scene point cloud, the start and the end locations, and the start and

---

30 Frames
60 Frames
90 Frames
120 Frames
Figure 6: Motions generated by sampling with different duration T. The language deseriptions in these two cases are walk to the end table that is farthest from the door and sit on the coffee table. Our model is capable of generating motions with various duration.
the end human body poses as input; the goal is predict all human poses in between. For model architecture, we follow Wang et al. [2021a] to include a RouteNet and PoseNet. The RouteNet takes the scene point cloud, the start and the end locations and orientations inputs; it generates possible route between the start and end location. The PoseNet takes the same scene point cloud, the start and the end human body poses, and the route predicted by RouteNet as inputs; it outputs the parameters for all the human poses along the route.
We train the RouteNet and PoseNet using HUMANISE for ten epochs and fine-tune them using PROX [Hassan et al., 2019] for another ten epochs. For a fair comparison, we reproduce the results of the same model trained on PROX for 20 epochs under the same setting. We evaluate both models on the test split of PROX. The reconstruction errors with metrics from Wang et al. [2021a] are reported in Tab. 3. The model pre-trained evaluation metrics by a large margin. This result shows our dataset can alleviate the limitations of existing HSI datasets and potentially support more applications in HSI related topics.
HUMANISE outperforms the model trained only with PROX on all
Table 3: Results of human motion synthesis on PROX dataset.
Method
translation
orientation
pose
MPJPE
MPVPE
[Wang et al., 2021a]
7.65
9.38
44.52
242.50
222.13
HUMANISE pre-trained
6.54
8.80
42.54
222.65
201.92
6 Conclusion
In this work, we propose a large-scale and semantic-rich HSI dataset, HUMANISE. It contains diverse and physically plausible interactions equipped with language descriptions. HUMANISE enables a new task: language-condingg conditiood human motion generation in 3D scenes. We further design a lece--andlannguge conditioned generative model that produces diverse and semantically consistent human motions based on HUMANISE.
I Limitations Our work is primarily limited to two aspects. (i) The human motions are relatively short since action-specific motions usually last within seconds. A large-scale, long-duration HSI dataset is still in need for the scene understanding community. (ii) The problem of language grounding in 3D scenes is critical for our task but far from being solved, as also revealed in recent 3D grounding works [Achlioptas et al., 2020; Thomason et al., 2022]. A robust grounding module could help generate more meaningful human motions based on language description.
Acknowledgments and disclosure of funding We sincerely thank Chao Xu and Nan Jiang for their valuable advice on this project. We would like to thank the anonymous reviewers for their constructive comments. Z. Wang, Y. Chen, T. Liu, and S. Huang were supported in part by the National Key R&D Z. Wang and Liang were supported in part by the National Natural Science Foundation of China (NSFC) (No.62172043).
Program of China (2021ZD0150200)
10

---

References
Achlioptas, P., AbdeIrehcem, A., Xia, F., Elhoseiny, M., and Guibas, L. (2020). Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. In European Conference Computer Vision (ECCV). 2, 3, 5, 10
Ahuja, C. and Morency, L.-P. (2019). Language2pose: Natural language grounded pose forecasting. In International Conference on 3D Vision (3DV).
Anderson, P., Wu, Q.. Teney, D., Bruce, J., Johnson, M., Sünderhauf, N., Reid, I., Gould, S., and Van Den Hengel, A. (2018). Assion-and-lagguage navigation: Interpreting visually-groundel navigation instructions in real environments. In Conference on Computer Vision and Pattern Recognition (CVPR). 3
Azuma, D., Miyanishi. T., Kurita, S., and Kawanabe, M. (2021). Scanqa: 3d question answering for spatial scene understanding. arXiv preprint arXiv:2112.10482
Barsoum, E., Kender, J., and Liu, Z. (2018). Hp-gan: Probabilistic 3d human motion prediction via gan. In Conference on Computer Vision and Pattern Recognition (CVPR). 3
Bentley, J. L. (1975). Multidimensional binary search trees used for associative searching. Communi- cations of the ACM, 18(9):509-517. 4
Cao, Z., Gao, H., Mangalam, K., Cai, Q.-.., Vo, M., and Malik, J. (2020). Long-term human motion prediction with scene context. In European Conference Computer Vision (ECCV). 2, 3,5
Chao, Y.W.,, Yang, J., Chen, W., and Deng, J. (2019). Learning to sit: Synthesizing human-chair interactions via hierarchical control. arXiv preprint
3
Chen, D. Z., Chang, A. X., and Nießner, M. (2020). Scanrefer: 3d object localization in rgh-d scans using natural language. In European Conference on Computer Vision (ECCV). 3,
Chen, Y., Huang, S., Yuan, T., Qi, S., Zhu, Y., and Zhu, S.-C. (2019). Holistic++ scene understanding: Single- 3d holistic scene parsing and human pose estimation with human-objes interaction and physical commonsense. In International Conference on Computer Vision (ICCV). 3
Chen, Y., Li, Q., Kong, D., Kei, L., Gao, T., Zhu, Y., and Huang, S. (2021). Yourefit: Embodied reference understanding with language and gesture. International Conference on Computer Vision (ICCV). 3
Dai, A., Chang. A. X., Savva, M., Halber, M., Funkhouser, T., and Nießner, M. (2017). Scannet: Richly-annotater 3d reconstructions of indoor scenes. In Conference on Computer Vision and Pattern Recognition (CVPR). 3,
Das, A., Datta, S., Gkioxari, G.. Lee, S., Parikh, D., and Batra, D. (2018). Embodied question answering. In Conference on Computer Vision and Pattern Recognition (CVPR). 3
Gan, C., Zhang, Y., Wu, J., Gong, B., and Tenenbaum, J. B. (2020), Look, listen, and act: Towards audio-visual embodied navigation. In International Conference on Robotics and Automation (ICRA).
Ghosh, A., Cheema, N., Oguz, C., Theobalt, C., and Slusallek, (2021). Synthesis of compositional animations from textual descriptions. In International Conference on Computer Vision (ICCV).
Gordon, D., Kadian, A., Parikh, D., Hoffman, J., and Batra, D. (2019). Splitnet: Sim2sim and task2task transfer for embodied visual navigation. In Conference on Computer Vision and Pattern Recognition (CVPR).
Guo, C., Zuo, X., Wang, S., Zou, S., Sun, Q., Deng, A., Gong, M., and Cheng, L. (2020). Ac- tion2motion: Conditioned generation of 3d human motions. In International Conference on Multimedia. 3
Hassan, M., Ceylan, D., Villegas, R., Saito, J., Yang, J., Zhou, Y., and Black, M. J. (2021a). Stochastic scene-aware motion prediction. In International Conference Computer Vision (ICCV). 2,

---

Hassan, M., Choutas, V., Tzionas, D., and Black, M. J. (2019). Resolving 3d human pose ambiguities with 3d scene constraints. In International Conference on Computer Vision (ICCV). 3, 5, 10
Hassan, M., Ghosh, P., Tesch, J. Tzionas, D., and Black, M. (2021b). Populating 3d scenes by learning human-scene interaction. In Conference on Computer Vision and Putlern Recognition (CVPR).
Hong, Y., Li, Q., Zhu, S.-C., and Huang, S. (2021). Vigrammar: Grounded grammar induction of vision and language. In International Conference on Computer Vision (ICCV). 3
Huang, C.-H. P., Yi, Höschle, M., Safroshkin, M., Alexiadis, T., Polikovsky, S., Scharstein, D., and Black, M. J. (2022). Capturing and inferring dense full-body human-scene contact. In Conference Computer Vision and Pattern Recognition (CVPR). 3
lonescu, C., Papava, D., Olaru, V., and Sminchisescu, C. (2013). Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. Transactions on Pullern Analysis and Machine Intelligncce (TPAMI), (6667):3325-1333 7
Jia, B., Chen, Y., Huang, S., Zhu, Y., and Zhu, S.-c. (2020). Lemma: A multi-view dataset for learning multi-agent multi-task activities. In European Conference on Computer Vision (ECCV).
Kenton, J. D. M.-W. C. and Toutanova, L. K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics; Human Language Technologies (NAACL-HLT). 5,7
Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint
Kingma, D. P. and Welling, M. (2013). Auto-encoding variational bayes. arXiv preprint 1rxiv:1312.6114.
Li, C., Zhang, Z., Lee, W. and Lee, G. H. (2018). Convolutional sequence to sequence model for human dynamics. In Conference on Computer Vision and Pattern Recognition (CVPR). 3
Li, Y., Chen, X., Zhao, H., Gong, J., Zhou, G., Rossano, F., and Zhu, Y. (2022). Understanding embodied reference with touch-line transformer. arXiv preprint arXiv:2210.0566. 3
Liang, W., Liu, J., Lang, Y., Ning, B., and Yu, L.-F. (2019). Functional workspace optimization via learning personal preferences from virtual experiences. IEEE Transactions on Visualization and Computer Graph (TVCG), 1335)1883
Lin, X. and Amer, M. R. (2018). Human motion modeling using dvgans. arXiv preprint arXiv: 10652. 3
Mahmood, N., Ghorbani, N., Troje, N. F., Pons-Moll, G., and Black, M. J. (2019). Amass: Archive of motion capture as surface shapes. In International Conference on Computer Vision (ICCV). 2,
Pavlakos, G., Choutas, V., Ghorbani, N., Bolkart, T., Osman, A. A., Tzionas, D., and Black, M. J. (2019). Expressive body capture: 3d hands, face, and body from a single image. In Conference on Computer Vision and Pattern Recognition (CVPR). 5
Petrovich, M., Black, M. J., and Varol, G. (2021). Action-conditioned 3d human motion synthesis with transformer vac. In International Conference on Computer Vision (ICCV). 3, 6,9
Punnakkal, A. R., Chandrasekaran, A., Athanasiou, N., Rairos-Rammirz, A., and Black, M. J. (2021). Babel: Bodies, action and behavior with english labels. Conference on Computer Vision and Pattern Recognition (CVPR). 5.5
Qi, C. R., Yi, L., Su, H., and Guibas, J. (2017). Pointnet++: Deep hicrarchical feature learning on point in metric space. In Advances in Neural Information Processing Systems (NIPS). 7
Roh, J., Desingh, K., Farhadi, A., and Fox, D. (2022). Languagerefer: Spatial-language model for 3d visual grounding. In Conference Robot Learning (CoRL). 3.9
12

---

Savva, M.. Chang, A. X., Hanrahan, P., Fisher, M., and Nießner, M. (2016). Pigraphs: learning interaction snapshots from observations. ACM Transactions on Graphies (TOG), 35(4):1-12. 3,
Sohn, K., Lee, H., and Yan, X. (2015). Learning structured output representation using deep conditional generative models. Advances in Neural Information Processing Systems (NIPS), 28.
Starke, S., Zhang, H., Komura, T., and Saito, J. (2019). Neural state machine for character-scene interactions. ACM Transactions on Graphics (TOG), 38(6):209-1. 2,
Taheri, O., Ghorbani, N., Black, M.J., and Tzionas, D. (2020). Grab: A dataset of whole-body human grasping of objects. In European Conference on Computer Vision (ECCV). 3
Thomason, J., Shridhar, M., Bisk, Y., Paxton, C., and Zettlemoyer, (2022). Language grounding with 3d objects. In Conference on Robot Learning (CoRL). 3, 10
Wang, H., Liang W., and Yu, L.-F. (2020). Scene mover: automatic move planning for scene arrangement by deep reinforcement learning. ACM Transactions on Graphics (TOG), 39(6):1-15.
Wang. J., Xu, H., Xu, J., Liu, S., and Wang. X. (2021a). Synthesizing long-term 3d human motion and interaction in 3d scenes. In Conference on Computer Vision and Pattern Recognition (CVPR). 3, 7, 10
Wang, J., Yan, S., Dai, B., and Lin, D. (2021b). Scene-aware generative network for human motion synthesis. In Conference on Computer Vision and Pattern Recognition (CVPR). 2, 3
Wang, Z., Chen, L.., Rathore, S., Shin, D., and Fowlkes, C. (2019). Geometric pose affordance: 3d human pose with scene constraints. arXiv preprint arXiv: /905.07718.
Xie, K., Wang. T., Iqbal, U., Guo, Y., Fidler, S., and Shkurti, F. (2021). Physics-based human motion estimation and synthesis from videos. In International Conference Computer Vision (ICCV).
Ye, S., Chen, D., Han, S., and Liao, J. (2021). 3d question answering. arXiv preprint arXiv:2112.08359 3
Yi, H., Huang, C.-H. P., Tzionas, D., Kocabas, M., Hassan, M., Tang, S., Thies, J., and Black, M. J. (2022). Human-aware object placement for visual environment reconstruction. In Conference on Computer Vision and Pattern Recognition (CVPR). 2
Yuan, Y. and Kitani, K. (2020). Dlow:
Diversifying latent flows for diverse human motion prediction. In European Conference on Computer Vision (ECCV). 3,
Zhang. S., Zhang, Y., Ma, Q., Black, M. and Tang, S. (2020a). Generating person-scene scene interactions in 3d scenes. In International Conference on 3D Vision (3DV). 2, 3
Zhang, Y., Black, M. J., and Tang. S. (2021). We are more than our joints: Predicting how 3d bodies move. In Conference on Computer Vision and Pattern Recognition (CVPR). 6,
Zhang, Y., Hassan, M., Neumann, H., Black, M. J., and Tang, S. (2020b). Generating 3d people in scenes without people. In Conference on Computer Vision and Pattern Recognition (CVPR). 2,
Zhao, H., Jiang, L.,, Jia, J., Torr, H., and Koltun, V. (2021). Point transformer. In International Conference on Computer Vision (ICCV). 5,7
Zhou, Y., Barnes, C., Lu, J., Yang, J., and Li, H. (2019). On the continuity of rotation representations in neural networks. In Conference on Computer Vision and Pattern Recognition (CVPR). 5
13


Reviews:
[{"review_id": "rDyXig_RTv", "reviewer": "Reviewer_1Evw", "summary": "This work proposes a new dataset containing paired human motion, scene semantic labels, and motion language annotation. Its main contribution lies in an automatic pipeline for generating human-scene interactions that are both semantically and language labeled. Leveraging a large-scale motion dataset, semantically labeled 3D scenes, rule-based instruction generation, and collision/contact constraints, this dataset contains better quality motion and human-scene interactions than previous video-based datasets (e.g. PROX). Leveraging this dataset, a generative model is proposed to tackle the new task, language and scene-conditioned 3D human motion generation. ", "questions": "**Dataset Generation and motion diversity** \n\n- What is the motion diversity and number of samples for each action? BABEL provides action labels for the AMASS dataset, but for actions such as walking and sitting up and down, there are more samples than lying on beds. For the 4k motion sequences, are they all motion sequences of different action/motion? What is the data distribution among the actions?\n\n**Qualitative Results**\n\n- In the provided video, at timestamp 6:48, the right bottom motion does not correspond to any meaningful human motion.\n- Quantitative comparison with PROX is not entirely fair and does not really provide additional insight. The motion sequences from PROX are recovered from optimization-based methods and is not for the purpose of human motion generation (more for pose estimation). It is bound to have lesser quality than MoCap sequences from AMASS dataset.", "limitations": "Yes, the authors have addressed the limitations adequately. ", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n\n**Automatic data generation**\n\n- The main strength of this work lies in the proposed dataset and automatic data generation pipeline. The idea of using existing MoCap data from AMASS and scene scans to create paired human scene interaction is intuitive and interesting. Such a pipeline does not rely on any additional data capture equipment and can help reuse existing datasets for HSI-related tasks. The proposed contract and collision constraints can largely alleviate the violation of physical constraints in this automatic data generation framework.\n\n**New Task: scene and language conditioned motion generation** \n\n- While language/action-conditioned human motion generation [1,2], scene-conditioned motion generation [3], and goal-conditioned motion generation [4] has been separately studied to a certain extent, language ***and*** scene-conditioned human motion generation has not been explored as much due to the lack of annotated dataset. All the challenges that separately exist in the above tasks will be more apparent in these combined tasks (motion realism, physical validity, multi-modal nature of human motion). This paper proposed a general baseline for further studying this interesting and impactful task.\n- The proposed VAE network, though similar to prior arts in conditioned human motion generation [1,2] does serve as a decent baseline for future works in this task. The auxiliary task loss is a viable approach to gain better performance.\n\n## Weakness:\n\n**Dataset Creation Methodology:**\n\n- While the created dataset is certainly useful for future HSI research, the limited action types and motion diversity (see in questions) takes away some of the novelty. The proposed framework can be viewed as fitting a scene for existing motion sequences, and largely relies on the action labels for matching the motion sequences. It is more interesting to see how diverse the **generated** motion from learning-based methods, in the hope that learning-based methods and learn from a few interaction samples and actually learn affordance.\n\n**Auxiliary task loss:**\n\n- While the auxiliary task is effective in boosting motion reconstruction performance, it takes away the language conditioning and more or less makes the task degenerates into an action classification and goal-reaching task.\n\n**Qualitative Results:**\n\n- More qualitative evaluation is needed to fully evaluate the quality of the dataset and the generative methods. Since motion is best seen in videos, it is important to include more quantitative results for both the dataset and the generative method for better evaluation. It is easy to pick sequences that fit the current text description, but the overall dataset quality is hard to judge based on a handful of samples.\n\n**Motion Diversity:**\n\n- For a motion generation task, it is important to include evaluation metrics on motion diversity. While the included metrics such as goal distance and action score are important for scene and language conditioning, given the existence of the auxiliary task loss, the proposed method can easily memorize a few motions that match goal and actions.\n\n## Small issues:\n\n- L2: mediocore “characteristics”\n- L87: study HSI related topics\n\n[1] Petrovich, Mathis, Michael J. Black and Gül Varol. “Action-Conditioned 3D Human Motion Synthesis with Transformer VAE.” *2021 IEEE/CVF International Conference on Computer Vision (ICCV)* (2021): 10965-10975.\n\n[2] Ahuja, Chaitanya and Louis-Philippe Morency. “Language2Pose: Natural Language Grounded Pose Forecasting.” *2019 International Conference on 3D Vision (3DV)* (2019): 719-728.\n\n[3] Cao, Zhe, Hang Gao, Karttikeya Mangalam, Qi-Zhi Cai, Minh Vo and Jitendra Malik. “Long-term Human Motion Prediction with Scene Context.” *ArXiv* abs/2007.03672 (2020): n. pag.\n\n[4]Hassan, Mohamed et al. “Stochastic Scene-Aware Motion Prediction.” *2021 IEEE/CVF International Conference on Computer Vision (ICCV)* (2021): 11354-11364.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "cudd8ExUfO4", "reviewer": "Reviewer_yyeu", "summary": "The key motivation behind this work is animating human body motion in affordance with the environment ( in this case, 3d indoor scenes), where these animations are grounded semantically in natural language. A precursor to this challenging task is the curation of a large-scale dataset with aligned 3d scenes, human motion and language. The authors have made contributions in the curation of such a dataset named HUMANISE, and an approach to generated diverse and semantically consistent human motions given the 3d scene and language.", "questions": "- L123-124: How are the interesting target objects sampled based on the action label?\n- Rest of the questions are in the previous section", "limitations": "The limitations have been discussed at length in the paper, barring one on trade-off between low-level nuances and data curation simplicity (more in the weaknesses section). ", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- Generating animation in affordance with environments is a useful and challenging problem. How a person behaves in the environment can depend completely on (1) high level conditions such as how the objects are arranged in the environment and also on (2) low-level conditions such as the subtle differences between the size of the chair, or the hand placement depending on where a person sits on a big couch. Addition of language to this challenging problem is a welcome addition and simplicity of the data curation is commendable. I hope that the data collection scripts would be released for the community to use.\n- Animation generation in context of scenes and language is complex. To that end, both quantitative and qualitative experiments were performed which make the analysis more solid.\n\n### Weaknesses\n- The descriptions are generated using a fixed template and four actions. That can be quite limiting to the diversity of the possible sentences, especially in out-of-domain language contexts. How does the model behave in those scenarios\n* Consider the example about \"sit on the armchair near the desk\" example used in the Introduction. While the semantics of finding the right chair and sitting down on it are the high level conditions been looked after, low level nuances (discussed in [1]) are ignored here such as the positions of the hands. Are they on the armrests or are they on the table or something else? \n* This brings me to a concern about the construction of the dataset. As existing animations are aligned with 3d scenes, low-level nuances will likely be incorrect. While I understand that this is a trade-off of simple data curation vs low-level accuracy, how much do we lose when we don't care about these low-level nuances?\n* How is the training and test split done? For example, consider \"sit on the coffee table\" in the train set. Is there a semantically similar sentence in the test set such as \"sit on the coffee table near the couch\"? If so, I would be curious to see some analysis of cases where this is not true because they can be a plausible edge-case (or out-of-domain scenario).  \n* I would like to put forth one argument (not necessarily a weakness) on which I hope to have a constructive discussion with the authors. Let's assume we only care about the high-level semantics. This reduces the problem of first finding the target object (from the 3d scene and language), followed by figuring out the correct action to perform (from the language) and finally use these two pieces of information to generate a plausible animation (as discussed in the model section of the paper). Finding the target object using natural language and 3d scenes is a relatively well studied problem. Determining the action from the language is relatively straightforward given that the language sentences are template based and there are only 4 actions. And finally target based animation is also a well-studied problem in the graphics community. Where does the technical novelty of this paper lie? Are the authors positioning the paper that combines these three ideas to solve a more complex problem, or is there something I am missing out?\n\n#### Minor Suggestions\n- In figure 4 column 2, where is the TV. Some of the images are not very clear to make out the details.\n- Table 2: the \\hline is probably intended between walk and w/o self-attn.\n\n[1] - Starke, Sebastian, et al. \"Neural state machine for character-scene interactions.\" _ACM Trans. Graph._ 38.6 (2019): 209-1.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NjfbqFkRMft", "reviewer": "Reviewer_vzPA", "summary": "The paper focuses on human-scene interaction, and the authors propose a new dataset and a new task of language + scene conditioned synthesis of 3D human motion. Given a 3D scene as RGB point clouds and a language description (“sit on the couch”), the goal is to synthesize realistic and plausible motion in the given scene. \n\nThey introduce the HUMANISE dataset, which contains (4892) AMASS 3D motion sequences aligned with (512) ScanNet indoor scenes. The authors do motion alignment of AMASS sequences with the scenes using collision and contact constraints. The language descriptions for these aligned motion sequences are generated using templates.  They propose a reasonable encoder-decoder model to conditionally synthesize human motion sequences and provide clear ablations to back their design.", "questions": "Given the dataset, I’m curious to hear the authors' thoughts on how they expect the community to build on it. \n1) Difficulty or quality of the dataset: It seems like the dataset is not particularly challenging regarding both language description and length of motions. Given this, the models trained on HUMANISE may never generalize to novel descriptions & scenes. \n2) How the dataset generation process is a viable solution to generating rich HSI datasets with language descriptions? This seems to be one possible approach, but I’m not sure it’s a promising one. Finding a motion sequence recorded in isolation to have rich interactions in an unknown scene seems unlikely. To solve the task in the right way, one might actually need to capture motion occurring in scenes (which is a very challenging task in itself). \n\nMinor fixes:\n- Table 1 caption: HUAMNISE → HUMANISE\n- Line 114: (i.e., ScanNet [Dai et al., 2017] → (i.e., ScanNet [Dai et al., 2017])\n", "limitations": "Authors have discussed limitations and societal impact of their work.\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The problem of 3D motion generation conditioned on language and scene is an interesting and important one. HUMANISE is the first dataset and approach to tackling this. \n- The qualitative and quantitive results of the proposed model are good. \n- The paper is also clearly written. All the arguments and technical decisions made in the paper are well-backed by reasoning or experiments. \n\nWeakness:\n\nMy primary concern is with the dataset itself and the methodology of creating the dataset. \n\nFirst, most sequences are very short (averaging < 2 sec at 30fps). There are only four action types considered (walk/stand/sit/lie), with “lie” being < 4% of the data. The other action types don’t have rich interactions with the environment (“walk to the table”, “sit on the couch”). Similarly, the language description is not extremely interesting as it mostly composes an action type + object type. The power of language would be in expressing more interesting compositions and interactions. \n\nGiven this, I feel the representation in Table 1 is not giving a complete picture. Yes, HUMANISE has many more scenes and clips, but the variation in diversity should also be stated. \n\nFundamentally these limitations might be due to trying to align a motion to a 3D scene it doesn’t belong. This greatly decreases the possible motions that could be aligned and the possible object and scene interactions that the motion + language can describe (other than walk/sit/stand/jump etc.)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7oPleudlLtP", "reviewer": "Reviewer_bBY2", "summary": "This paper released a synthesized dataset for language-conditioned human motion generation in 3D scenes, which aligns the existing 3D human motion datasets in 3D scene datasets to synthesize human motions in 3d scene. Besides, this paper proposes a baseline model based on conditional VAE for the language-conditioned human motion generation in 3D scene. This topic is promising and meaningful.", "questions": "1) Can you report some experimental comparisons of your model with other conditional generation models? Why do you use the CVAE model, and what are the effects of other conditional generation models? It is insufficient to judge the advanced property of the CVAE model only by ablation studies.\n2) How about the Params and the GFLOPs of your model?\n3) Do the dataset and the model proposed in this paper adequately consider the body-scene interactions? In other words, are the motions in the dataset or generated by model appropriate to the environment? For example, human sitting on the couch and sitting on the toilet may have different postures. Simply applying the same motion to different environments may result in unreasonable actions.\n", "limitations": "Authors are advised to provide more and longer-sequence video demonstrations for the proposed dataset.\nThe body-scene interactions should be considered and refined in the proposed dataset for plausible human motion generation. \n", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nA large-scale and semantic-rich synthetic HSI dataset is proposed in this work, which enables a new task: language-conditioned human motion generation in 3D scene.\n\nWeaknesses:\n1) Though this is the large-scale and semantic-rich synthetic HSI dataset, the number of the action categories is limitted. There are only four interacitve indoor actions, i.e sit, stand up, lie down, and walk. The rich human-object interactions like opening a refrigerator, close the door are not included;\n2) The sequence length of each sample in the proposed dataset seems too short to express rich semantic information;\n3) As the motion dataset are aligned, not actually acted, in the 3D scene, some motion in the interaction cannot well presented in the dataset, for example torch a chair before siting on it;\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept