Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper focuses on human-scene interaction, and the authors propose a new dataset and a new task of language + scene conditioned synthesis of 3D human motion. Given a 3D scene as RGB point clouds and a language description (“sit on the couch”), the goal is to synthesize realistic and plausible motion in the given scene. 

They introduce the HUMANISE dataset, which contains (4892) AMASS 3D motion sequences aligned with (512) ScanNet indoor scenes. The authors do motion alignment of AMASS sequences with the scenes using collision and contact constraints. The language descriptions for these aligned motion sequences are generated using templates.  They propose a reasonable encoder-decoder model to conditionally synthesize human motion sequences and provide clear ablations to back their design.
strengths_and_weaknesses: Strengths:
- The problem of 3D motion generation conditioned on language and scene is an interesting and important one. HUMANISE is the first dataset and approach to tackling this. 
- The qualitative and quantitive results of the proposed model are good. 
- The paper is also clearly written. All the arguments and technical decisions made in the paper are well-backed by reasoning or experiments. 

Weakness:

My primary concern is with the dataset itself and the methodology of creating the dataset. 

First, most sequences are very short (averaging < 2 sec at 30fps). There are only four action types considered (walk/stand/sit/lie), with “lie” being < 4% of the data. The other action types don’t have rich interactions with the environment (“walk to the table”, “sit on the couch”). Similarly, the language description is not extremely interesting as it mostly composes an action type + object type. The power of language would be in expressing more interesting compositions and interactions. 

Given this, I feel the representation in Table 1 is not giving a complete picture. Yes, HUMANISE has many more scenes and clips, but the variation in diversity should also be stated. 

Fundamentally these limitations might be due to trying to align a motion to a 3D scene it doesn’t belong. This greatly decreases the possible motions that could be aligned and the possible object and scene interactions that the motion + language can describe (other than walk/sit/stand/jump etc.)
questions: Given the dataset, I’m curious to hear the authors' thoughts on how they expect the community to build on it. 
1) Difficulty or quality of the dataset: It seems like the dataset is not particularly challenging regarding both language description and length of motions. Given this, the models trained on HUMANISE may never generalize to novel descriptions & scenes. 
2) How the dataset generation process is a viable solution to generating rich HSI datasets with language descriptions? This seems to be one possible approach, but I’m not sure it’s a promising one. Finding a motion sequence recorded in isolation to have rich interactions in an unknown scene seems unlikely. To solve the task in the right way, one might actually need to capture motion occurring in scenes (which is a very challenging task in itself). 

Minor fixes:
- Table 1 caption: HUAMNISE → HUMANISE
- Line 114: (i.e., ScanNet [Dai et al., 2017] → (i.e., ScanNet [Dai et al., 2017])

limitations: Authors have discussed limitations and societal impact of their work.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper released a synthesized dataset for language-conditioned human motion generation in 3D scenes, which aligns the existing 3D human motion datasets in 3D scene datasets to synthesize human motions in 3d scene. Besides, this paper proposes a baseline model based on conditional VAE for the language-conditioned human motion generation in 3D scene. This topic is promising and meaningful.
strengths_and_weaknesses: Strengths:
A large-scale and semantic-rich synthetic HSI dataset is proposed in this work, which enables a new task: language-conditioned human motion generation in 3D scene.

Weaknesses:
1) Though this is the large-scale and semantic-rich synthetic HSI dataset, the number of the action categories is limitted. There are only four interacitve indoor actions, i.e sit, stand up, lie down, and walk. The rich human-object interactions like opening a refrigerator, close the door are not included;
2) The sequence length of each sample in the proposed dataset seems too short to express rich semantic information;
3) As the motion dataset are aligned, not actually acted, in the 3D scene, some motion in the interaction cannot well presented in the dataset, for example torch a chair before siting on it;

questions: 1) Can you report some experimental comparisons of your model with other conditional generation models? Why do you use the CVAE model, and what are the effects of other conditional generation models? It is insufficient to judge the advanced property of the CVAE model only by ablation studies.
2) How about the Params and the GFLOPs of your model?
3) Do the dataset and the model proposed in this paper adequately consider the body-scene interactions? In other words, are the motions in the dataset or generated by model appropriate to the environment? For example, human sitting on the couch and sitting on the toilet may have different postures. Simply applying the same motion to different environments may result in unreasonable actions.

limitations: Authors are advised to provide more and longer-sequence video demonstrations for the proposed dataset.
The body-scene interactions should be considered and refined in the proposed dataset for plausible human motion generation.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work proposes a new dataset containing paired human motion, scene semantic labels, and motion language annotation. Its main contribution lies in an automatic pipeline for generating human-scene interactions that are both semantically and language labeled. Leveraging a large-scale motion dataset, semantically labeled 3D scenes, rule-based instruction generation, and collision/contact constraints, this dataset contains better quality motion and human-scene interactions than previous video-based datasets (e.g. PROX). Leveraging this dataset, a generative model is proposed to tackle the new task, language and scene-conditioned 3D human motion generation. 
strengths_and_weaknesses: ## Strength

**Automatic data generation**

- The main strength of this work lies in the proposed dataset and automatic data generation pipeline. The idea of using existing MoCap data from AMASS and scene scans to create paired human scene interaction is intuitive and interesting. Such a pipeline does not rely on any additional data capture equipment and can help reuse existing datasets for HSI-related tasks. The proposed contract and collision constraints can largely alleviate the violation of physical constraints in this automatic data generation framework.

**New Task: scene and language conditioned motion generation** 

- While language/action-conditioned human motion generation [1,2], scene-conditioned motion generation [3], and goal-conditioned motion generation [4] has been separately studied to a certain extent, language ***and*** scene-conditioned human motion generation has not been explored as much due to the lack of annotated dataset. All the challenges that separately exist in the above tasks will be more apparent in these combined tasks (motion realism, physical validity, multi-modal nature of human motion). This paper proposed a general baseline for further studying this interesting and impactful task.
- The proposed VAE network, though similar to prior arts in conditioned human motion generation [1,2] does serve as a decent baseline for future works in this task. The auxiliary task loss is a viable approach to gain better performance.

## Weakness:

**Dataset Creation Methodology:**

- While the created dataset is certainly useful for future HSI research, the limited action types and motion diversity (see in questions) takes away some of the novelty. The proposed framework can be viewed as fitting a scene for existing motion sequences, and largely relies on the action labels for matching the motion sequences. It is more interesting to see how diverse the **generated** motion from learning-based methods, in the hope that learning-based methods and learn from a few interaction samples and actually learn affordance.

**Auxiliary task loss:**

- While the auxiliary task is effective in boosting motion reconstruction performance, it takes away the language conditioning and more or less makes the task degenerates into an action classification and goal-reaching task.

**Qualitative Results:**

- More qualitative evaluation is needed to fully evaluate the quality of the dataset and the generative methods. Since motion is best seen in videos, it is important to include more quantitative results for both the dataset and the generative method for better evaluation. It is easy to pick sequences that fit the current text description, but the overall dataset quality is hard to judge based on a handful of samples.

**Motion Diversity:**

- For a motion generation task, it is important to include evaluation metrics on motion diversity. While the included metrics such as goal distance and action score are important for scene and language conditioning, given the existence of the auxiliary task loss, the proposed method can easily memorize a few motions that match goal and actions.

## Small issues:

- L2: mediocore “characteristics”
- L87: study HSI related topics

[1] Petrovich, Mathis, Michael J. Black and Gül Varol. “Action-Conditioned 3D Human Motion Synthesis with Transformer VAE.” *2021 IEEE/CVF International Conference on Computer Vision (ICCV)* (2021): 10965-10975.

[2] Ahuja, Chaitanya and Louis-Philippe Morency. “Language2Pose: Natural Language Grounded Pose Forecasting.” *2019 International Conference on 3D Vision (3DV)* (2019): 719-728.

[3] Cao, Zhe, Hang Gao, Karttikeya Mangalam, Qi-Zhi Cai, Minh Vo and Jitendra Malik. “Long-term Human Motion Prediction with Scene Context.” *ArXiv* abs/2007.03672 (2020): n. pag.

[4]Hassan, Mohamed et al. “Stochastic Scene-Aware Motion Prediction.” *2021 IEEE/CVF International Conference on Computer Vision (ICCV)* (2021): 11354-11364.
questions: **Dataset Generation and motion diversity** 

- What is the motion diversity and number of samples for each action? BABEL provides action labels for the AMASS dataset, but for actions such as walking and sitting up and down, there are more samples than lying on beds. For the 4k motion sequences, are they all motion sequences of different action/motion? What is the data distribution among the actions?

**Qualitative Results**

- In the provided video, at timestamp 6:48, the right bottom motion does not correspond to any meaningful human motion.
- Quantitative comparison with PROX is not entirely fair and does not really provide additional insight. The motion sequences from PROX are recovered from optimization-based methods and is not for the purpose of human motion generation (more for pose estimation). It is bound to have lesser quality than MoCap sequences from AMASS dataset.
limitations: Yes, the authors have addressed the limitations adequately.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The key motivation behind this work is animating human body motion in affordance with the environment ( in this case, 3d indoor scenes), where these animations are grounded semantically in natural language. A precursor to this challenging task is the curation of a large-scale dataset with aligned 3d scenes, human motion and language. The authors have made contributions in the curation of such a dataset named HUMANISE, and an approach to generated diverse and semantically consistent human motions given the 3d scene and language.
strengths_and_weaknesses: ### Strengths
- Generating animation in affordance with environments is a useful and challenging problem. How a person behaves in the environment can depend completely on (1) high level conditions such as how the objects are arranged in the environment and also on (2) low-level conditions such as the subtle differences between the size of the chair, or the hand placement depending on where a person sits on a big couch. Addition of language to this challenging problem is a welcome addition and simplicity of the data curation is commendable. I hope that the data collection scripts would be released for the community to use.
- Animation generation in context of scenes and language is complex. To that end, both quantitative and qualitative experiments were performed which make the analysis more solid.

### Weaknesses
- The descriptions are generated using a fixed template and four actions. That can be quite limiting to the diversity of the possible sentences, especially in out-of-domain language contexts. How does the model behave in those scenarios
* Consider the example about "sit on the armchair near the desk" example used in the Introduction. While the semantics of finding the right chair and sitting down on it are the high level conditions been looked after, low level nuances (discussed in [1]) are ignored here such as the positions of the hands. Are they on the armrests or are they on the table or something else? 
* This brings me to a concern about the construction of the dataset. As existing animations are aligned with 3d scenes, low-level nuances will likely be incorrect. While I understand that this is a trade-off of simple data curation vs low-level accuracy, how much do we lose when we don't care about these low-level nuances?
* How is the training and test split done? For example, consider "sit on the coffee table" in the train set. Is there a semantically similar sentence in the test set such as "sit on the coffee table near the couch"? If so, I would be curious to see some analysis of cases where this is not true because they can be a plausible edge-case (or out-of-domain scenario).  
* I would like to put forth one argument (not necessarily a weakness) on which I hope to have a constructive discussion with the authors. Let's assume we only care about the high-level semantics. This reduces the problem of first finding the target object (from the 3d scene and language), followed by figuring out the correct action to perform (from the language) and finally use these two pieces of information to generate a plausible animation (as discussed in the model section of the paper). Finding the target object using natural language and 3d scenes is a relatively well studied problem. Determining the action from the language is relatively straightforward given that the language sentences are template based and there are only 4 actions. And finally target based animation is also a well-studied problem in the graphics community. Where does the technical novelty of this paper lie? Are the authors positioning the paper that combines these three ideas to solve a more complex problem, or is there something I am missing out?

#### Minor Suggestions
- In figure 4 column 2, where is the TV. Some of the images are not very clear to make out the details.
- Table 2: the \hline is probably intended between walk and w/o self-attn.

[1] - Starke, Sebastian, et al. "Neural state machine for character-scene interactions." _ACM Trans. Graph._ 38.6 (2019): 209-1.
questions: - L123-124: How are the interesting target objects sampled based on the action label?
- Rest of the questions are in the previous section
limitations: The limitations have been discussed at length in the paper, barring one on trade-off between low-level nuances and data curation simplicity (more in the weaknesses section).
review_title: null

Meta-review: metareview: Paper was reviewed by four reviewers and received: 1 x Borderline Accept, 1 x Borderline Reject, 1 x Weak Accept and 1 x Accept. The general sentiment of reviewers was positive. Main identified concerns were with lack of diversity in the dataset and potential realism issues arising from construction of the dataset (placing pre-recorded motions into different scenes). Some of these concerns have been somewhat alleviated through the rebuttal. That said, [bBY2] remained concerned with the realism of interactions. AC agrees with [vzPA] that collecting "natural" real-world data for this problem would be difficult and laborious and that the proposed dataset can serve as a good bridge towards this ultimate goal and could be a stepping stone for future research. Therefore the decision is to Accept the paper. 
confidence: Certain
Decision: Accept