Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work proposes a two-stage human motion forecasting framework that explicitly models human-scene contact. It proposes to decouple the problem into two stages: past pose conditioned contact forecasting, and contact-conditioned pose forecasting. Specifically, it proposes to use a Discrete Cosine Transform (DCT) based network to predict contacts based on past contact, human pose, and scene point clouds. After the future contact is predicted, a series of networks are used to predict the human’s global translation, rotation, and body joint positions. 
strengths_and_weaknesses: ## Strength

**Explicit Contact Modelling**

- The proposed two-stage pipeline is intuitive and performs well in the context of human motion prediction. Contact and physical constraints play an important role in governing human motion and based on contact human motion are a lot less ambiguous. The idea of explicitly predicting the future contact of humans in a known scene to guide motion prediction is interesting.

**Performance compared to State-of-the-art**

- The proposed method outperforms SOTA methods in the motion prediction task.

## Weakness

**Novelty in lieu of motion generation methods**

- Given the existence of methods such as SAMP [1], where human motion is generated based on path and scene context, the proposed framework has limited novelty. While the settings are slightly different (motion and interaction generation vs forecasting), the methodology is largely similar. The two-stage modeling has been largely explored (first generate goals or subgoals, then generate local motion), and this work mainly excels at better modality (explicit contact).

**Lack of generative modeling** 

- While human motion is multi-modal, the lack of generative and stochastic modeling means that the estimated human motion could be memorizing past observed interactions (especially in PROX and GTA-IM datasets, where the motion are largely similar).

**Lacking Qualitative Results**

- Since motion is better seen in videos, it would be better if more qualitative samples are provided.

[1] Hassan, Mohamed et al. “Stochastic Scene-Aware Motion Prediction.” 2021 IEEE/CVF International Conference on Computer Vision (ICCV) (2021): 11354-11364.
questions: It would be great if the authors could compare more closely with the motion generation literature (such as SAMP) and discuss differences. 
limitations: The authors have discussed limitations adequately.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes to tackle scene-aware 3D human motion forecasting by explicitly modeling the human-scene interactions, i.e., representing the contact between human body joints and scene points with a distance-based contact map. They also introduce a two-stage pipeline that first predicts the future contact map with the given motion history; then forecasts the future global translation and local poses. The proposed method can predict more physically plausible motions and avoid artifacts such as “ghost motion”.

**Main Contributions:** This paper proposes a contact map representation explicit modeling human-scene interactions and propose a two-stage framework to forecast human motions with given motion histories and 3D scenes.
strengths_and_weaknesses: - Strengths:

1.	This paper explicitly models the human-scene interactions with a contact map, which measures the distance between the human joints and the scene points. The contact map enables more physically plausible and realistic human-scene interaction generation.
2.	The proposed two-stage prediction pipeline disentangles contact prediction and human pose forecasting, thus capable of explicitly encouraging consistency between human motions and contact points in given 3D scenes.

- Weaknesses:

1. The contact map computed between joints and scene points is too coarse. It is more plausible to compute the contact map between body surface vertices and scene points because the human body surface rather than joints contact with the environment.
2. The contact map has been widely used in grasp generation tasks[ Jiang et al., ICCV2021; Brahmbhatt et al., CVPR2019 ]. I think you should discuss the contact map in the literature review. And the idea of using the contact map to model human-scene interactions is not very appealing.
questions: 1. How do you use the motion features extracted by GRU in PVCNN? The PVCNN processes the 3D point cloud at point level, but the motion feature is the feature of the given motion history. Additionally, the notation $\mathbf{X}$ in Eq.6 is inaccurate, as $\mathcal{F}$ takes as input the latent feature of motion history instead of past human poses $\mathbf{X}$. Similar problems in Eq.9 and Eq.10.

2. The partial problem setting follows GTA-IM, which forecasts future human motions with a multi-stage pipeline. The quantitative evaluation should include the comparison between the proposed method and the variant of the method in GTA-IM.

3. How about the model's ability of forecasting long-term motions (more than 5 seconds)? The current setting only predicts the future motions in 2 seconds which is too short for humans to have distinguishable movements.
limitations: When considering the contact between human and scene, I think the shape-based human body representation, e.g., SMPL and marked-based representation, is more reasonable to model the contact between body surface and scene. This representation can produce a more fine-grained contact map, which thus can model more realistic details of human-scene interactions.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper promotes explicit contacts modeling when handling the challenging scene-aware human motion forecasting problem,

To achieve that, a dense scene-joint distance maps are utilized to densely model human dynamics when interacting with the static scenes,
followed by a novel discrete pre-processing with DCT to get sparse principal features, while also enabling residual motion prediction of
the contacts point in frequency domain. 

A two-stage pipeline is assembled together to get better future motion predictions for all 3D body joints, with stage 1 combining GRU-based dynamics modeling and PVCNN-based 3D scene encoding for sequential contact distance maps prediction, and stage 2 carrying out contacts-guided sequential motion forecasting in stage 2.

strengths_and_weaknesses: **Strengths**
[Novelty]
As mentioned above, dynamic contacts modeling and the way to use it are the shining points in this paper, in which the authors leverage an effective point-joint distance field followed by a novel frequency transformation(DCT) to better capture sparse smooth motion patterns. Per-joint closest scene point is used to better guide the motion prediction. This design 

[Completeness]
Three baselines and their proposed methods are validated in two common datasets, including both real and synthetic ones. They also provide both quantitative and qualitative results, including a video demo in the supp. 

[Effectiveness]
The author gets consistently better motion forecasting results(global and local) on long-term motion predictions on two benchmarks compared to all the baselines.

**Weaknesses**
1. Firstly, even though the whole method seems to be novel, I do not think that line 49-54 is carefully written to
well capture the whole work, in (ii) the two-stage pipeline itself should not be considered as a key contribution,
it has overlaps with (i), also I did not see any unique technical contribution statements when tackling this conditional
motion synthesis task. The author needs a better clarification on the contributions.

2. Even though so many efforts(DCT/IDCT transformation, GRU models, PVCNN) have been conducted to get better
contact maps forecasting, it seems that only one nearest scene point is selected and used per joint point, I am
wondering whether such a heavy pipeline is really necessary. 

3. The result in table 1 seems to show that the proposed method does not perform well enough on short-term predictions, even though it performs consistently better on predictions >= 1s

questions: 1. Related to Weakness 2, is this single closest contact priors
would be enough to guide the motion forecasting network? The single contact point looks like a special form of
signed distance function to me.

2. The motion forecasting network is still a very implicit design when leveraging contact points,
like for the global translation, the author seems to want the MLP module to learn root motion implicitly from all the closest points,
very similar to a learnable center point. If this is possible, I am wondering how this implicit design would be better than
a simple xyz averaging over all closest points belonging to the feet-nearby joints (maybe plus a certain offset).
Plus, it is not clear that in Figure 2, how does GRU take all three inputs to update the local joints offsets prediction?

3. How does number of scene points affect the prediction performance? Do we need to select all the scene points in order to train the stage 1 model? The author does not mention the computation efficiency clearly in the implementation section.
limitations: See above
Minor: Line 179, should be '...are then fed into...'
review_title: null

Meta-review: metareview: Three expert reviewers have recommended accepting the paper after the discussion period.  Reviewers like the overall idea and framework.  The AC agrees and recommends acceptance.  Please carefully revise the paper based on the reviews. 
confidence: Less certain
Decision: Accept