Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper describes a model for multi-person motion prediction. It is based on the encode-decoder architecture. Both are constructed with self-attention transformers. The model takes in the absolution 3D coordinates of body joints from multiple persons and predict the future movement of the joints in 3D space for a timespan of 1-3seconds. 

Additionally, the authors introduced several pactices in the model design including: 1) the use of the last seen frame in the observed as queries; 2) the use of DCT to encode joints; 3) The separation of a global encoder and a local encoder. The global encodes input of multiple persons while a local encoder encoder data from a single person. The model is then trained with both the reconstruction loss and a adversarial loss implemented via a Transformer based discriminator model. 

The authors conducted experiments on human motion sequence datasets. Improved results over compared baselines are observed. The authors further provided analysis on the query design and number of transformer layers used. Qualitative analysis of the prediction is also provided. 
main_review: ## Originality

As far as I know this method is new for the tasks of multiple person motion prediction. There is a related field of object trajectory prediction but in that task mostly the trajectories of centers of mass are concerned while in this work each joint's location needs to be predicted. 

## Quality
The model itself is described clearly. It achieves good improvement over the baselines. The authors have provided a few ablation studies of design choices. 

There are several things that are not assessed as listed below:
- whether the local range Transformer is necessary given that the global transformer already sees all input data. 
- the use of DCT as input encoding
- the combined prediction of absolute joint locations vs. center of mass trajectory + relative joint locations
Some of these items are claimed as motivation of design choices so they warrant a study. 

The description of the discriminator is a little bit vague. One questions not answered is why it is not designed as a similar architecture as the prediction part. In experiments, it is only giving marginal improvements on most of the datasets (Table 1/2/3). So this seems either a missing opportunity or deadweight in the design. The authors please clarify. 

 ## Significance
Being able to predict the motion of multiple person simultaneously is a nice and practical property of the proposed method. Based on this I feel it will be useful for practitioners. The idea of local vs. global in prediction is interesting but in the current form it lack thorough investigation. 

## Additional questions

I have the following additional questions: 

1. The MPJPE metric has no unit. Is it in mm, cm, m, or any other metric system?
2. Predicting future motion is by its natural non-deterministic. Here we are comparing the prediction with the "groundtruth", which is one possible instantiation among many possible futures. Is there any consideration in this aspect?
 
limitations_and_societal_impact: The authors has addressed the societal impact of the proposed method. 

I didn't see much discussion about the limitation of the work.
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: The paper proposes a learning framework based on transformers for 3D human body motion prediction considering the motion and interaction of other people in the scene. The proposed approach contains a local-range transformer encoder to encode each individual motion and a global-range transformer encoder for encoding social interactions. Then a transformer decoder is used to predict the future motion for each person. The results are evaluated on three datasets, CMU-Mocap, MuPoTS-3D, and 3DPW  and compared against few state-of-the-art methods. 
main_review: 1- Clarity: The main problem, motivation and the idea of the paper including its technical details are very clear. Though, the paper contains quite a few typos and grammatical errors/

2- Contribution: the paper does not offer any novel theoretical and practical contribution. The method uses some arbitrary architecture variants of transformer encoders followed by a transformer decoder to predict the 3D human body skeleton of each person. The idea of social interaction for predicting human motion is not new, even in the context of the human skeleton body pose (e.g. 2 and 3). Transformers has been already used to incorporate human interaction for the human trajectory forecasting problem, eg Yu et al, "Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction", ECCV 2020. 

3- Experiments: The results of the proposed method have only been compared against a few out of many existing frameworks on human body skeleton prediction and only tested on few arbitrary datasets. Also, no details for the implementation (or hyper-parameters) of the competing methods are provided in the main text or supp material file. It is also not clear how these methods are compared because the frameworks [43] and [44] are designed to predict human body skeleton motion only (no trajectory motion is considered or predicted), while the evaluated datasets and framework [2] consider both tasks jointly. I couldn't find any public implementation for the framework [2]. it is very important to elaborate the implementation and hyperparameters details of competing methods to ensure a fair comparison. I could also see there exist a benchmark named SoMoF which aims to provide a benchmark for the same problem this paper targeted.  This approach could be evaluated on this benchmark instead. Human body pose and motion are not only affected by the other human body motion but also the objects and obstacles in the scene. several human body motion forecasting frameworks have been already proposed to model these physical interactions e.g. Cao et al. Long-term human motion prediction with scene context. ECCV, 2020 and the comparison with these relevant works fare missing,
limitations_and_societal_impact: The authors attempted to address the limitation and potential negative social impact of their work on page 9.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a Transformer-based deep learning architecture to predict multi-person 3D motion trajectories. To better model the social influence of neighbors, both a local-range (individual motion) and a global-range (social interactions) Transformer encoders are used.
The proposed framework is benchmarked on several 3D motion prediction datasets, namely CMU-Mocap, Panoptic, MuPoTS-3D, and 3DPW (and some combinations of these). The results show an improvement w.r.t. the select baselines of 20-30%. From a qualitative point of view, the model predictions seem smoother and more natural compared to previous efforts. Furthermore, the model is able to predict many people at the same time, and different social groups appear to cluster together when examining the social attention scores.
main_review: The submission presents a state-of-the-art model for 3-D human motion trajectory prediction. More specifically, the model uses two encoders and one decoder based on Transformers and a discriminator consisting of fully-connected layers. I think the overall quality of the paper is good and both the architecture and results are systematically presented.

The comparison with previously published results is properly done, and the proposed method is able to outperform state-of-the-art performance by a noticeable margin. The two considered experimental settings, namely generalization and fusion, allow the reader to understand the impact of considering more than 2/3 people as well as the generalization capability of the model.

However, I have some concerns regarding several statements reported in the paper. For example:
- the authors states in the abstract that the model 'also generates diverse social interactions'. I think this aspect should be better investigated (or explained) since it is not clear from the qualitative and quantitative results. What does 'diverse' mean in this case? 'Different' among the people in the scene?
- Then, regarding the second experimental setting, the authors fuse several scenes from different datasets but they use absolute coordinates. How the different coordinates have been taken into account, if they are different? And, most importantly, people are involved in different actions and fusing them will
result in not natural interactions. Is my interpretation correct? 
- In Section 3, it is stated that 'x^k_n contains both the trajectory and the pose information'. It is not clear to me the difference since the dimension of this vector is '3J' and not '3J+1' as it should be with the addition of the trajectory information. 
- Absolute coordinates are then transformed into relative motion (x_(i+1) - x_i) in this work while, as stated, previous works use joint positions centered at the origin.  Is this not the same thing? If not, why?

Besides the above concerns, I report some minor comments below:
- Page 1, line 23: 'ma' should be 'may';
- Page 1, line 25: 'response' should be 'respond';
- Page 1, line 27: 'Such a model will need have the following'. This sentence should be correctly rephrased;
- Page 4, line 130: 'update' is repeated two times;
- Page 5, line 160: 'this single pose to as'. 'to' should be removed;
- Page 6, line 210: ', We sample' should be ', we sample';
- Page 7, line 272: 'cross' should be 'across';
- It would be better, in my opinion, to report the number of observed and predicted time-steps in section 3 (Method) since it is a very important detail of the task; 
- The authors perform experiments considering 9-15 people. It would be nice to see a qualitative example of this setting in the paper.
limitations_and_societal_impact: I believe limitations and potential negative societal impact of this work have been adequately addressed by the authors.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: - The paper proposes to tackle the problem of multi person 3D motion prediction. 
- Given the trajectories of multiple people in the scene, the model reasons about both long and short term context using separate transformer based encoders for both. 
- A transformer based decoder then takes a single pose at a time step for each person and uses encoded features to predict the future trajectory. 
- Experiments are performed on the CMU-Mocap, MuPoTs-3D and 3DPW dataset. and the results show improvements over state-of-the-art. 
- Qualitative visualizations show that the model can uncover behaviors, interactions and group people in crowds. 

main_review: Strengths:
- The problem is properly motivated. 
- Transformers for the problem, are natural given that it can handle variable length input and handle long range temporal dependencies. 
- The idea of combining global and local in the proposed way is interesting. Not all global context might be important for all pose sequences, the proposed approach helps tackle that. 
- The proposed approach shows considerable improvement over state-of-the-art approaches
- The prediction visualizations when groups of people were involved were quite interesting and the accompanying video in the supplementary was helpful. 
- The experimentation is thorough and includes a user study. 

Concerns:
- DCT : It is not clear to me why DCT was used for this task. The paper [44] uses DCT to encode temporal information and then works with the DCT bases for input to a graph model. In this paper, the transformer encoder already captures the temporal information. Was it merely an empirical observation (as given in the supplementary) ?
- L136-137 : Was the position embedding added to the concatenated embedding or to the output of the linear layer ? 
- Global-range transformer : Should you have an input dimension which differentiates embeddings of the different people from each other ? Spatial position encoding could potentially be useful for this but I am curious why that encoding was added after the encoder ?
- Will the calculation of SPE scale ? Since it requires computing distance between every pair. 
- While the user study does indicate a slight improvement compared to the others, the difference seems to be small. The GT values seem small too. Is that expected ?
- Did the authors experiment with any data augmentations ?

Other suggestions :
- Typo : L23 : ‘ma’ 
- Rephrase : L 120-121
- Typo : L134 : ‘contact’ -> ‘concatenate’ ?
- Rephrase : L167-168



limitations_and_societal_impact: - The authors have adequately discussed the limitations and societal impact of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: --- Summary

This paper proposes a model for multi-person 3D motion prediction. Different from the previous works, the proposed model uses the Transformer architecture and is able to better model the multi-person social interactions. The model consists of a local-range encoder to encode the motion of each individual person, a global-range encoder to encode the social interactions, and a transformer decoder to generate the trajectory predictions. And it also has a discriminator module to make the generated trajectories more natural. The proposed model is trained with both the reconstruction loss and the discriminator loss. The evaluations were performed on multiple datasets including CMU-Mocap, Panoptic, MuPoTS-3D, and 3DPW, and the proposed model outperforms the baselines by 20% ~ 30%.
main_review: --- Strengths

- The evaluation is thorough. The proposed model was compared against multiple state-of-the-art baselines on multiple datasets, and it was shown to outperform the baselines by a large margin. The authors also performed a few ablation studies to show that the proposed design components are necessary. There was also a user study performed with Amazon Mechanical Turk that compared the quality of the predictions.

- The paper is well-written.


--- Issues and suggestions

- It's interesting to have the user study that quantitatively compares the quality of the predictions, but all models compared (and including the ground-truth) seem to end up having very close scores, so maybe the design of this study can be improved?


--- Other comments

- Typo in Line 22: "ma".

limitations_and_societal_impact: Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a transformer-based model to predict future 3D poses of a group of people, given 3D poses from previous observations. The authors explicitly model local interactions (ie dependencies between poses at previous time steps for a given person) and global interactions (dependencies between different people in the scene), and obtain state-of-the-art results on 3 different datasets. The transformer model is an encoder-decoder type structure, where the present and past poses are encoded, and the query to the decoder is the pose at the current frame. The model then predicts 3D poses for T future frames.
main_review: The method proposed by the authors, using an encoder-decoder structure of transformers (similar to that of the original transformer for machine translation), is a good way of modelling temporal dependencies in 3D pose of a single person, and also interactions with other people in the scene. The transformer-based architecture can also naturally handle a variable sequence length at the input.

The authors use two encoders, one to model local interactions, and another to model global interactions between all people in the scene. The "Global range transformer encoder" has N times more input tokens than the local transformer, where N is the number of people. Therefore, since the "Global transformer" is a superset of the "Local transformer", is the local transformer strictly necessary? And would any potential benefits of having a "Local + Global" transformer combination disappear as the training dataset size increases? Results from other large-scale transformer papers [17] suggest that additional inductive biases in the network design becomes less important with larger dataset and model sizes. I would appreciate the author's comments on this.

An interesting choice made by the authors was to progressively increase the input sequence length to the model during inference. Ie, the input pose, and all previous predictions made in the sequence are used as the input to the encoder of the model. It would be good to ablate the effect of this design choice: What if the input sequence length is always fixed (except the beginning of the sequence)?

The authors also include some qualitative results showing that the decoder groups together groups of interacting people. However, the authors use a synthetically created dataset where they mix pose trajectories from different sources (ie CMU-Mocap and Panoptic). Therefore, it is easy to distinguish these groups based on the positional embedding alone, as the "mixed samples" are far apart in 3D space. These results would be more interesting to me, if the model was able to distinguish groups of people from the same original scene. Right now, it appears that the model is distinguishing people from different dataset splits instead.

The model produces good results in terms of MJPJE accuracy. However, future prediction by definition can have multiple possible outcomes, and the proposed method always predicts a single prediction, rather than diverse ones (ie [36]).

It would also be good to discuss this concurrent work in the final version (although I understand that it is too recent for the authors to compare to)

M Petrovich et al. Action-Conditioned 3D Human Motion Synthesis with Transformer VAE. arxiv 2104.05670. 2021

There are also a couple of typos in the text

Line 23: "ma" -> "may"
Line 24: "response accordingly" --> "respond accordingly"
Line 70 and 103: "in the world coordinate" -> "in world coordinates" / "in the world coordinate frame"
Line 137: "plus the sinusoidal embedding" --> "add the sinusoidal embedding"



Overall, I am quite borderline with this paper. It shows that a transformer-based encoder-decoder model is effective for the task of 3D future prediction. However, it does not appear that original to me, given the success of transformers in so many different domains. And the proposed model is fairly similar to the encoder-decoder model for machine translation in the original transformer paper. Although the authors perform explicit modelling of "local" and "global" interactions, I do not think they have really adapted the transformer architecture specifically to the future prediction task by for example, generating diverse outputs, or by conditioning the future prediction on some input.
limitations_and_societal_impact: Both of these are barely described in the main paper. The checklist says they are detailed in the supplementary, but I did not see it there.
review_title: null

Meta-review: The majority of the reviewers agreed that this is a solid paper that deserves acceptance. The negative assessment by one of the reviewers was considered overly harsh by the ACs. Authors are highly encouraged to address the key comments reported by reviewers as well as to implement all the improvements (as indicated by authors in the rebuttal) in the final camera-ready version. 
Decision: Accept (Poster)