Multi-Person 3D Motion Prediction with Multi-Range Transformers

Abstract: We propose a novel framework for multi-person 3D motion trajectory prediction. Our key observation is that a human's action and behaviors may highly depend on the other persons around. Thus, instead of predicting each human pose trajectory in isolation, we introduce a Multi-Range Transformers model which contains of a local-range encoder for individual motion and a global-range encoder for social interactions. The Transformer decoder then performs prediction for each person by taking a corresponding pose as a query which attends to both local and global-range encoder features. Our model not only outperforms state-of-the-art methods on long-term 3D motion prediction, but also generates diverse social interactions. More interestingly, our model can even predict 15-person motion simultaneously by automatically dividing the persons into different interaction groups.  Project page with code is available at https://jiashunwang.github.io/MRT/.


Reviews and author responses:

 We sincerely thank you for the comments. We will add the quantitative and qualitative results in our rebuttal to the revised paper. Besides that, we still have the following points to clarify:
 
For using MPJPE, there are previous studies that used this metric for sequences longer than 1 second, *e.g.* Cao et al. “Long-term Human Motion Prediction with Scene Context”[11]. We believe MPJPE is still meaningful as a metric to evaluate a trajectory despite the inevitable limitations. In addition, we have also provided results in less than one second and reported the root and pose errors separately in the rebuttal. 

For user study, we follow [64, 73] in using 3-5 users in one comparison. We will improve the evaluation system by using ranks instead of scores as metrics and by adding more users for each comparison. 

For visual results, we argue that our method predicts natural interaction motion while other baselines just repeat the history sequences (LTD, HRI) or produce frozen motion (SocialPool), *e.g.* 01:35-02:30 in the supplementary video. For the multi-person motion prediction task, our method has achieved large improvements compared to previous methods.


This work focuses on the 3D human motion prediction in scenes containing multiple agents. While prior work focused on single human 3D motion forecasting, this work focuses on jointly predicting the motion of all agents in a scene. To do this, this work proposes a novel Transformer-based architecture (named Multi-Range Transformer) which allows the future prediction of each agent to be based on the encoding of the entire past scene. The architecture consists of a local and global transformer encoder to encode the agent in question and all other agents' past, respectively. A Transformer decoder is employed to predict the future motion of the agent question. A discriminator  Transformer is used to assess the "realness" of a generated trajectory with the ground-truth future poses acting as positive examples. The approach is evaluated on multiple datasets quantitatively comparing to prior approaches and qualitatively using AMT.  Strengths:
- This work proposes a novel transformer-based architecture to perform 3D motion prediction of multi-agent scenes.
- The related works section addresses prior relevant work in this field adequately.
- Extensive evaluation is performed on multiple datasets using quantitative as well as qualitative approaches, showing superior results to baseline approaches.
- The attention diagrams (6, 3 in suppl.) provide the intuition behind why using attention for multi-agent scenes is beneficial.

Weaknesses:
- It is unclear if the strength of the method is really due to the use of the transformer for encoding and decoding or due to predicting $\Delta x$ at every timestep. It would have been a stronger comparison if, for example, SocialPool was predicting the same $Delta x$ output as the Multi-Range Transformer. This is something the authors note could be contributing to the better performance (in paragraph starting at line 262).
- The user study seems to indicate a marginal qualitative improvement over prior methods, and it is unclear if it is statistically significant given the standard deviation. Some details in regards to the number of turkers employed would be beneficial for the reviewers.
- A limitation of this proposed approach is that it does not address the potential for multiple futures scene rollouts. For example, if one of two interacting agent moves in one way, the other agent can react to it in multiple ways. I do not see a way for the model to encompass these possibilities.
- The motivation behind using the adversarial loss in addition to the reconstruction loss is lacking. What is the purpose of having a discriminator here? What would happen if we removed the reconstruction loss and only trained with the adversarial loss?

Clarifications:
- If I understand correctly, the output of the SPE() function is a scalar representing the distance between two agents. How exactly are these scalar values added to the output of the global-range transformer output? element-wise?
- Is moving distance (Fig 4) how far the model predicts an agent moves in the scene? Perhaps the name should be changed to "distance traversed".
- The placement of Tables 1-3 and Figure 3 makes it a little straining to look at. It would be good to reorganize the figures should this work be accepted.

There was some grammar issues throughout the manuscript that should be addressed in the final version should this work be accepted. Here are some examples:
line 23: "behaves ma highly depend".
line 167: " architecture outputs sequence directly prevents generating..."
line 185-186: "different lengths of sequence in this way input during training."
line 257: "even not as good".
line 263: "other methods intend to predict..." --> "other methods tend to predict..."
 This work addresses concerns over the societal impact of surveillance and privacy invasion. However, it does not address the limitations of the approach. The conclusion should be lengthened to discuss what could be done to improve the current approach. This can include extending the approach to multi-modal predictions where the model predicts multiple future scenes that are all realistic.

 I would like to thank the authors for providing a thoughtful rebuttal that addressed many of my concerns. As a result, I have decided to update my score by one point. Although I do like the method and think it has some novelty, as pointed out by reviewer 7KAw, I believe this work needs further revisions on the evaluation side to better assess its contributions and to provide support for its claims.

 I thank the authors for their extensive rebuttal and additional results. I decided to keep my initial score. The problem is interesting and the proposed method is solid. My main concern is with the evaluations. It requires a revision to clearly support the contribution. 
I respectfully disagree with the authors regarding the use of the MPJPE metric for sequences longer than 1 second.  It becomes inconclusive as the sequence length increases. The test set does not consist of only the deterministic motions such as hand-shaking. For example, in a walking sequence, a small amount of change in the pace may easily lead to very large errors while it is still perfectly natural. 
As a complementary evaluation, the authors present a user study. However, the results are not statistically significant as pointed out by the reviewer 3D5S. 3 users are not enough to report an average performance, resulting in a very large variance. 
I watched the provided samples again. Some of the results look okay, but very often the characters are just doing something. It looks smooth most of the time and the bone length variation is not always a problem, but is the produced motion really meaningful or better than the baselines?

 Thanks to the authors for addressing my comments, and I decide to keep my original rating.

 Thank you for your patient response. The authors addressed almost all points in the rebuttal.

 Dear reviewer 7KAw, thank you for your detailed and thorough review. We have made a general response to provide further discussion and insights into our method. We sincerely thank you for the advice. We will revise them in the future version. In the following, we seek to address each of your concerns.

---

**Q**: *“Based on the description of the method I do not think it is possible as the  stacks the features of all  people into one vector (Ln. 137, 140).”*

**A**: $E_{glob}$ is a matrix $\in R^{Nk \times d_K}$, where $N$ represents $N$ persons, $k$ represents $k$ time steps, $d_K$ is the dimension of the embedding. When we are training with different people in a batch, we will choose an $N_{max}$ to make sure every $N<=N_{max}$ and padding the rest part with 0. When $N<N_{max}$, we will use a mask to cover the padding part. 

---

**Q**: *“I find these numbers to be of limited value because a baseline might have a low MPJPE at 3 seconds, but that does not necessarily mean it performs badly. In my opinion, it would make more sense to report MPJPE for shorter time horizons up to 1 second, and adopt distribution-based metrics for longer time horizons, e.g. like Aksan et al. [5].”*

**A**: It is a very short period for interaction if we predict less than one second. In most cases, people can not distinguish what persons are doing visually in such a short time and that is not a problem we are aiming to solve. We argue that if we are modeling the interaction, e.g. shaking hands, the motions will be restricted to a certain extent, because they are jointly determined by the past behavior of multiple persons. It does not have as much freedom as the single-person-motion. In that case, we think MPJPE can be applied for a longer time. So currently we predict one result and compare it with the ground truth between 1 to 3 seconds. (We also provide 0.25-second and 0.5-second comparisons in the following tables.)
 
On the other hand, in addition to MPJPE, we conduct a user study and also provide many qualitative comparisons in the supp video to show that those baselines not only have higher MPJPE but also have poor visual quality. In some cases, baselines just predictit repeated motion.  Considering these evaluations together, we think those baselines with higher MPJPE are indeed performing badly.

The distribution metric in Aksan et al. [5] does not take the multi-person interaction into consideration: predictions for each person should be limited by multi-persons. This metric could not reflect whether people are interacting in a right way. Thus we think it is not suitable for our task.

---

**Q**: *“how much the global root misalignment affects the error and how much pose errors are contributing. two baselines [43, 44] do not consider global root translation[...].”*

**A**: Original versions of [43, 44] do not consider global root translation. But since they take 3D coordinates as inputs, we adapt it to our training data and retrain a model for our task.
We have provided the quantitative comparisons of the root error and pose error (MPJPE with aligning) separately in the following tables. It can be seen that for the root and pose prediction, our method surpasses other baselines in most cases.
 
**Root**:

CMU-Mocap |  |  |  |  |  |   MuPoTS-3D |  |  |  |  |  |   3DPW |  |  |  |  |  |  
--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | 
 | 0.25s | 0.5s | 1s | 2s | 3s |    |0.25s | 0.5s | 1s | 2s | 3s | |0.25s | 0.5s | 1s | 2s | 3s |
LTD | 0.48 | 0.67 | 0.97 | 1.73 | 2.62 |  |0.49 | 0.68 | 0.89 | 1.39 | 1.91||1.49|2.45|4.28|6.79|8.41
HRI | 0.26 | 0.48 | 0.96 | 2.06 | 3.11 |  |0.25 | 0.40 |0.66 | 1.3 | 2.16 | |0.88 | 1.97 | 3.67 | 6.42 | 8.64 
SocialPool | **0.22** | 0.47 | 0.96 | 2.01 |2.96 |  | 0.24 | 0.42 | 0.96 | 1.38 | 2.21 | |**0.83**| 1.90 | 3.76 | 6.86 | 9.07
Ours | **0.22** | **0.37** | **0.60** | **1.12** | **1.71** |  | **0.23** | **0.39** | **0.67** | **1.25** | **1.86**| |0.93 |  **1.87** | **3.42** | **5.69** | **7.30**

**Pose**:

CMU-Mocap |  |  |  |  |  |   MuPoTS-3D |  |  |  |  |  |   3DPW |  |  |  |  |  |  
--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | 
 | 0.25s | 0.5s | 1s | 2s | 3s |    |0.25s | 0.5s | 1s | 2s | 3s | |0.25s | 0.5s | 1s | 2s | 3s |
LTD | 0.62 | 0.85 | 0.98 | 1.21 | 1.37 |  |0.54 | 0.69 | 0.88 | 1.14 | 1.31||1.15|1.34|1.54|1.76|1.98
HRI | 0.49 | 0.76 | 1.05 | 1.37 | 1.58 |  |0.32 | 0.50 |0.73 | 1.07 | 1.30 | | 0.75 | 1.13 | **1.43** | **1.75** | 1.94 
SocialPool | 0.40 | 0.71 | 1.03 | 1.41 | 1.71 |  | 0.31 | 0.48 | 0.72 | 1.08 | 1.30 | |**0.74**| 1.19 | 1.60 | 1.95 | 2.15
Ours | **0.37** | **0.56** | **0.79** | **1.05** | **1.22** |  | **0.30** | **0.46** | **0.69** | **0.99** | **1.19**| | 0.79 |  **1.09** | 1.52 | **1.75** | **1.93**

---

**Q**: *“While it is appreciated that a user study on the quality of the generated motion is provided[...], all the reported averages are well within one standard deviation of the ground-truth average. How should we interpret the values? It is also not mentioned how big the sample size is.”*

**A**: We generate 50 samples on each dataset with different methods given the same input and ask 3 users to score each method in one comparison. Ground truth results for some datasets are obtained by pose estimation techniques so they are not perfect as well. Since we let users score 1, 2, 3, 4 or 5, we find the users intend to give scores mostly around 3 or 4. This may lead to relatively small differences between the methods but in this case the average score is still statistically meaningful to compare. More visual results and comparisons can be found in our supplementary video and website: https://anonymousaut.github.io/Anonymous-Result/. These can also show our improvement over other methods. We will redesign our interface for ranking the approaches in our revision of the paper.

---

**Q**:  *“I do not find the visual quality of the results to be very compelling. There seem to be a lot of foot skating artifacts[...].”*

**A**: Our results are not perfect but they have large improvements over previous methods as the video we showed in supp. At the same time, using position-based representation is more challenging to produce high quality motion and could lead to bone stretching [51]. We have tried DCT[44] to improve the smoothness of the results. But when the motion type is very unique, bone stretching may happen. 

---

**Q**: *“The paper mixes several motion capture datasets together (CMU, 3DPW, MuPoTS-3D, Panoptic). These datasets all have different skeletal representations[...]. Another possibility would be to use a large-scale dataset that is already unified, i.e. AMASS. This is already used by related work [5].”*

**A**: The large scale dataset AMASS is not for multi-person interaction motion, so we do not use it. The datasets we used are all including some multi-person interaction. It is not easy to register all other datasets to one reference skeleton representation because some multi-persons datasets only provide joint positions with noise, e.g. MuPoTS-3D and Panoptic. At the same time, the usage scenario of our method is dealing with this kind of noisy joint positions, since a lot of multi-person 3D data are obtained by pose estimation. Strictly unified representation is not very necessary, since those noisy data may already have different lengths of “neck”. Although our unified method is not perfect, it can be applied to all kinds of 3d human skeleton datasets in a very quick way, which is what we want to pursue.

---

**Q**: *“It is not clear how different sequences are merged to create multi-person sequences[...]. As it stands now, a simple nearest neighbor query might do a good job already.”*

**A**: The datasets we used all include multi-person interaction. Our goal for merging these datasets is to explore whether our method could deal with scenes with a large number of people. So we randomly put these different interacting persons into a plane ground. And there is a possibility that people who are not interacting are close to each other, which is a quite challenging situation. However, observing the attention weights in our architecture, we find our method could automatically distinguish different interaction groups, even though they are quite close. For example, in Fig.3 in the supp, person 1 is quite close to person 10 and person 11, but the distributions show that person 10 and person 11 are in one group while person 1 is not. In this example, a simple nearest neighbor query can not do a good job.


 Dear reviewer yyef, thank you for your detailed and thorough review. We have made a general response to provide further discussion and insights into our method. We sincerely thank you for the advice and pointing out the typos. We will revise them in the future version. In the following, we seek to address each of your concerns.

**Q**: *“What about the long-term (longer than 3 seconds into the future) motion prediction results?”*

**A**: We think that predicting 1-3 seconds is a proper time length to see the interaction. When we are modeling the interaction,e.g. shaking hands, the motions will be restricted to a certain extent, because they are jointly determined by the past behavior of multiple persons. So in a period of time, this result will be predictable. However, when predicting a very long period (e.g. 10 seconds) for multi-person, it would become a far more difficult problem than single-person prediction. It is crucial to ensure the rationality of multi-person interaction during the whole time. This is a quite challenging problem and we will try to explore it in our future work.  

---

**Q**: *“Previously, from my experience, position encoding is not critical for the motion prediction task. Is the designed Temporal position encoding & Spatial position encoding critical? Could you perform an ablation study to see the effect.”*

**A**: We have performed ablation study of the spatial positional encoding in Table 1 2 and 3. The results show our spatial positional encoding could improve the performance, especially when there is a large number of people in the scene. For the temporal positional encoding, we argue it is quite important in our task, otherwise the model doesn’t even know whether a person is moving forward or backward. 


 Dear reviewer 3D5S, thank you for your detailed and thorough review. We have made a general response to provide further discussion and insights into our method. We sincerely thank you for the advice and pointing out the typos. We will revise them in the future version. In the following, we seek to address each of your concerns.

**Q**: *“It would have been a stronger comparison if, for example, SocialPool was predicting the same output as the Multi-Range Transformer.”*

**A**: Yes, currently, the SocialPool is using the offset $\Delta x$ representation as it was proposed in [2].

---

**Q**: *“The user study seems to indicate a marginal qualitative improvement over prior methods, and it is unclear if it is statistically significant given the standard deviation. Some details in regards to the number of turkers employed would be beneficial for the reviewers.”*

**A**: We generate 50 samples on each dataset with different methods given the same input and ask 3 users to score each method in one comparison. Ground truth results for some datasets are obtained by pose estimation techniques so they are not perfect as well. Since we let users score 1, 2, 3, 4 or 5, we find the users intend to give scores mostly around 3 or 4. This may lead to relatively small differences between the methods but in this case the average score is still statistically meaningful to compare. More visual results and comparisons can be found in our supplementary video and website: https://anonymousaut.github.io/Anonymous-Result/. These can also show our improvement over other methods. We will redesign our interface for ranking the approaches in our revision of the paper.

 ---

**Q**: *“The motivation behind using the adversarial loss in addition to the reconstruction loss is lacking. What is the purpose of having a discriminator here? What would happen if we removed the reconstruction loss and only trained with the adversarial loss?”*

**A**: Please refer to our answer on “Usage of discriminator” in the general response.  
 
---

**Q**: *“If I understand correctly, the output of the SPE() function is a scalar representing the distance between two agents. How exactly are these scalar values added to the output of the global-range transformer output? element-wise?”*

**A**: Yes, it is a scalar representing the distance between the query person and every person at different time-step.  Each scalar will be copied $d_K$ times (dimension of the embedding) and added to the feature in an element-wise.

---

**Q**: *“Is moving distance (Fig 4) how far the model predicts an agent moves in the scene? Perhaps the name should be changed to "distance traversed".”*

**A**: Yes, thank you for the advice. We will revise it in the future version.

---

**Q**: *“The placement of Tables 1-3 and Figure 3 makes it a little straining to look at. It would be good to reorganize the figures should this work be accepted.”*

**A**: Thank you for the advice. We will regorganize them in the future version.

---

**Q**: *“A limitation of this proposed approach is that it does not address the potential for multiple futures scene rollouts.” and “The conclusion should be lengthened to discuss what could be done to improve the current approach. This can include extending the approach to multi-modal predictions where the model predicts multiple future scenes that are all realistic.”*

**A**: We admit this is a potential limitation of our work. But we argue that if we are modeling the interaction, e.g. shaking hands, the motions will be restricted to a certain extent, because they are jointly determined by the past behavior of multiple persons. It does not have as much freedom as the single-person-motion. So currently we predict one result and compare it with the ground truth between 1 to 3 seconds. 

On the other hand, diverse motion prediction [69, 72] is a quite attractive topic for single-person motion prediction nowadays, but considering that for multi-person, things will become more difficult. The multi-possibility needs to be appropriate for all the interacting persons at the same time. And for single-person diverse prediction, studies are pursuing more differentiated results. But for multi-person, we think it not a suitable pursuit because motions will be restricted to a certain extent, as they are jointly determined other persons. At the same time, besides the user study, evaluating the rationality of the diverse interaction will become harder and could be explored. Overall, this is an interesting and also more challenging problem and we look forward to extending our work to it in the future.


 Dear reviewer jyew, thank you for your detailed and thorough review. We have made a general response to provide further discussion and insights into our method. In the following, we seek to address each of your concerns.

**Q**: *“In particular, the paper only compares with two previous state-of-the-art methods on the same task, and one method on modelling social interaction (which is not originally designed in the same domain). While I appreciate the authors provide the user study to quantify the qualitative results, I would suggest adding more methods (both for the task, and for interaction modelling) in comparison to make the experimental results more rigorous and thorough.”*

**A**: Currently, predicting multi-person 3d motion is a relatively under-explored problem, especially when modeling the pose and trajectory together. We try our best to find methods suitable for our task to compare as **reviewer 3D5S** claims *“extensive evaluation is performed on multiple datasets using quantitative as well as qualitative approaches”*. We select two methods that are based on single-person prediction and one for multi-person prediction and compare them in many existing datasets. Some previous works are not suitable for this task, for example, a line of angle-based methods for single-person motion prediction can not deal with datasets such as MuPoTS-3D and Panoptic. In addition to the comparison of the existing SOTA methods, we have also conducted extensive ablation studies to prove the effectiveness of each module in our method. At the same time, we will keep trying to look for more concurrent works to discuss and compare.

---

**Q**: *“Needs more discussions on the limitations of the work.”*

**A**: Multiple possibility prediction is a problem we haven’t considered at this stage. Diverse motion prediction [69, 72] is a quite attractive topic for single-person motion prediction nowadays. But considering that for multi-person, things will become more difficult, because the multi-possibility needs to be appropriate for all the interacting persons at the same time. This is an interesting and more challenging problem and could be our future work.


 Dear reviewers, we appreciate all your feedback and in the following, we provide our response to general comments and further explain the insights of our method. We welcome further discussion with each of the reviewers to address any remaining concerns.

**Usage of discriminator**

Our goal is not only learning to predict accurate interaction but also predicting smooth movements. We provide quantitative comparisons in Table 1,2 and 3. We also provide a [video comparison](https://raw.githubusercontent.com/AnonymousAut/Anonymous-Result/gh-pages/rebuttal/disc.gif) to show the effectiveness of the discriminator. Using a discriminator could improve the naturalness of the motion. On the other hand, some training data are not very smooth as well since they are obtained by pose estimation and a discriminator could help to learn to predict smoother results.

**Multiple possibility prediction**

Unlike single person motion prediction, multiple-person motion is restricted by the reactions to the other persons. Thus the uncertainty for prediction is relatively smaller here. Besides, we are predicting 1 to 3 seconds of future by taking 1 second inputs, thus we can apply a deterministic model to perform the task. Besides comparing with ground truth, we have conducted a user study to compare the naturalness of the prediction results in Tab. 4. The qualitative results are also provided in the supplementary video and website. 

We do agree that diverse motion prediction [69, 72] is an interesting direction. We see our work as the first step to deal with long-term, multiple person motion prediction using Transformers. We will look into incorporating this aspect as a future extension of work. 


 **Q**: *“The differences are sometimes marginal. E.g. in Table 2 on MuPoTS-3D for 1 second, the SocialPool baseline [2] and all ablation experiments score exactly the same[...], and so does feeding half the sequence or the entire sequence.”*

**A**: We argue that, when predicting a relatively short time length, results would be closer and this is related to the motion type and the distribution of the dataset. But as the forecasting length becomes longer, our method will surpass other methods in most cases, which is what we expect. We perform experiments on several datasets to make sure the comparison is fair and objective. On the other hand, when evaluating our method and other baselines, qualitative and quantitative results could be considered comprehensively.  Our method has large improvements over previous methods as the video we showed in supp. We also show qualitative comparisons with [w/o discriminator](https://raw.githubusercontent.com/AnonymousAut/Anonymous-Result/gh-pages/rebuttal/disc.gif), [w/o DCT](https://raw.githubusercontent.com/AnonymousAut/Anonymous-Result/gh-pages/rebuttal/dct.gif) and [w/o Local](https://raw.githubusercontent.com/AnonymousAut/Anonymous-Result/gh-pages/rebuttal/local.gif).

---

**Q**: *“It is not clear from the paper that leaving out the bottleneck leads to the repetition of existing motion.”*

**A**: As the supp video shows, [43] [44] are both repeating the existing motion. These methods use longer input as the seed motion. This inspired us when we designed the network architecture. At the same time, we show that taking longer input as query has worse performance in Tab. 5. We also give a visual comparison between our method and using longer input as query [here](https://raw.githubusercontent.com/AnonymousAut/Anonymous-Result/gh-pages/rebuttal/longer.gif)

---

**Q**: *“It is not clear from the paper that adding adversarial training helps in making natural and continuous predictions”*

**A**: We provie a qualitative comparison with w/o discriminator [here](https://raw.githubusercontent.com/AnonymousAut/Anonymous-Result/gh-pages/rebuttal/disc.gif). For more discussions about the discriminator, please refer to our answer on “Usage of discriminator” in the general response. 

---

**Q**: *“Lastly, the paper does not really adequately address its limitations.[...]. the error accumulation problem could be amplified compared to the single-person prediction case.”*

**A**: A person's prediction result could be improved with other people's motion, this is proved by comparing our method with the single-person based methods [43,44]. Error accumulation usually increases when predicting longer. However, our method improves more when predicting longer compared to single-person based methods. We think multi-person interaction could create a restriction on the prediction and then error accumulation is reduced compared to single-person based methods since they can not model the interaction.  

On the other hand, since the transformer has the ability to handle variable length input, in our method, we progressively increase the input sequence length to the encoders during training and testing as described in Ln. 184-188. Using this strategy, our model can always see everyone’s motion in the very beginning and we find this could reduce the error accumulation during the recursively prediction. To prove it, we conduct an experiment using a fixed length (one second motion) as input to the encoders and we report the quantitative comparisons using MPJPE on CMU-Mocap in the following table. It can be seen our strategy is helpful, especially for long-term prediction. 

| | 1 second | 2 seconds | 3 seconds
--- | --- | --- | ---
fixed length | **0.96** | 1.91 | 2.91
ours | **0.96** | **1.57** | **2.18** 


This paper tackles the problem of multi-person 3D motion trajectory prediction, and proposes a method that focuses on modelling the social interactions between people. In particular, the proposed method consists of a motion generator that generates future motion trajectory given past trajectory, and a motion discriminator that classifies natural and unreal trajectories. Both modules are trained jointly in an adversarial way. To model social interactions, the motion generator consists of a Transformer based encoder, which has both a local branch that encodes single-person motion, and a global branch that encodes multi-person interactions; and a Transformer based decoder that predicts the future trajectory. Expeirments on multiple datasets show improvements over previous methods both quantitatively and qualitatively (via human study).  Overall the paper is easy to read, technically sound, and shows good results.
My main concern with the paper is about the experimental evaluation. In particular, the paper only compares with two previous state-of-the-art methods on the same task, and one method on modelling social interaction (which is not originally designed in the same domain). This makes the experimental results not convincing enough. While I appreciate the authors provide the user study to quantify the qualitative results, I would suggest adding more methods (both for the task, and for interaction modelling) in comparison to make the experimental results more rigorous and thorough. Needs more discussions on the limitations of the work.

This paper proposes a method to predict the motion of multiple people into the future for 3 seconds. For each person, the predictions are made based on the entire history of that person and the history of all other people present in the scene as well.
To do so, a Transformer-based architecture is used. The last known frame of each person serves as a query and the keys and values are computed from features that were extracted from both the person-specific history and the histories of the remaining people. Those features themselves were computed via a Transformer-based encoder. The method operates on 3D joint positions which are pre-processed with the Discrete Cosine Transform (DCT).
The method is evaluated on multi-person sequences that are native to some datasets (CMU, 3DPW, Panoptic). Because the availability of such sequences is limited, other sequences are mixed to create synthetic scenes with up to 15 people. The mean per-joint positional error (MPJPE) is used as a metric for all prediction horizons and a user study is provided.  **Originality**
The task of predicting human motion depending on how other people move in the scene is not new, but relatively little work exists compared to single-person forecasting. The proposed method uses well-known building blocks (DCT, Transformer, adversarial training) which are combined reasonably for the task at hand. The paper also tackles the problem of estimating global root translation, which is typically neglected in other motion prediction papers. Related work could be discussed more in-depth. Specifically, I find long citation lists like on Ln. 75 or 86 not very useful.

**Quality**
The description of the network architecture is mostly fine. My only question is if the architecture can support a varying number of people at training and test time. Based on the description of the method I do not think it is possible as the $E_{glob}$ stacks the features of all $N$ people into one vector (Ln. 137, 140). However, the evaluation of MuPoTS-3D in Tab. 2 (2 ~ 3 persons) seems to suggest otherwise. Does passing N or (N-1) subjects change only the input “sequence” length? Could the authors clarify this?

In the following I have a number of concerns regarding the evaluation and the data:
- As Fragkiadaki et al. [19] have noted, comparing motions for long time horizons (i.e. > 1 second) with the MPJPE is problematic due to the stochastic nature of human motion. This paper evaluates the MPJPE at prediction horizons of 1, 2, and 3 seconds. I find these numbers to be of limited value because a baseline might have a low MPJPE at 3 seconds, but that does not necessarily mean it performs badly. In my opinion, it would make more sense to report MPJPE for shorter time horizons up to 1 second, and adopt distribution-based metrics for longer time horizons, e.g. like Aksan et al. [5]. The authors do include a distribution-based metric in Fig. 4, but this only shows how far a joint traveled between the beginning and end of the sequence, which carries little information about the validity of the actual poses.
- The MPJPE is computed on poses that contain global root translation (Ln. 226-227). This makes it impossible to know how much the global root misalignment affects the error and how much pose errors are contributing. Intuitively, predicting the global root position is difficult, so I assume its contribution to the error metric is substantial. In any case, the root misalignment and pose error should be reported separately. This is furthermore important because the two baselines [43, 44] do not consider global root translation, and hence comparing them on a metric that includes global root translation could be deemed unfair.
- While it is appreciated that a user study on the quality of the generated motion is provided (Tab. 4), I am not sure about the significance of the results. All the reported averages are well within one standard deviation of the ground-truth average. How should we interpret the values? It is also not mentioned how big the sample size is. 
- I do not find the visual quality of the results to be very compelling. There seem to be a lot of foot skating artifacts. In the example where two people are dancing (example 3 starting at 03:28), there seem to be drastic variations of the bone length. There is no discussion about these shortcomings in the paper.
- The paper mixes several motion capture datasets together (CMU, 3DPW, MuPoTS-3D, Panoptic). These datasets all have different skeletal representations. In the supplementary, it is explained how those representations are unified. As far as I can tell, the authors just pick a subset of joints that is common to all 4 datasets. In my opinion, this is insufficient. Although all 4 datasets are likely to have - for example - a “neck” joint, it is not guaranteed that they define the neck joint exactly the same way. E.g., in one dataset it might be closer to the head than in the other. To align the data properly, one would have to define a reference skeleton (e.g. the CMU skeleton) and then register all other datasets to that reference skeleton. Another possibility would be to use a large-scale dataset that is already unified, i.e. AMASS. This is already used by related work [5].
- It is not clear how different sequences are merged to create multi-person sequences. While I appreciate the effort to create synthetic multi-person sequences it should be discussed how they are merged exactly, and what the advantages/downsides of the specific merging strategy are. The paper mentions that attention weights can be used to define interaction groups. This is potentially an interesting mechanism, but it highly depends on how the sequences were stitched together. As it stands now, a simple nearest neighbor query might do a good job already.
- Ablation studies to show the impact of the global range encoder, the discriminator, the spatial positional encoding, and the length of the decoder query are provided. The differences are sometimes marginal. E.g. in Table 2 on MuPoTS-3D for 1 second, the SocialPool baseline [2] and all ablation experiments score exactly the same. In Table 5 (ablation on decoder query length) feeding 1, 2, or 4 frames performs exactly the same, and so does feeding half the sequence or the entire sequence. Furthermore, the following two claims are not well supported:
  - Ln. 45-48 “we create a bottleneck to force the Transformer to exploit the relations between different time steps and persons in the encoders, instead of just repeating the existing motion alone”. It is not clear from the paper that leaving out the bottleneck leads to the repetition of existing motion.
  - Ln. 170-172 “To ensure a natural and continuous long-term motion, we use a discriminator D to adversarially train the Predictor P”. It is not clear from the paper that adding adversarial training helps in making natural and continuous predictions (the MPJPE does not allow us to make a statement about whether motions are “natural and continuous”).
- Lastly, the paper does not really adequately address its limitations. A weakness that should be discussed is the error accumulation problem. Because each prediction is now based on as many (potentially erroneous) predictions as we have people in the scene, the error accumulation problem could be amplified compared to the single-person prediction case.

**Clarity**
The overall organization of the paper is adequate. It does however contain typos and formulations that hinder the readability on many occasions. I think a general proofreading pass could address those.

**Significance**
Because of the above-mentioned flaws, I do not think this paper advances the state-of-the-art in a demonstrable way. Although considering societal contexts in motion prediction can be interesting to other researchers, I am not sure the setting this paper constructed is ideal to do so.

**Summary**
I generally find motion prediction conditioned on other people’s movements an interesting topic. I also think that the Transformer framework is a good fit for this setting, which I’m considering as the strength of the paper. However, the various weaknesses in evaluation and data preprocessing - outlined in more detail under “Quality” in the “Main Review” section - clearly outweigh the strengths of this paper. This paper thus seems premature and requires a major revision.
 The broader impact statement is very short and vague. It could be discussed in more detail what “the invasion of people’s privacy through the human motion and behavior” exactly means.


They propose a novel framework for multi-person 3D motion trajectory prediction. The multi-Range Transformer model contains a local-range encoder for individual motion and a global-range encoder for social interactions.   Originality: Based on Transformer, this paper proposes Multi-Range Transformer model multi-person 3D motion trajectory prediction. This is a novel task. The proposed method also has some specially designed architecture for this task.  

Quality: Multi-Range Transformer achieves a significant improvement over state-of-the-art approaches for long-term predictions, and the gain enlarges as they increase the future prediction time steps from 1 second to 3 seconds.  

Clarity: The method is clearly explained.  

Significance: Most of the previous work focus on single human 3D motion prediction. Multi-person motion prediction is a novel task. This paper also observes how a human acts and behaves highly depend on the people around. 

 

Typos: 

Line 23: “acts and behaves ma highly” -> “acts and behaves may highly”  1. What about the long-term (longer than 3 seconds into the future) motion prediction results? 
2. Previously, from my experience, position encoding is not critical for the motion prediction task. Is the designed Temporal position encoding & Spatial position encoding critical? Could you perform an ablation study to see the effect of  
1) With Temporal position encoding & Spatial position encoding 
2) Only Temporal position encoding 
3) Only Spatial position encoding 

Meta-review: The final scores for this paper are just below the usual threshold for acceptance. 
While this paper explores an interesting approach to modelling multiple 3D agent motions, the experimental work was assessed by half of the reviewers as needing more work in a number of ways to fully evaluate the method. The number of users used for the human evaluation was seen as being too small and the specifics of the metrics used to evaluate the method were put into question with respect to evaluating motions of longer duration. With another revision and re-submission I am confident the authors will have a successful submission, but at this time the AC recommends rejection.


Decision: accept