Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a transformer-based VAE model for motion prediction that can output multi-model and scene-consistent predictions. Specifically, the transformer is employed for modeling both social and temporal information (but separately, first temporal and then social and repeat). The proposed method achieves state-of-the-art performance on the nuScenes dataset and top performance on the Argoverse dataset
main_review: Paper Strengths

1. The proposed method using the transformer to model both social and temporal information in both the encoder and decoder makes sense. Also, the ability to generate scene-consistent predictions are verified with both qualitative and quantitative results on TrajNet

2. The computation requirement of the proposed method is very appealing compared to other transformer-based motion prediction methods, as explained in section 4.2. With experience in running other similar methods, training usually takes 2-3 days on top-performing GPUs, whereas the proposed method only requires training on 1080ti for 10 hours. 

3. The experiments are extensive, where the proposed method has been evaluated on three motion prediction datasets and one stroke completion dataset

Paper Weaknesses

1. Why is the proposed method very fast compared to others? Although a summary of computing requirements is provided, it would be interesting to perform a deeper analysis as to why the proposed transformer-based method is faster than other transformer-based methods, smaller feature dimensions? The fewer number of blocks? This could be helpful to the community in the future to develop efficient and high-performance transformer-based prediction methods

2. The ablation experiment can be expanded. Currently, the only ablation experiment is done in Table 3 on TrajNet with two variations of removing social modeling. However, the TrajNet challenge is a bit outdated and data annotation is imprecise. Would be more convincing to perform the ablation on the nuScenes or Argoverse dataset. Also, there are other components of the proposed method that are not ablated. For example, how is the proposed transformer-based approach compared to RNN-based or Graph-based methods? How is the map affecting the performance? How are the seed parameters useful? How many time/social blocks are needed?

3. The novelty is a bit weak considering other works have similar transformer-based motion prediction methods, especially [A1] as mentioned in section 5. Even ignoring this “concurrent” work, other works including [A2, A3] also use stacked transformers to do both social and temporal modeling. It is incorrect to me that the paper claims [A2, A3] did not encode the time and social dimensions using transformers. It would be great if the differences with these works can be explained in a more detailed and precise way

4. The experiments on TrajNet and Omniglot are not very convincing, where no quantitative comparison is properly made with prior work. For example, on the TrajNet challenge, there are plenty of prior motion prediction methods that have been evaluated on this dataset, so it should be easy to have a table for comparison. For stroke completion, I am not very familiar with this dataset, but only comparing with a simple LSTM baseline is not convincing. Including experiments on more datasets is great, but would be good to show convincing experiments. 

References

[A1] AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting. ICCV 2021

[A2] Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction. ECCV 2020

[A3] Multimodal Motion Prediction with Stacked Transformers. CVPR 2021


Post-rebuttal review

After carefully reading other reviews and the authors’ comments, I would like to increase my rating to a score of 8 -- accept good paper (or a score of 7 if there is such an option). The change to my score is mainly because I am satisfied with most of the authors’ comments to my concerns, which include: 1) the clarification on why the proposed method is much faster; 2) the expanded ablation experiments; 4) the clarification for experiments on Omnlglot. I strongly appreciate the authors' efforts on clarifying these questions and concerns!

As a result, the main comment left on my side is 3) the limited novelty compared to prior work. This does not mean that I find the authors’ reply unconvincing. In fact, it is quite clear and honest. But essentially, the main difference to prior work seems to be just the use of seed parameters that increases performance and also runtime speed, since other points are already proposed in prior work, including joint social-temporal modeling in encoders/decoders using transformers, and scene-consistent predictions. But I would like to emphasize that this relatively incremental novelty is fine to me, and personally, I would like to have a try on the seeing parameters proposed in this work. So it would be wonderful for the community if the code can be carefully documented and released (currently I did not find it in the supp or any anonymous URL). Finally, as a minor point, it would be great if the map or context image can be overlaid on the trajectories for the provided video results. 

In short, if there is no other significant downside/concern pointed by other reviewers, I would like to stick to my rating above and recommend accepting this paper


summary_of_the_review: Novelty is slightly weak but empirical performance is strong and the method might be practically useful
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper tackles the multi-agent trajectory prediction problem, primarily for autonomous driving, but experiments also include TrajNet and predicting Omniglot strokes. The authors claim their contributions in a very general sense:

- Novel method on modeling sequences of structured continuous variables and capture multi-modal distributions.
- Strong results on nuscenes, argoverse, trajnet and omniglot stroke prediction.

The key modeling novelty is the latent variable sequential set transformer which applies self-attention across different agents and across time in the scene. 

I mostly agree with the authors' assessment in general. I have some concerns on the claimed novelty (see below).
main_review: (+): Extensive experiments and results: The authors conducted experiments on 4 datasets, covering 3 different tasks (argo and nuscenes are both prediction for AV). These are very extensive. I do recognize that the method generalizes well to multiple different tasks.

(-) Novelty: I think the novelty should be good if it wasn't for the SceneTransformer paper. According to ICLR review guidelines, Arxiv papers published 30 days prior to the ICLR deadline are considered prior work. The SceneTransformer's formulation is very similar to that proposed in this paper, e.g. using transformers to model cross-agent and cross-time information propagation. Although SceneTransformer does not have an object query that is the latent variable Z, it was still able to learn a diverse set of future predictions. It's unclear if the explicit formulation of latent variable makes a difference. It seems that on Argoverse, this paper performs very similarly than SceneTransformer, which further confirms my view that the methods are very similar.
summary_of_the_review: I am giving a borderline reject for now due to the novelty concerns in light of a very similar paper in my opinion. I know that there are a lot of devils in the detail between the two works, and it would have been an obvious accept if this works outperforms SceneTransformer significantly.

In the rebuttal, may I ask the authors to further highlight the difference between their approach to SceneTransformer? And where do they see that their method is intuitively better than SceneTransformer?

POST REBUTTAL: The authors pointed out that SceneTransformer should be considered concurrent work. If that is the case, I have no issues to accept this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This submission presents a Transformer-based architecture for trajectory prediction tasks involving multiple agents. Such a multi-agent setting requires learning of spatio-temporal representations capturing both the long-term temporal dependencies as well as the social interactions between the agents. The paper formulates the task as modeling of the sequence of sets where every entry in the set corresponds to an agent’s observation. The proposed architecture augments the set transformer with a discrete latent variable to be able to make multiple predictions into the future. The given seed sequence is first encoded into a context representation which is later used to make future predictions with a decoder where the agents can be modeled jointly or independent from each other. 
main_review: Strengths: The paper proposes a solid architecture by considering the requirements of the problem. The proposed architecture relies on Transformers and the self-attention function which is an effective tool in modeling relations between different entities. Trajectory forecasting, especially in a multi-agent setting, is highly stochastic and may result in many feasible predictions. The paper incorporates a set of learnable seed parameters into the Transformer architecture to enable multi-moded prediction where the seeds are expected to lead to different modes. The model is evaluated in established benchmarks and achieves competitive performance with a lower computational requirement.

Weaknesses and Questions: The main contributions are the set assumption and the discrete latent variable. Unfortunately, the paper does not provide empirical evidence for these design choices. Ablations evaluating the proposed approaches could be insightful. 

Could the authors clarify why the proposed set assumption is feasible for multi-agent tasks? I can see the advantage of this design choice if the set of agents is dynamic (i.e., gets updated in time). However, this scenario is not covered in the experimental setup. It would be useful to see if the set assumption contributes to the performance. 

It wasn’t clear to me how the discrete latent variable is modeled. It is partly due to the cluttered notation (see the next comment). I kindly ask the authors to provide more details. An ablation evaluating the proposed model without using the discrete latent variable could be useful. I also would like to point out alternatives in case the authors are not aware of this line of work. Latent variables in sequence models have been studied [1]. While it is reported to be effective, there are more straightforward approaches to get multi-moded predictions [2]. Have the authors experimented with continuous latent variables or mixture density outputs for multi-modal predictions?

I find the notation not rigorous. Several symbols are used in multiple definitions, making the paper hard to follow. For example, 

1- $Z$: 
The discrete latent variable appears in the objective function in section 3.3 with the symbol $Z$. It is also defined to be the seed as “learnable seed parameters $Z \in \mathcal{R}^{(d_K; M; T)}$” in section 3.2. Are they the same? 

2- $K$:
Section 3.1: “... having M elements (or agents) with K attributes (e.g. x/y position).”  
Section 4.1: “... AutoBot-Ego trained with K = 10.” or “AutoBot-Ego obtains the best overall Min ADE out of 5 predictions (K = 5).” 

I have the following questions:

3- Could the authors clarify this statement in section 4.1:  “In order to generate ten predictions from five modes, at test time, we create combinations between the latent modes one-hot vector by summing one-hot vectors.”
4- What is the difference between the “AutoBot-Ego (K=10)” and “AutoBot-Ego (ensemble)” entries in Tab. 1?

Finally, [3] seems to be relevant and should be included in the related work discussion if the authors also agree. 

[1] Chung, Junyoung, et al. "A recurrent latent variable model for sequential data." Advances in neural information processing systems 28 (2015): 2980-2988.

[2] Graves, Alex. "Generating sequences with recurrent neural networks." arXiv preprint arXiv:1308.0850 (2013).

[3] Li, Jiachen, et al. "Evolvegraph: Multi-agent trajectory prediction with dynamic relational reasoning." arXiv preprint arXiv:2003.13924 (2020).

-- Post-rebuttal edit --
I thank the authors for the clarifications. I read other reviews and the responses. I am glad to see that the authors addressed the questions and provided additional experimental evidence. The ablations improve the quality of the submission. In fact, the latent variable ablation can be included in the main submission. Although the technical novelty is slightly limited, the proposed approach is solid. I increase my score to 6. 
summary_of_the_review: The proposed architecture is tailored for the task of multi-agent trajectory prediction task and presented to be effective in the benchmarks. However, we do not know if it is due to the superior performance of the underlying computation units (i.e., transformer blocks) or the proposed extensions (i.e., discrete latent variable/learnable seed, set assumption). I also think that the presentation could be improved.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper adapts transformer to multi-agent motion forecasting. The attention layers are applied on time and agent axis to capture motion and social information. A latent variable is introduced on the output to capture discrete motion for each agent.
Extensive experiments are conducted on various dataset with good performance. The training time of the proposed model is significantly faster than previous methods.
main_review: Pros:
+ This proposed model is simple and effective. It applies multi-head attention modules on temporal and social axis both in encoder and decoder to capture motion and interaction among agents.
+ The proposed model has been tested with comprehensive experiments, including on toy examples, real-world dataset such as nuScenes, Argoverse and TrajNet, as well as an image generation dataset (Omniglot). It shows comparable results compared to previous state-of-the-art models.
+ The latent variable is used to represent the discrete future for each agents. In spirit, it is same as Multiple Future Prediciton (Tang et al 2019), however, this paper adapts it into the transformer model and make it work as well.
+ The training of this model is very fast, as pointed out on page 6, it consumes 0.4% computation of SceneTransformer and 5% of Jean's method on Argoverse. This shows the efficiency of the proposed model.
+ Code is provided and it's nice and clean.

Cons:
- It's related to Axial Attention in Multidimensional Transformers, Jonathan Ho et al, 2019, and should cite that work.
- In the provided code, the decoder is autoregressive instead of what is proposed in the main paper. It should be updated accordingly, and more ablation/discussion on these two type of decoder would be very helpful.
- Minor point: for equation (5) on L(\theta) = Q + const, the const should be KL divergence and is not a const w.r.t. \theta.

summary_of_the_review: This paper compiles an interesting model for multi-agent motion forecasting. Although the transformer module is similar to Axial Transformer [Ho 2019] and the latent variable formulation and optimization are similar to MFP [Tang 2019], it is the first to successfully apply those two module together for motion forecasting task, with clean and concise code released. Thus I would vote for weak accept.
review_title: null

Meta-review: This paper studies the problem of motion prediction for multiple agents in a scene using transformer-based VAE like architecture. The paper received mixed reviews initially which generally tended towards borderline acceptance. All reviews appreciated extensive experiments but had some clarifications and requests for ablations. The authors provided a strong rebuttal that addressed many of the reviewers' concerns. The paper was discussed and all the reviewers updated their reviews in the post-rebuttal phase. Reviewers unanimously agree that the paper should be accepted. AC agrees with the reviewers and suggests strong acceptance. The authors are urged to incorporate reviewers' comments in the camera-ready.
Decision: Accept (Spotlight)