THOMAS: Trajectory Heatmap Output with learned Multi-Agent Sampling
Authors: Thomas Gilles
Stefano Sabatini
Dzmitry Tsishkou
Bogdan Stanciulescu
Fabien Moutarde
Venue: ICLR 2022 Poster (2022)
Keywords: Trajectory prediction
Multi-agent
Motion forecasting
Motion estimation
Autonomous driving

Abstract: In this paper, we propose THOMAS, a joint multi-agent trajectory prediction framework allowing for an efficient and consistent prediction of multi-agent multi-modal trajectories. We present a unified model architecture for simultaneous agent future heatmap estimation, in which we leverage hierarchical and sparse image generation for fast and memory-efficient inference. We propose a learnable trajectory recombination model that takes as input a set of predicted trajectories for each agent and outputs its consistent reordered recombination. This recombination module is able to realign the initially independent modalities so that they do no collide and are coherent with each other.  We report our results on the Interaction multi-agent prediction challenge and rank $1^{st}$ on the online test leaderboard.

Published as a conference paper at ICLR 2022
THOMAS: TRAJECTORY HEATMAP OUTPUT WITH LEARNED MULTI-AGENT SAMPLING
Thomas Steffano Sabatini', Dzmitry Tsishkou', Bogdan Stanciulescu' Fabien Moutarde? Ioo team, Paris Research Center, Huawei Technologies France CCenter for robotics, MINES ParisTech (stefano. sabatini, dzmitry. telyystssskkoon tiskkouu @huawei. com (thomas. gilles, bogdan stanscilsscniit...uuu fabien. moutarde) Senptaarrdeemmeeee-pisseeeee fr
ABSTRACT
In this paper, we propose THOMAS, joint multi-agent trajectory prediction framework allowing for an efficient and consistent prediction of multi-agent multi- modal trajectories. We present unified model architecture for simultaneous agent future heatmap estimation, which we leverage hierarchical and sparse image generation for fast and memory-efficieee inference. We propose learnable jectory recombination model that takes as input a set of predicted trajectories for each agent and outputs its consistent reordered recombination. This recombina- tion module is able to realign the initially independent modalities so that they do collide and coherent with each other. We report results the Interac- tion multi-agent prediction challenge and rank 1st on the online test leaderboard.
INTRODUCTION
Motion forecasting is an essential step in the pipeline of an autonomous driving vehicle, transform- ing perception data into future prediction which are then leveraged to plan the future moves of the autonomous cars. The self-driving stacks needs predict the future trajectories for all the neighbor agents, in fast and coherent way.
The interactivity between agents plays an important role for accurate trajectory prediction. Agents need to be aware of their neighbors in order to adapt their speed, yield right of way and merge in neighbor lanes. To so, different interaction mechanisms have been developed, such as social
pooling (Alahi al. 2016; Lee et al., 2017 Deo & Trivedi (0018), graphs (Salzmann et al.) 2020
Zeng et al. 2021) or attention (Mercat et al. 2020 Mcssaoud et al. 2020; Luo et al.) 2020 Gao
a 2020 Liang al. 2020). benefiting from the progress of powerful transformer architectures
(Li et al. 2020; Yuan et al.) [2021: Girgis et al., 2021 Ngiam et al., (021). These mechanisms
allow agents look and share features with neighbors and to take them into account in their own predictions.
Multi-modality is another important aspect of the possible future trajectories. A car can indeed chose to turn right or left, or decide to realise certain maneuver in various ways. Uncertainty modeled variance of Gaussians insufficient model these multiple cases, as can only represent continuous spread and cannot show multiple discrete possibilities. Therefore, current state-of-the-an produces not one but K possible trajectories for each agent predicted, and most recent benchmarks (Caesar et al. 2020: CCang et al., 2019; Zhhan et al. 2019 Ettinger et al., (2221) include multi- modality in their metrics, taking only the minimum error over predicted set K trajectories.
However, up until very recently and the opening of multi-agent joint interaction challenges (Ettinger) et al. 2021 Zhan al.) (022), no motion forecasting prediction datasets were taking into account the coherence of modalities between different agents predicted at the same time. As result, the most probable predicted modality given agent could crash with the most probable modality of another agent.
Our THOMAS model encodes the past trajectories of all the agents present in the scene, as well the HD-Map lanelet graph, and then predicts for each agent sparse heatmap representing the future probability distribution at a fixed timestep in the future, using hicrarchical refinement for very efficient decoding. A deterministic sampling algorithm then iteratively selects the best K trajectory

---

Published as a conference paper at ICLR 2022
endpoints according to the heatmap for each agent, in order to represent wide and diverse spec- trum of modalities. Given this wide spectrum of endpoints, a recombination module takes care of addressing consistency in the scene among agents.
Our contributions are summarized as follow:
We propose hierarchical heatmap decoder allowing for unconstrained heatmap genera- tion with optimized computational costs, enabling efficient simultaneous multi-agent pre- diction.
We design a novel recombination model able to recombine the sampled endpoints to obtain scene-consistent trajectories across the agents.
2
RELATED WORK
Learning-based models have quickly overtaken physics-based methods for trajectory prediction for several reasons. First, the sequential nature of trajectories is a logical application for recurrent architectures (Alahi
2016; Altché & de La Fortelle| 2017: Lee al., 2017 Mereat et al. 2020 Khandelwal et al.) 2020). Then, benefiting from the latest progresses in computer vision,
convolutional layers can easily be applied to bird-view rasters of the map context (Lee et al. 2017 Tang & Salakhutdinov 2019) Cui et aal.]2011)) Hong et al. 2019 (Salzmann et al.] | Chai et al.,
Gilles et al. b0. A more compact representation closer to the trajectory space can encode surrounding HD-Maps (usually formalized as connected lanelets), using Graph Neural Networks
2020 2020]Gilles
(Gao et al. 2020 Liang et al. 2020 Zeng et al. 2021 Gilles et al.) (021a). Finally, some point-
broader way to trajectory prediction, both lanes and trajectories can be considered as ordered set of points.
based approaches Ye et al. 2021) can be applied
Multi-modalite in prediction can be obtained through multiple prediction head in the model (Cui et al. 2019: Liang et al.] 2020) Ngiam et al. 2021] [eo et al.] 2021). However, some methods
rather adopt candidate-based approach where potential endpoints
obtained either from anchor trajectories through clustering (Chai et al. 2020: Phan-Minh et al. (2020) or from a model-based
of candidates from the context graph (Zhang et al. 2020; Zhao et al., 2020 Zeng et al. 2021 Kim et al., 2021) or a dense grid Gilles et al.]] 22222 a). Another family of approaches use variational inference to generate diverse predictions through latent variables (Lee
generator (Song al. (021). Other approaches use a broader
around the target agent (Deo & Trivedi| 2020 Gu al. a |021]]
|
al.
2017
Rhinehart et al. 2018
Tang & Salakhutdinov 2019 Casas al. 2020) or GAN (Cuupta
et al. 2018 Rhinehart et al.
2018 Saddghian et al.) (019) but the sampling of these trajectories is
stochastic and does not provide any probability value for each sample.
While very little work has directly tackled multi-agent prediction and evaluation so far, multiple methods hint at the ability to predict multiple agents at the same time (Liang et al., 2020 Ivanovic et al. 2020 Zeng al. 2021) even if they then focus on more single-agent oriented framework. Other works (Alahi et al. 2016 Tang & Salakhutdinov 2019 Rhinchart et al.) 2019 Girgis et al. 2021 autoregressive roll-outs
condition the future step of an agent the previous steps of all the other agents. Scene' Transforrne (Ngiam et al. 2021) repeats each agent features across possible modalities, and performs self-attention operations inside each modality before using a loss computed jointly among agents to train a model and evaluate on the WOMD dataset (Ettinger et al., 202T) interaction track. ILVM (Casas et al. 2020) uses scene latent representations conditioned on all agents to generate cenee-consittent samples, but its variational inference does not provide a confidence score for each modality, hence LookOut (Cui et al., (02T)) proposes scenario scoring function and a diverse sampler to improve sample efficiency. AIR2 (Wu & Wu 2021) extends Multipath (Chai et al.) 2020) and produces a cross-distribution for two agents along all possible trajectory anchors, but scales exponentially with the number of agents, making impractical for real-time implementation that could encounter more than 10 agents at the same time.
The approach most related to this paper is GOHOME (Gilles et al. 2021a), which uses similar graph encoder and then leverage lane rasters to generate a probability heatmap in a sparse manner. However, this lane-gem aen-ggeerated heatmap remains constrained to the drivable area and to fixed lane- widths. Another close approach is the one of DenseTNT (Gu et al. 202T), which also uses attention to grid of points in order to obtain dense prediction, but their grid also remains constrained to
2

---

Published as a conference paper at ICLR 2022
Endpoint recombination
Heatmap
estimation
Endpoint
sampling
Figure 1: Illustration of the THOMAS multi- prediction pipeline
neighborhood of the drivable area. Finally, none of these previous methods considers the problems of scene consistency for multi-agent prediction.
METHOD
Our goal to predict the future 7 timesteps of agents using their past history made of H timesteps and the HD-Map context. Similar to recent works (Zhao et al. [0220] Zeng et al. 2021 Gu et al., 2021), we will divide the problem into goal-based prediction followed by full trajectory reconstruc- tion. Our prediction pipeline is displayed in Fig, [ We first encode each agent trajectory and the HD-Map context graph into a common representation. We then decode a future probability heatmap for each agent in the scene, which we sample heuristically maximize coverage. Finally, we combine the sampled endpoints into scene-consistent modalities across agents and build the full trajectories for each agent.
Our pipeline shares the same graph encoder, sampling algorithm and full trajectory generation as GOHOME (Gilles et al. (202Ia), but uses a novel efficient hierarchical heatmap process that enables to scale to simultancous multi-agent prediction. Furthermore, we add a novel scene-consistency module that recombines the marginal outputs into a joint prediction.
3.1 MODEL BACKBONE
3.1.1 GRAPH ENCODER
We use the same encoder the GOHOME model (Gilles et al. encoded though Traj Encoder using ID CNN followed by a UGRU recurrent layer (Rozenberg al. 2021). and the HD-Map is encoded as lanelet graph using GNN CraphEnncoie made of graph convolutions. agent features, followed by self-attention Agents2Agents observe interaction between agents. The final result is an encoding Fa for each agent, where history, context and interactions have been summarized. This encoding F. is used in the decoder operations and is also stored to be potentially used in modality recombination described Sec. 3.22 The resulting architecture of these
2021a). The agent trajectories are
then run cross-attention Lames2Agenis to add context information to the
encoding operations is illustrated in the first half of Fig. 2
Ageil Oogn
Humap1
HD Kip
Agant
Gayh
LansszAgerts
Age
Nas Agants festura
Figure 2: Model architecture for multi-agent prediction with shared backbone
3

---

Published as a conference paper at ICLR 2022
Figure 3: Hicrarchical iterative refinement of the grid probabilities. First, the full grid is evaluated at very low resolution, then the highest cells are up- up-sampled and evaluated at a higher resolution, until final resolution is reached. We highlight in grey the restricted area considered for refinement at each step.
3.1.2 HIERARCHICAL GRID DECODER
Our aim here is decode each agent encoding into a heatmap representing its future probability distribution at prediction horizon T. Since create this heatmap for each agent in the scene, the decoding process has to be fast so that it can be applied to a great number of agents parallel. We use hierarchical predictions at various levels of resolutions so that the decoder has the possibility of predicting over the full surroundings of the agent but learns to refine with more precision only in places where the agent will end up with high probability. This hierarchical process is illustrated in Fig. 3
Starting from initial full dense grid probability low resolution Ro Ro pixels, we iteratively refine the resolution by a fixed factor f until we reach the desired final resolution Rfinal X R fiinat- At each iteration we select only the N₂ highest ranking grid points from the previous iteration, and upsample only these points to the R4 x R4 = RI-1/f x R.-1/s. At each step, the grid points features are computed by 2-layer MLP applied on the point coordinates, these features are then concatenated to the agent encoding and passed through a linear layer, and finally enriched by a 2- layer cross-attentieg on the graph lane features, before applying linear layer with sigmoid get the probability.
For given W output range, this hierarchical process allows the model to only operate on W/R, x W/Ro + >, N₁ x f2 grid points instead of the W/Ryinat X W ///Remul available. In practice, for final output range of 192 meters with desired Rymeel = 0.5m resolution, we start with an initial resolution of Rg Sm and use two iterations of (N1, = (16, 64) points each and upscaling factor f = This way, we compute only 1856 grid points from the 147 456 available, with no performance loss.
The heatmap is trained on each resolution level using as pixel-wise focal loss as Gilles et al. (021b), detailed in Appendix A.3 with the ground truth being Gaussian centered at the target agent future position.
3.1.3 FULL TRAJECTORY GENERATION
Having heatmap as output grants us the good property of letting us decide to privilege coverage in order to give a wide spectrum of candidates the scene recombination process. From each heatmap, therefore decode K end points using the same MissRate optimization algorithm as Gilles et al. 2021b). We then generate the full trajectories for each end point using also the same model, fully-connected MLP. The MLP takes the car history and predicted endpoint input and produces cooooddiinatss representing the full future trajectory. At training time, this model is trained using ground-truth endpoints.

---

Published as a conference paper at ICLR 2022
Agent 1
Agent
Agent Agent
Ax
AxKxD
Ageent modallsy A
Agent mo allallyyy.oo At
Agent modaltty
Agent modality
Age modality
Agennt alliy
Agent mon.ttmmtalllii dalitalyyy...
Agen moda
sanyen
Keyy
Scalar
product
Scene modality
Scene modality S.
Queries
Cross-attention
Scene modality
Scene modality
LxD
Linear recombination
LxAx2
Figure Illustration of THOMAS methods for generation of scene-consistent agent modalities
3.2 MODALITY RECOMBINATION FOR MULTI- AGENT CONSISTENT PREDICTION
The difficulty in multimodal multi-agent prediction resides in having coherent modalities between each agent. Since the modalities considered beenewwise, each agent's most probable modality has to match with the most probable prediction of the other agents, and so on. Moreover, these modalities must not collide with each other, they should represent realistic scenarios.
Initially, the prediction output of the model can be defined as marginal, as all A agents have been predicted and sampled independently in order to get K possible endpoint each. Our goal is to output a set of scene predictions J = (I,, A) from the marginal prediction M = (A, K),, where each scene modality 1 belonging to is association of endpoints pf for each agent To achieve this, our main hypothesis lays in the fact that good trajectory proposals are already present in the marginal predictions, but they need be coherently aligned among agents achieve consistent scene predictions. For given agent a, the modality selected for the scene would be combination of this agent available marginal modalities PL M. Each scene modality [ would select different association between the agents.
of overall
We illustrate our scene modality generation process in Fig. The model learns L scene modality embeddings Sj of D features each. K x A agent modality vectors Ag are also derived from each agent modality endpoint. These vectors are obtained through 2-layer MLP applied on the agent modality coordinates pk to which the stored agent encoding F, (previously described in Sec. 3.1.1) concatenated in order to help the model recognise modalities from the same agent. The scene modality vectors S are 'specialized" by querying the available modality proposals Ağ of each agent through cross-attention layers. Subsequently. a matching score cf between each scene modality S and each agent modality A% is computed. This matching score can be intuitively interpreted as the selection of the role (maneuver k) that agent a would play in the overall traffic scene
d.a
Sp.Ag"
Since argmax is non-differentiable, we employ a soft argmax as a weighted linear combination of the agent modalities pf using a softmax on the k.a scores:
pf Σ softmar(c
ka
The recombination module trained with the scene-level minimum displacement error, where for each modality inside single scene all predicted agent displacement losses averaged before taking the minimum (winner-takes takes-lll loss) over the L scene modalities. formal definiton this loss, also used as metric, is given in Eq. in Sec. 4.2 (minSFDF:.).
5

---

Published as a conference paper at ICLR 2022
4 EXPERIMENTS
4.1 DATASET
We use the Interaction v1.2 dataset that has recently opened a new multi-agent track in the context of its Interpret challenge. It contains 47 584 training cases, 11 794 validation cases and 2 644 testing cases, with each case containing between and 40 agents to predict simultaneously.
4.2 METRICS
For a set of k predictions Ptz for each agent in scene, we report the minimum Final Displacement Error (minFDE;; to the ground truth pe and the MissRate (MR;). whose definition of miss specified according to the evaluated dataset in Appendix A In their usual marginal definition, these metries are averaged over agents after the minimum operation, which means that the best modality of cach agent is selected independently for each and then averaged:
1 Σ minFDE, = A A
min: lpph 12, MR = - Σ A
ming 1 missk
(1)
For consistent scene multi-agent prediction, we report the joint scene-level metrics, where the aver- age operation over the agents done before the minimum operator. this formulation, the mini- mum is taken over scene modalities, meaning that only the best scene (joint over agents) modality taken into account:
I minSFDEA = mink Σ Ii 12 SMR, = ming Á
Σ
a miss k
In other words, the marginal metrics pick their optimal solutions in pool of & to the power of predicted solutions, while the joint metries restrict this pool only possibilities, making a much more complex problem.
We also report the scene collision rate (SCR) which is the percentage of modalities where two or more agents collide together, and the consistent collision-free joint Miss Rate (cSMR), which is SM where colliding modalities are also counted misses.
4.3 COMPARISON WITH STATE-OF-THES
We compare our THOMAS model performance with other joint predictions methods ILVM (Casas al. 2020) and Trene'rrasformer (Ngiam al., 2021]. For fair comparison, use GOHOME encoder for each of the method, and adapt them accordingly so that they predict only endpoints similar our method. For each method, we focus on implementing the key idea meant solve scene consistency and keep the remaining part of the model as close as possible to our approach for fair comparison:
ILVM (Casas et al. 2020) uses variational inference to learn latent representation of the scene conditioned of each agent with Scene Interaction Module, and decodes it with a similar Scene Interaction Module. The required number of modalities is obtained by sampling the latent space as many times as require. Even though the sampling is independent for each agent, the latent space generated conditionally on all the agents.
Transfrrrnr (Ngiam et al., 2021) duplicates the agent encoding with the number of modal- ities required and adds one-hot encoding specific to each modality. They then apply a shared transformer architecture on all modalities to encode intra-modality interactions between agents and generate the final trajectories.
More details on the state-of-the-art method and the re-implementations we used are given in Ap- pendix A.4 The results are reported in Tab. [ While having comparable marginal distance perfor- mance (demonstrating that our model is not inherently more powerful or leveraging more informa- tion), THOMAS significantly outperforms other methods every joint metric. SMR is improved by about 25% and SCR by almost 30%, leading to a combined cSMR decreased by also more than 25%. We also report numbers from the Interpret multi-agent online test set leaderboard in Tab.

---

Published as a conference paper at ICLR 2022
Table 1: Comparison of consistent solutions on Interpret multi-agent validation track
Marginal metrics
Joint metrics
mADE
mFDE
MR
mFDE
MR
SCR
cMR
2020) SceneTrantones Trantormer Ngam et al. 2021
ILVM (Casas al.
0.30
0.62
10.8
0.84
19.8
5.7
21.3
0.29
0,59
10.5
0.84
15.7
3.4
17.3
THOMAS
0.31
0.60
8.2
0.76
11.8
2.4
12.7
of Appendix A.5 For better comparison with existing single-agent trajectory prediction state-of- the-art, we provide evaluation numbers on the single-agents benchmarks Argoverse (Chang et al., 2019) and NuScenes (Caesar et al. 2020) in Appendix A.7
4.4 ABLATION STUDIES
4.4.1
RECOMBINATION MODULE
We establish the following baselines assess the effects our THOMAS recombination.
Scalar output: we train model with the GOHOME graph encoder and multimodal scalar re- gression head similar to Liang et al. (2020. We optimize with either marginal and joint loss formulation.
Heatmap output with deterministic sampling: we try various sampling methods applied on the heatmap, with either the deterministic sampling described in Gilles et al. (2021a) or a learned sampling trained to directly regress the sampled modalities from the input heatmap.
We report the comparison between the aforementioned baselines and THOMAS in Tab. 2 Com- pared to these baselines, THOMAS can be seen as an hybrid sampling method that takes the result of deterministic sampling as input and learns recombine it into more coherent solution. With regard to the joint algorithmic sampling that only tackled collisions but has little to no effect on actual consistency, highlighted by the big drop in collision rate from 7.2% 2.6% but similar joint SMR, THOMAS actually brings lot of consistency in the multi-agent prediction and drops the joint SMR from 14.8% to 11.8%.
Table 2: Comparison of consistent solutions on Interpret multi-agent validation track
Marginal metrics
Joint metrics
Output
Sampling
Objective
mADE
mFDE
MR
mFDE
MR
Col
cMR
Scalar
Marg
0.28
0.59
10.4
1.04
23.7
6.4
24.9
Scalar
Joint
0.34
0.77
16.2
0.90
19.9
49
21.7
Heat
Learned
Marg
0.26
0.46
4.9
0.98
20.9
4.1
21.9
Heat
Learned
Joint
0.29
0.58
9.8
0.88
15.2
3.0
16.4
Heat
Algo
Marg
0.29
0.54
3.8
0.83
14.8
7.2
15.9
Heat
Algo
Joint
0.29
0.54
0.83
14.8
2.6
15.6
Heat
Combi
Joint
0.31
0.60
8.2
0.76
11.8
2.4
12.7
Usually, scalar marginal models already suffer from learning difficulties as only one output modality, the closest one to ground, can be trained at a time. Some modalities may therefore converge faster to acceptable solutions, and benefit from much increased number of training samples compared to the others. This problem is aggravated in the joint training case, since the modality selected is the same for all agents in a training sample. The joint scalar model therefore actually fails to learn multi-modality as illustrated by higher marginal minFDEg than any other model, and an extremely high crossCollisionRaaee since some modalities never train and always point to the same coordinates regardless of the queried agent. Note that, despite similar training loss, SceneTranste Transformer doesn't 't suffer of the same pitfalls in Tab. D as is shares the same weights between all modalities and only differentiates them in the initialization of the features.

---

Published as a conference paper at ICLR 2022
4.5 HIERARCHICAL HEATMAP REFINEMENT
We assess the speed gain of our proposed hierarchical decoder compared to the lane rasters of GO- HOME Gilles al. aa). Tab. 3 We report training time for epochs with a batchsize of 16, and inference time for 32 and 128 simultaneous agents.
Table 3: Comparison of consistent solutions on Interpret multi-agent validation track
Training
Inference 32 agents
Inference 128 agents
GOHOME THOMAS
12.8 hours
36 ms
90 ms
7.5 hours
20 ms
31 ms
For additional comparison, the other existing dense prediction method DenseTNT (Gu et al. 2021 reports an inference speed of 100ms per sample for their model.
4.6 QUALITATIVE EXAMPLES
In this section, we will mostly compare the model before recombination, which we will refer by the Before model, to the model after recombination, referenced as the After model. We display four qualitative examples in Fig. with colliding modalities in the fere model (in dashed orange) and the solved modality (in full line orange) after recombination. For each model @Before-dasher or A full,, the highlighted modality in orange the best modality according to the SM Rg metric among the 6 available modalities. We also display in dashed grey the other 5 predicted Bef modalities, and highlight that the recombination model indeed selects modalities already available in the vanilla set and reorders them so that non-colliding modalities are aligned together.
4
Figure 5: Qualitative examples recombination model assembling collision-fre free modalities to- gether compared initial colliding modalities. For each example display the general context with highlighted agents and area of interest, then two zooms on the agents, one displaying the initial best modality before recombination dashed orange and all the other available modalities in grey. The second zooms shows the best modality after recombination in full line orange.
We also show more qualitative examples in Fig. 6 where we highlight the comparison in modality diversity between the Be Bore model (in dashed lines) and the After model (in full lines). While the Before model tries to spread the modalities for all agents to minimize marginal miss-rate, the recombined model presents much less spread compared to the original model, maintaining a multimodal behavior only in presence of very different possible agent intentions such as different possible exits or turn choices. For most other agents, almost all modalities are located at the same position, that is, the deemed the most likely by the model. Thus, if the truly uncertain agents
8

---

Published as a conference paper at ICLR 2022
Figure 6: Qualitative examples of recombination model selecting fewer but more pronounced and impactful agent modalities compared to initial colliding modalities. For each example display on the left the vanilla model modalities with dashed lines, with the initial best modality in dashed orange and all the other available modalities in grey. On the right display the selected modalities after recombination, where the model focuses on the most likely occurrence in most agents.
have to select the second or third most likely modality, the other agents still have their own most likely modality
CONCLUSION
We have presented THOMAS, recombination module that can be added after any trajectory pre- diction module outputing multi-modal predictions. By design, THOMAS allows to generate scene- consistent modalities across all agents by making the scene modalities select coherent agent modal- ities and restricting the modality budget on the agents that truly need it. We show significant per- formance increase when adding the THOMAS module compared the vanilla model and achieve tate-offthe-ar results compared to already existing methods tackling scene-consistent predictions.

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We use the publicly available Interaction 1.2 dataset (Zhan al. 2019 available http:// challenge
com./ccco//// datp/atttttttttcmma down Load We detail dataset pre- processing Appendix|A.2) training process Appendix A.3 and give precise illustration of our model architecture with each layer size in A.8
REFERENCES
Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social Istm: Human trajectory prediction in crowded spaces. In CVPR, 2016.
Florent Altché and Arnaud de La Fortelle. An lstm network for highway trajectory prediction. In ITSC, 2017.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Kr- ishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes prediction competition. wtwpeeiiiooooottps.sss.....tt hetps: //eval. ai /tw////w//////// oca.leewww..aallgggggaaa///////r cessed: 2021-11-20.
Ac-
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving. In CVPR, 2020.
Sergio Casas, Cole Gulino, Simon Suo, Katie Luo, Renjie Liao, and Raquel Urtasun. Implicit latent variable model for scene-consistent motion forecasting. In ECCV, 2020.
Yuning Chai, Benjamin Sapp. Mayank Bansal, and Dragomir Anguelov. Multipath: Multiple prob- abilistic anchor trajectory hypotheses for behavior prediction. In CoRL, 2020.
Ming-Fang Chang. John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hart- nett, De Wang. Peter Carr, Simon Lucey, Deva Ramanan, et al. Argoverse motion fore- casting competition. httpps://vve http:///ww...eeeeeg/////aaeeegeo Leaderboarr//1777 Accessed: 2021-11-20.
wwtwwb//wweeeeeeaa..eee..eeeeeee http////chhaaeeggeeee.eeee http://haaaeeeeeeeeeeeeeeeeeeeee /44/
Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hart- nett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, et al. Argoverse: 3d tracking and forecasting with rich maps. In CVPR, 2019.
Alexander Cui, Sergio Casas, Abbas Sadat, Renjie Liao, and Raquel Urtasun. Lookout: multi-future prediction and planning for self-driving. In ICCV, 2021.
Diverse
Henggang Cui, Vladan Radosavljevic, Fang-Chin Chich Chou, Tsung Lin, Thi Nguyen, Tzu-Kuo Huang. Jeff Schneider, and Nemanja Djuric. Multimodal trajectory predictions for autonomous driving using deep convolutional networks. In ICRA, 2019.
Nachiket Deo and Mohan M Trivedi. Convolutional social pooling for vehicle trajectory prediction. In CVPR, 2018
Nachiket Deo and Mohan M Trivedi. Trajectory forecasts in unknown environments conditioned on grid-based plans. arXiv:2001. 00035, 2020.
Nachiket Deo, Eric M Wolff, and Oscar Beijbom. Multimodal trajectory prediction conditioned on
lane-graph traversals. In CoRL, 2021.
Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles Qi, Yin Zhou, al. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. arXiv:2104. 10133, 2021.
Jiyang Gao, Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguelov, Congcong Li, and Cordelia Schmid. Vectornet: Encoding hd maps and agent dynamics from vectorized representation. In CVPR, 2020.
10

---

Published as a conference paper at ICLR 2022
Thomas Gilles, Stefano Sabatini, Dzmitry Tsiskkou, Bogdan Stanciulescu, and Fabien Moutarde. Gohome: Graph-oriented heatmap output forfuture motion estimation. arXiv preprint arXiv:2108.09640, 2021a.
Thomas Gilles, Stefano Sabatini, Dzmitry Tsishkou, Bogdan Stanciulescu, and Fabien Moutarde. Home: Heatmap output for future motion estimation. In ITSC, 2021b.
Roger Girgis, Florian Golemo, Felipe Codevilla, Jim Aldon D'Souza, Martin Weiss, Samira Ebrahimi Kahou, Felix Heide, and Christopher Pal. Autobots: Latent variable sequen- tial set transformers. arXiv:2 00563, 2021.
Junru Gu, Chen Sun, and Hang Zhao.
Densetnt: End-to-end trajectory prediction from dense goal
sets. In ICCV, 2021.
Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks. In CVPR, 2018.
Joey Hong, Benjamin Sapp, and James Philbin. Rules of the road: Predicting driving behavior with convolutional model of semantic interactions. In CVPR, 2019.
Boris Ivanovic, Amine Elhafsi, Guy Rosman, Adrien Gaidon, and Marco Pavone. Mats: An inter- pretable trajectory forecasting representation for planning and control. In CoRL, 2020.
Siddhesh Khandelwal, William Qi, Jagjeet Singh, Andrew Hartnett, and Deva Ramanan. What-if
motion prediction for autonomous driving. arXiv:2008. 10587, 2020.
ByeoungDo Kim, Scong Hyeon Park, Seokhwan Lee, Elbek Khoshimjonov, Dongsuk Kum, Junsoo Kim, Jeong Soo Kim, and Jun Won Choi. Lapred: Lane-aware prediction of multi-modal future trajectories of dynamic agents. In CVPR, 2021.
Namhoon Lee, Wongun Choi, Paul Vernaza, Christopher B Choy, Philip HS Torr, and Manmohan Chandraker. Desire: Distant future prediction in dynamic scenes with interacting agents. In CVPR, 2017.
Lingyum Luke Li, Bin Yang, Ming Liang, Wenyuan Zeng, Mengye Ren, Sean Segal, and Raquel Urtasun. End-to-end contextual perception and prediction with interaction transformer. IROS, 2020.
Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng. and Raquel Urtasun. Learning lane graph representations for motion forecasting. In ECCV, 2020.
Chenxu Luo, Lin Sun, Dariush Dabiri, and Alan Yuille. Probabilistic multi-modal trajectory predic- tion with lane attention for autonomous vehicles. arXiv:2007.02574, 2020.
Jean Mercat, Thomas Gilles, Nicole El Zoghby, Guillaume Sandou, Dominique Beauvois, and Guillermo Pita Gil. Multi-head attention for multi-modal joint vehicle motion forecasting. In ICRA, 2020.
Kaouther Messaoud, Nachiket Deo. Mohan M Trivedi, and Fawzi Nashashibi. Multi-head at- tention with joint agent-map representation for trajectory prediction in autonomous driving. 20Kk::2005.02555. 2020.
Xiaoyu Mo, Yang Xing, and Chen Lv. Recog: A deep learning framework with heterogeneous graph for interaction- aware trajectory prediction. arXiv:2012.05032, 2020.
Sriram Narayanan, Ramin Moslemi, Francesco Pittaluga, Buyu Liu, and Manmohan Chandraker. Divide-and-conquer for lane-aware diverse trajectory prediction. In CVPR, 2021.
Jiquan Ngiam. Benjamin Caine, Vijay Vasudevan, Zhengdong Zhang, Hao-Tien Lewis Chiang, Jeffrey Ling, Rebecca Roclofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, et al. Scene transformer: A unified multi-task model for behavior prediction and planning. arXiv preprint 1rXiv:2106.084/7, 2021.
Tung Phan-Minh, Elena Corina Grigore, Freddy A Boulton, Oscar Beijbom, and Eric M Wolff. Covemet: Multimodal behavior prediction using trajectory sets. In CVPR, 2020.

---

Published as a conference paper at ICLR 2022
Nicholas Rhinehart, Kris M Kitani, and Paul Veraza. R2p2: A reparameterized pushforward policy for diverse, precise generative path forecasting. In ECCV, 2018.
Nicholas Rhinehart, Rowan McAllister, Kris Kitani, and Sergey Levine. Precog: Prediction condi- tioned on goals in visual multi-agent settings. In CVPR, 2019.
Raphaël Rozenberg, Joseph Gesnouin, and Fabien Moutarde. Asymmetrical bi-rnn for pedestrian trajectory encoding. arXiv:2/06.04419, 2021.
Amir Sadeghian, Vineet Kosaraju, Ali Sadeghian, Noriaki Hirose, Hamid Rezatofighi, and Silvio Savarese. Sophie: An attentive gan for predicting paths compliant to social and physical con- straints. In CVPR, 2019.
Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically-fessible trajectory forecasting with heterogeneous data. In ECCV, 2020.
Haoran Song, Di Luan, Wenchao Ding, Michael Yu Wang, and Qifeng Chen. Learning to predict vehicle trajectories with model-based planning. In CoRL, 2021.
Yichuan Charlie Tang and Ruslan Salakhutdinov. Multiple futures prediction. In NeurIPS, 2019.
Chuhua Wang, Yuchen Wang, Mingze Xu, and J Crandall. Stepwise goal-driven networks for trajectory prediction. 1xXiv:2/33./4107 2021.
David Wu and Yumnan Wu. Air for interaction prediction. Waymo Open Dataset Challenges Re- ports, CVPR Workshop on Autonomous Driving, http:::/cvp:t/02t wad. vision/ 2021.
Maosheng Ye, Tongyi Cao, and Qifeng Chen. Tpen: Temporal point cloud networks for motion
forecasting. In CVPR, 2021.
Ye Yuan, Xinshuo Weng, Yanglan Ou, and Kris Kitani. Agentformer: Agent-aware transformers for socio-temporal multi-agent forecasting, 1002: 2021.
Wenyuan Zeng, Ming Liang, Renjie Liao, and Raquel Urtasun. Lanerenn: Distributed representa-
tions for graph-centric motion forecasting. In IROS, 2021.
Wei Zhan, Liting Sun, Di Wang. Haojie Shi, Aubrey Clausse, Maximilian Naumann, Julius Kum- merle, Hendrik Konigshof, Christoph Stiller, Arnaud de La Fortelle, et al. Interaction dataset: An international, adversarial and cooperative motion dataset in interactive driving scenarios with semantic maps. arXiv: 19100009,,,3 2019.
Wei Zhan, Liting Sun, Di Wang, Haojie Shi, Aubrey Clausse, Maximilian Naumann, Julius Kum- merle, Hendrik Konigshof, Christoph Stiller, Arnadd de La Fortelle, et al. Interpret: Interaction- dataset-based prediction challenge. http: http://hhaalnngeeee interact wwweeeraattttnteeeeeeeeee dataset.... com/prediction challenge intro 2021. Accessed: 2021-10-03.
Lingyao Zhang, Po-Hsun Su, Jerrick Hoang, Galen Clark Haynes, and Micol Marchetti-Bowich Map-adaptive goal-based trajectory prediction. In CoR1,,0000
Hang Zhao, Jiyang Gao, Tian Lan, Chen Sun, Benjamin Sapp. Balakrishnan Varadarajan, Yue Shen, Yi Shen, Yuning Chai, Cordelia Schmid, et al. Tnt: Target-driven trajectory prediction. CoRL, 2020.
12

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 METRIC DETAILS
We adopt for each dataset evaluation the miss definition as specified by the dataset benchmark. Argoverse (Chang et al. 2019) and NuScenes (Caesar et al. (2020) define miss as the prediction being further than 2 meters to the ground truth, while Waymo (Ettinger et al., (202)) and Interaction Zhan et al. count miss when the prediction is closer than lateral (Im) and longitudinal
threshold with regard to speed:
8 < 1.4m/s
Threshold
1 11-14
1.4m/s 8 S v S 11m/s
v 2 11m/s
A.2 DATASET PROCESSING
We use the training/validation split provided in Interaction 1.2. We select for each scene a reference agent on which the scene will centered and oriented according its heading at prediction time. At training time, this reference agent is chosen randomly across all agents. At validation and testing time, we compute the barycenter of all agents and select the closest agent as the scene reference point. We use for training all agents that have ground-truth available at prediction horizon T=3s.
We use the provided dataset HD-Maps to create the context graphs. As in Liang et al. 22020), we four relations: predeces successor, left, right) obtained from the lane graph connectivity. We upsample long lanelets to have a maximum of 10 points in each lanelet.
For each agent, sampled at 10Hz. We also provide the model a mask indicating if the agent was present in a given time-frame, and pad with zeros when the agent is not tracked at specific timestep.
their trajectory history made of position, yaw and speed in the past second
A.3 TRAINING DETAILS
We train all models with Adam optimizer and batchsize 32. and divide it by 2 at epochs 3, 6. and 13, before stopping the training epoch 16. We use ReLU activation after every linear layer unless specified otherwise, and LayerNormalization after every attention and graph convolution layer.
We initialize the learning rate at 1
During training, since scene can contain up to 40 agents, which is too much for gradient compu- tation in batches of 32, restrict the number of predicted agents by randomly sampling them across available agents. We do not use other data augmentation.
The final heatmap Y predicted cover range of 192 meters resolution 0.5m, hence (384, 384) image. For heatmap generation, we use the same pixel-wise focal loss over the pixels p as Gilles al. (2021b):
(og({p) (11 (1-Y,) log(1 8,)
if Y else
L -o,-
with f(Y,, X,)
where Y is defined as Gaussian centered the agent future position at prediction horizon T with a standard deviation 4 pixels, equivalent to 2m.
A.4 BASELINES IMPLEMENTATIONS
A.4.1 IMPLICIT LATENT VARIABLE MODEL.
We use a GOHOME encoder for the prior, posterior and decoder Scene Interaction Modules. We weight the KI. term with B which worked best according experiments.
13

---

Published as a conference paper at ICLR 2022
A.4.2 SCENE TRANSFORMER
The initial paper applies a transformer architecture on a [F' A, DJ tensor where F the potential modality dimension, A the agent dimension and T the time dimension, with D the feature embed- ding, with factorized self-attention the agent and time dimensions separately, so that agents can look at each-other inside specific scene modality. The resulting output is optimized using jointly formalized loss. For our implementation, we get rid the T dimension as we focus on endpoint prediction and coherence between the A agents. The initial encoded [4, D] tensor is obtained with a GOHOME encoder, multiplied across the futures and concatenated with a modality-specifit one-hot encoding as in ingiam et al. (2021 to obtain the [F, A, DJ tensor. We then apply two layers of agent self-attention similar to the original paper, before decoding the endpoints through a MLP.
A.4.3 COLLISION ENDPOINT SAMPLING BASELINE
We designed a deterministic sampling algorithm based on the heatmaps generated in previous section in order sample endpoints for each agent in a collision aware manner. We the sampling algorithm as Gilles et al. (2021a) based on MR optimization, but add a sequential iteration over the agents for each modalities.
For a single modality k, we predict the possible endpoint of a first agent a by taking the maxi- mum accumulated predicted probability under area of radius r. We then not only set to zero the heatmap values of this agent heatmap ZE. around the sampled location so not to sample it in the next modalities k', but also set to zero the same same modality k so that these other agents cannot be sampled at the same position for this modality. This way, try to enforee collision-free endpoints, and expect that considering collisions brings logic to improve the overall consistency of the predictions. However, as will be highlighted in Sec. 4.4 this methods significantly improves the collision rate without the need for any additional learned model but it does barely improve the multi agent consistency.
the heatmaps I of the other agents a' on the
A.5 RESULTS ON INTERPRET CHALLENGE
We also report our numbers from the Interpret multi-agent track challenge online leaderboard in Table is to be noted that the DenseTNT solution explicitely checks for collisions in the search for its proposed predictions, which we don't, t, hence their 0% collision rate (SCR) and its direct impact on consistent collision-free joint Miss Rate (cSMR).
Table Results on Interpret multi aulti-agent regular scene leaderboard (test set)
minSADE
minSFDE
SMR
SCR
cSMR
MoliNet
0.73
2.55
44.4
7.5
47.4
ReCoG2 (Mo al. 2020)
0.47
1.16
23.8
6.9
26.8
Dense (Gu et al. 2021)
0.42
1.13
22.4
0.0
22.4
THOMAS
0.42
0.97
17.9
12.8
25.2
Table 5: Results on Interpret multi-agent conditional scene leaderboard (test set)
minSADE
minSFDE
SMR
SCR
cSMR
ReCoG2 (Mo et al. 2020)
0.33
0.87
14.98
0.09
15.12
Dense TonseTNTia (Gu al. 2021)
0.28
0.89
15.02
0.0
15.02
THOMAS
0.31
0.72
10.67
0.84
11.63
A.6 SPEED PERFORMANCE TRADE-OFF WITH HIERARCHICAL REFINEMENT
We also display the trade-off between inference speed and coverage from hierarchical refinement Fig. / evaluated on the Interpret multi-agent dataset with marginal MissRateg. The curve is
obtained setting the number N of upsampled points the last refinement iteration from 128.
14

---

Published as a conference paper at ICLR 2022
From . 16 and lower, coverage performance starts to diminish while little speed gains made. We still kept relatively high N=64 in our model as we wanted to insure a wide coverage, and the time loss between 41 ms and 46 ms remains acceptable.
kilenetce fimee tmns)
Figure 7: Curve of MissRate, with regard to inference time with varying number of points upsam- pled at the last hicrarchical refinement iteration.
A.7 ADDITIONAL RESULTS ON SINGLE- AGENT DATASETS
Table Argoverse Leaderboard
Chang et al.)
K=1
K=6
minFDE
MR
minADE
minFDE
MR
LaneGCN (Liang et al.
2020)
3.78
59.1
0.87
1.36
16.3
Autobot Girgis et al. 2021)
0.89
1.41
16
TPCN Ye et al. 2021
3.64
58.6
0.85
1.35
15.9
Jean (Mercat et al. (2020)
4.24
68.6
1.00
1.42
13.1
Scene rans (Ngiam al.
202
4.06
59.2
0.80
1.23
12.6
LaneRCNN Zeng et al.
2021
3.69
56.9
0.90
1.45
12.3
PRIME (Song al. 202
3.82
58.7
1.22
1.56
11.5
DenseTTNTT (Gu et al. 2021
3.70
59.9
0.94
1.49
10.5
GOHOME (Gilles et 2021a)
3.65
57.2
0.94
1.45
10.5
HOME (Gilles et al. 2021b)
3.65
57.1
0.93
1.44
9.8
THOMAS
3,59
56.1
0.94
1.44
10.4
15

---

Published as a conference paper at ICLR 2022
Table 7: NuScenes Leaderboard
Caesar al.)
K=5
K=10
k=l
minADE
MR
minADE
MR
minFDE
CoverNet Phan-Minh et al. 2020)
1.96
67
1.48
Trajectron++ (Salzmann al. 2020)
1.88
70
1.51
9.52
ALAN Narayanan et al. 2021
1.87
60
1.22
49
9.98
SG-Net Wang el al. 202
1.86
67
1.40
9.25
WIMP Khandelwal et al.
2020
1.84
55
1.11
43
8.49
MHA-JAM (Messaoud et al.
2020)
1.81
59
1.24
8.57
CXX (Luo et al. 2020)
1.63
69
1.29
60
8,86
LaPred (Kim et al. 2202
1.53
1.12
8.12
P2T (Deo & Trivedi
2020
1.45
64
1.16
46
10.50
GOHOME (Gilles et al. 2021a
1.42
57
1.15
6.99
Autobot (Girgis al. 2021
1.37
62
1.03
44
8.19
PGP (Deo & Trivedi
2020
1.30
57
0.98
37
7.72
THOMAS
1.33
55
1.04
1.04
6.71
16

---

Published as a conference paper at ICLR 2022
A.8 DETAILED ARCHITECTURE
Agont
Input
TrajEncoder
LaneEncoder
Step Grid decoder
grld
UGAL
USRU,
Agent features A4
Lanclet features La
LanesZAgents
GraphEncoderi
Agent features
Concaterate, $6
Artention,
CrophCoon, $4
Uneer, 32
GrophComm
Graph features
CruphCom; 84
Attenioon
Map zware Agant features A
Gragntor,
linea,, RiRgeit
AgentaZAggntss
Artention,
Ego2Lanex
Liner,
Concatenate: 128
Interaction-neirr awlar Agent features x
Agent graphs Ax Lx64
GraphEncoder2
Gophhrow, 37
Hierarchical decoder
Go
Step Grid decoder
GrophCorn, 32
Top N.
scoring
Linear,
Step Grid decader
Sigrold
No
Upsampis
Lanelet scores Ax Lx1
High grid 6,
Step Grid decoder
Final res grid (G, 1)
Spore
Probability heatmap
Figure 8: Detailed illustration of our heatmap generator model
17

---

Published as a conference paper at ICLR 2022
A.9 HEATMAP QUALITATIVE RESULTS
U
Figure 9: Qualitative examples of heatmap output from our multi-agent model. All the heatmaps from one scene are the results from one single forward pass in our model predicting all agents at once. We use matching colors for the agent history, current location and their future predicted heatmap (best viewed color).
18


Reviews:
[{"review_id": "Z3dje7qbhPF", "reviewer": "Reviewer_kDAs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The paper is well written and easy to follow.\n2. Good performance was achieved on the Interaction multi-agent prediction challenge.\n\nWeaknesses:\n1. The method largely follows previous works, with little technical contributions. And even for these limited technical novelties, many of them are not well supported with experimemts.\nIn the end of introduction section, the authors summarize three main contributions:\n\ni) an efficient graph-based model. However, the implementation mainly follows LaneGCN and GOHOME.\n\nii) a collision-aware endpoint sampling. The algorithm mainly follows GOHOME, with a small modification in that when we sample an endpoint for actor a modality k, we not only mask out the same region for other modalities of actor a, but also mask out for other actors of modality k. However, this modification doesn't make sense to me, as we still have a scene-consistent modality recombination module afterwards, therefore the second mask-out won't make much difference here. I also don't see an ablation study for this specific change (ie, do not mask-out for other actors of modality k, which means exactly follows the original algorithm in GOHOME). I want to see this ablation for the last three rows in Table 3.\n\niii) a novel recombination model that could obtain scene-consistent trajectories across the agents. To me this is the core novelty of the paper. From the last two rows in Table 3 we can see that with the proposed recombination module joint metrics improve a lot. However, the marginal metrics degrade a lot (e.g., miss rate more than doubled). The only explanation is that with the proposed module, the same modality (also the sub-optimal one) of an actor is chosen for multiple times in scene-level multi-modalities. To me this may not be what we want: scene-level consistency, but sacrifices actor-level precision a lot. Further analysis is needed here to convince me that this trade-off is worthwhile.\n\n2. Another technical novelty that's not mentioned is the hierarchical iterative refinement of endpoints from the heatmap. However, no ablation study is done on this. I'd like to see a speed-performance comparison with original endpoint estimation model (as in GOHOME).\n\n3. Also, the overall runtime efficiency of the method is not given.\n\n4. GOHOME is the closest approach but I don't see it as a baseline in the experiments.", "summary_of_the_paper": "This paper addresses the task of multi-agent multi-modal trajectory prediction in the context of autonomous driving. Inspired by previous works, the problem is divided into first estimate the goal (end-points), and then re-construct the full trajectory. For end-point estimation, this paper proposes hierarchical iterative refinement from a probability heatmap, with collision-aware greedy sampling to generate collision-free multi-agent trajectories. To further produce scene-consistent multi-agent multi-modal trajectories, the paper also proposes a modality combination ranking module that re-orders the modality of each agent. The method is validated on the Interaction multi-agent prediction challenge and ranks 2nd on the multi-agent track (the 1st-rank entry was submitted 9 days after submission) and 1st on the conditional multi-agent track.", "summary_of_the_review": "Despite good results on the leaderboard, given the limited technical novelty without solid supporting evidences, I do not recommend this paper for acceptance at current status.\n\n**Updated review after rebuttal:**\n\nThanks to the authors for the revision that clarifies the technical novelity much better, and the additional experiments that compare with the GOHOME baseline thoroughly on more benchmarks. These address most of my concerns and therefore I raise my rating.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "VvnHVEaJ3H0", "reviewer": "Reviewer_PgCY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n### Strong Points\n\nThe idea of performing re-estimation based on the computed heat map end-points \nseems novel to me. This idea seems like a natural next step to improve multi agent motion \nprediction. The fact that you first get the end-point and then reason about the scene is \nintuitively sounding and seems to be a good solution for the problem.\n\nThe experiments on interact are a nice direction to be taken by the field.\n\nThe visualizations of the heat-maps and the different predictions was helpful.\n\n### Weak Points\n\n#### I could not find a clear distinction with literature\nWhile reading the related work section I found hard to contrast the ideas of the paper with the ones proposed on the literature.\nThe difference between this method and GOHOME was  not clear to me in this sense. \nFrom what I understood the main contribution here  is the modality combination based on the collision\nfree end-point sampling. However, this was not obvious for me when reading the paper.\n\n\n#### The implemented baselines seemed to have performed way worse than expected.\nFor example, both [1] and [7] are methods designed for *joint* prediction and scene level prediction. This is contradicted \nby the observations made by the authors on page 8, quote: \" ... This problem is aggravated in the joint training case, since the modality selected is\nthe same for all agents in a training sample....\". Both [1] and [3] fairly confidently claim producing multimodal joint predictions.\nEither the version implemented by the authors does not correctly match the other works in the literature or there are specific findings\non the limitations existent in other methods. \nAlso, the paper lack comparison with latent variable based [2][5][[6] multi modal prediction methods. They seem to perform\nbetter when the issue is diverse prediction. Definitely the mode does not tend to be the same for all predictions in those cases.\n\n\n#### A single dataset evalution might not be enough for the community to access the value of this paper.\n\n Motion prediction is currently a very competitive scene. Even though I appreciate the idea on focusing reporting results on multi-agents\nand joint prediction tasks, it is important to also evaluate the prediction on more standard benchmarks such as argoverse. \nWith results on argoverse, me and the rest of the community can more quickly access the the capabilities of the proposed \n heat map prediction pipeline, on producing consistent ego-agent motion prediction. I understand that this would not be the main\nresults, but it would be helpful. \n\n##### Computational cost is apparently not shown.\n \nI could not find the computational cost of running and training the proposed method on the paper. It seems that even\nwith the optimizations made on the heat map estimation the training time and inference time should be high.\nThose kinds of motion prediction models have the necessity to further scale with way more data. Thus, highly\ncomplex methods can be hard to train. There is also the case of potential future execution in an embedded hardware.\n\n### Specific Questions\n\nI am not  sure about the scene modality vectors ( Section 3.2.2), are they like seed parameters which are going to be used\nto incorporate the joint consistent scene level ? I couldn't really find any sort of detailed explanation on that.\n\nWhy the lack of ADE for scene level metrics or using similar metrics as [3] ?\n\n\n[1] Jiquan Ngiam, Benjamin Caine, Vijay Vasudevan, Zhengdong Zhang, Hao-Tien Lewis Chiang,\nJeffrey Ling, Rebecca Roelofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, et al. Scene\ntransformer: A unified multi-task model for behavior prediction and planning. arXiv preprint\narXiv:2106.08417, 2021\n[2] Tang, Charlie, and Russ R. Salakhutdinov. \"Multiple futures prediction.\" Advances in Neural Information Processing Systems 32 (2019): 15424-15434.\n[3] Li, Lingyun Luke, et al. \"End-to-end contextual perception and prediction with interaction transformer.\" 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020.\n[4] Yuan, Ye, et al. \"AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting.\" arXiv preprint arXiv:2103.14023 (2021).\n[5] Girgis, Roger, et al. \"Autobots: Latent Variable Sequential Set Transformers.\" arXiv preprint arXiv:2104.00563 (2021).\n[6] Casas, Sergio, et al. \"Implicit latent variable model for scene-consistent motion forecasting.\" Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, \n[7] Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, and Raquel Urtasun. Learning\nlane graph representations for motion forecasting. In ECCV, 2020\n", "summary_of_the_paper": "The paper proposes a multi-modal trajectory prediction pipeline.\nThe idea is that based on past trajectories and a map of the road a model\nis trained to predict a heat map with the most likely end-points.\nWith those end-points a trajectory can be inferred for each agent in a collision\nfree way by leveraging all heat  maps jointly.\nWith those trajectories, multiple, consistent scene level trajectories are predicted.\n\nThe main contribution of the paper is on the reasoning about the end-points\ngiven by the predicted heat map. ", "summary_of_the_review": "\nThe paper provides an incremental solution to heatmap based motion\npredictions that improves on scene level predictions.\nEven though this idea is interesting, the experiments were not able\nto convince me as this being an impactful approach for scene level prediction.\n\n# Post rebuttal review \n\nAfter carefully reading the authors rebuttal, I decided to increase my score.\nThe changes proposed by the authors and the explanations provided would increase the quality of the paper.\nThe two main problems: The fact that the contributions in comparison with GOHOME were not clear,\nand the lack of results in comparable benchmark were both carefully addressed.\n\nI am changing the score to a 6 provided that the authors proofread the paper as well recommended\nby reviewer 4LEw, add the other benchmark results and better compare with GOHOME.\n\nThe paper does not qualify as of containing major new contributions. However, I think\nthe incremental contributions shown are sufficient for me to lean towards acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "L-6W6TnWowE", "reviewer": "Reviewer_4LEw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The overall idea of iterative hierarchical heatmap refinement is interesting and appears to be novel. The experiments demonstrate the ideas validity, although there is definitely room for further strengthening.\n\n- The related work is thorough, one quick comment here is to also reference the following work in the discussion of methods which can predict multiple agents at the same time (beginning of the last paragraph before Section 3): \"MATS: An Interpretable Trajectory Forecasting Representation for Planning and Control\" by B. Ivanovic, A. Elhafsi, G. Rosman, A. Gaidon, M. Pavone in CoRL 2020.\n\n- Acronyms should be explained when they are first used. For instance, it might not be obvious what a UGRU is in Section 3.1.1. What does MR optimization mean in Section 3.1.3?\n\n- Do not use contractions in scientific writing (e.g., change \"don't\" to \"do not\").\n\n- A thorough round of editing is necessary, there seem to be writing errors that materially change the meaning of sentences. For instance, \"The final output is therefore no strictly a re-ordering...\" Should this have said \"not strictly\"? Same question for \"which is JointMR where colliding modalities are also counted as misses even if they are closer than the defined threshold.\" Should this have just said \"where colliding modalities are also counted as misses?\" It is not very clear what these metrics mean here.\n\n- The novelty with respect to the original GOHOME work needs to be made clearer, there seem to be many reused components and it is difficult to tell which parts are new in this work.\n\n- Related to the above point, it is a bit difficult to understand this work's core contributions in Sections 3.2.1 and 3.2.2, also because the writing makes this difficult (e.g., a spurious use of \"However\" makes it difficult to determine if a statement is being negated or not).\n\n- There are some key insights missing from the experimental section, which will be expanded on below.\n\n- Ultimately, the method is only evaluated on one dataset, and it is difficult to determine if its performance would generalize to other datasets (for instance, what about nuScenes, Lyft, Waymo Open, Argoverse, etc?).\n", "summary_of_the_paper": "This work proposes THOMAS, a hierarchical heatmap refinement scheme for multi-agent trajectory forecasting. At its core lies a GOHOME-style architecture with a few modifications (predicting only endpoints, iteraetively improving the heatmap output, collision-free output sampling, modality combination re-ranking) that yield scene-consistent predictions, which are demonstrated to yield better performance across a wide-variety of metrics on the Interaction dataset.", "summary_of_the_review": "While the overall idea of iterative hierarchical heatmap refinement is interesting, there are some significant areas of improvement that make it difficult to argue for the acceptance of this work as-is. For instance:\n- Writing errors in key areas make it difficult to parse what is going on internally in the model.\n- The novelty with respect to the original GOHOME model should be expanded upon, perhaps with a concrete sentence stating the core differences near the beginning of Section 3 and stating the relationship of this work to GOHOME in the related work.\n- The experiments are good at showing that the model works, but there are a bit of key insights missing. For example, why are only two rounds of heatmap upsampling done? What do the performance curves look like with increased points? What does the Pareto front look like with respect to computation time or required FLOPs (at least to indicate to a reader why stopping at two levels makes sense)?\n- Finally, another dataset would really strengthen this work and aid in convincing readers that the method's performance extends beyond the Interaction dataset.\n\nAddressing these would certainly ready this paper for publication in this or a similar venue.\n\n### Post-Rebuttal\n\nThe added results and comments in the author rebuttal address most of my concerns. I share the same overall sentiment as Reviewer PgCY, the paper may not have a single \"major\" novelty, but the set of presented incremental contributions are sufficient to convince me of the performance and utility of this work over prior approaches, and I believe this paper is now more suitable for publication.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Fog7GD7h7_e", "reviewer": "Reviewer_ZJY1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths:\n- __Originality__: the two main contributions outlined in the summary of the paper are fairly distinctive from previous works.\n  - The hierarchical heatmap prediction tackles one of the main issues of heatmap representations, which is their heavy memory consumption.\n  - The learnable recombination method is general enough to be suitable for any marginal prediction model, which makes me think it would have a solid impact on the community.\n- Introduction and related work are succinct and touch upon the main relevant points to the future prediction task.\n- Figures are useful to understand the method as well as to evaluate the performance qualitatively.\n\n### Weaknesses:\n- __Writing ambiguity and lack of details__: I am not sure the write-up is polished enough for submission to ICLR. I outline a few reasons below:\n  - Throughout the paper, there is no mention or description of the loss functions utilized, and barely any mention of the overall training process (with the exception of a sentence in 3.1.3). In the appendix, the authors describe how the heatmap output is optimized, but there is no mention about how the learnable recombination module is trained, which is one of the main contributions of the paper.\n  - Because of this lack of description of the overall training process and loss functions, it is unclear if in the experiments the authors utilize 2 separate THOMAS models - one for marginal prediction and one for joint prediction. For instance, are the marginal metrics and joint metrics in Table 1 coming from a single model, or two separately trained models? A sentence that contributes to this confusion is in 3.1.3, where the authors mention that they decode K end points using the same MR optimization algorithm as Gilles et al. (2021b). However, in 3.2.1 the authors enhance this algorithm with collision-free sampling. Please clarify this in the rebuttal.\n  - The paper criticizes generative models in the second paragraph of the related work for not providing a probability value for each prediction, which makes me expect the method to provide such scores. However, there is no mention about it in the approach or experiments sections.\n  - In Table 3, the Marg and Joint objectives are not defined.\n- It is unclear how some pieces tie together. My main concern is with respect to collision-free end point sampling (3.2.1) and recombination (3.2.2). The heuristic in 3.2.1 seems to assume that the modalities are \"already ordered\" since the algorithm basically avoids sampling close end points across actors for a given modality k. However, the recombination can perform arbitrary linear combinations across modes, thus not respecting this. Is the idea that only one of the two methods are used to perform inference/training on a model? If so, I would suggest moving 3.2.1 to the experiments as a baseline/ablation, and I definitely would not claim it as a contribution (particularly since it's just a small tweak on top of a previously proposed algorithm).\n- Experiments are not as thorough as I would expect. This is understandable for the joint prediction setting as there are very few baselines and the dataset/benchmarks on this task have just started. In fact, I appreciate the effort of the authors of re-implementing ILVM and SceneTransformer. However, with respect to marginal prediction, there are a lot of methods and benchmarks out there. Another aspect that would have been great to get more insight on is the hierarchical heatmap output. What's the tradeoff between computation (both runtime and memory) and accuracy, with respect to dense heatmap approaches? How was the current 3-layer hierarchy chosen?\n\n### Additional feedback:\nIn this section I provide additional feedback for rebuttal, but please address first the main points above.\n- Re: training details in the appendix. I find it surprising that the method is trained with random sampling of up to 8 agents, given that for joint prediction there might be other agents that are particularly relevant to the task. Was this better than lowering the batch size? If so, is it because it is rare that a scene contains > 8 actors in the INTERACTION dataset?\n- The equation that describes the weighted linear combination of agent modalities seems incomplete. As far as I understand, there should be a weighted sum in the equation, using the scores coming out of the softmax.\n- Related work is missing some relevant works. In the last paragraph, joint autoregressive methods should be mentioned (e.g., PRECOG[A] - see references at the bottom) as they were the first to model the interactions across agents at the output level. With respect to generative models and their random sampling and lack of confidence scores per modality, LookOut[B] is a recent method that addresses both these concerns with a learnable diverse sampler and scene scoring. The authors can treat this as concurrent work since it was just published at ICCV, but please include it in the discussion as it is highly related.\n- Is the notation of the joint metrics coming from the Interpret challenge? If not, I would suggest unifying this with the names proposed originally in ILVM (e.g., minJointFDE --> minSFDE, CrossCollisionRate --> SCR).\n- While the sparse and hierarchical heatmaps make sense to mitigate their high resource consumption, I am a bit concerned about the fact that endpoints are only sampled from the finest resolution, as this might not contain more unlikely but safety-critical modalities. Overall, the heuristic refinement of the most likely grid cells seems contradictory with the endpoint sampling objective of maximizing coverage.\n- The writing would benefit from more intuitions. For instance, the method samples heuristically from the heatmaps to maximize coverage, but why is this important to the overall self-driving task?\n- I personally did not get much out of figure 4, perhaps adding the shapes of the tensors could be helpful.\n- I would avoid calling the combination method an \"ordering\", as the method is more general than that.\n- How come the CrossCollisionRate is still 2.6% when using the collision-free end point sampling? I would expect it to be zero.\n\n### Additional references:\n- [A] Rhinehart, Nicholas, et al. \"Precog: Prediction conditioned on goals in visual multi-agent settings.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.\n- [B] Cui, Alexander, et al. \"LookOut: Diverse multi-future prediction and planning for self-driving.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.", "summary_of_the_paper": "The paper proposes a joint trajectory prediction framework. This means that the model produces multi-modal future predictions at the scene level (i.e., for all actors jointly). There has only been very recent research in this area despite its importance to autonomy and motion planning in particular, so the relevance of the paper to the broader motion forecasting field is high. I identify two main contributions in the paper: (i) hierarchical and sparse heatmap output representation and (ii) a recombination method that is pluggable in any multi-modal marginal prediction model.", "summary_of_the_review": "I think this paper has significant contributions in terms of the presented method. However, my main concerns are related to the lack of details in the writing as well as the thoroughness of the experiments and the insights that can be extracted from them. For these reasons, I am leaning towards rejection, but I look forward to the authors' response during the rebuttal period.\n\n### Post-rebuttal\n\nAfter careful consideration of the rebuttal, I am happy to increase my score and I lean towards acceptance. My doubts were clarified, and the authors have timely adapted the write-up. Since my main issue was with the lack of clarity in some parts and that has been improved I raise my score to a 6. I am in between 6 and 8, but I think 8 is a bit too much since the improvements are incremental. I would rate it as a 7 if I had the option given that the proposed linear recombination of marginal predictions is very flexible and could be easily adapted to multiple prior works in the literature, thus raising the future impact of this work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a joint multi-agent trajectory prediction framework for multiple agents using a "heatmap" estimation approach employing a hierarchical strategy and sparse image generation for for efficient inference. The method takes a set of predicted trajectories for each agent produces reorderings. The work yields a top result on a competitive leaderboard. 

While multiple reviewers were initially concerned about the paper not making a single major contribution, the author response discussion helped to clear up the degree of novelty. Further experiments provided during the review also led to multiple reviewers increasing their score. In the end, all reviewers recommend acceptance of this paper.

As such the AC recommends accepting this paper.
Decision: Accept (Poster)