Scene Transformer: A unified architecture for predicting future trajectories of multiple agents
Authors: Jiquan Ngiam
Vijay Vasudevan
Benjamin Caine
Zhengdong Zhang
Hao-Tien Lewis Chiang
Jeffrey Ling
Rebecca Roelofs
Alex Bewley
Chenxi Liu
Ashish Venugopal
David J Weiss
Benjamin Sapp
Zhifeng Chen
Jonathon Shlens
Venue: ICLR 2022 Poster (2022)
Keywords: trajectory prediction
motion forecasting
multi-task learning
attention
autonomous vehicles

Abstract: Predicting the motion of multiple agents is necessary for planning in dynamic environments. This task is challenging for autonomous driving since agents (e.g., vehicles and pedestrians) and their associated behaviors may be diverse and influence one another. Most prior work have focused on predicting independent futures for each agent based on all past motion, and planning against these independent predictions. However, planning against independent predictions can make it challenging to represent the future interaction possibilities between different agents, leading to sub-optimal planning. In this work, we formulate a model for predicting the behavior of all agents jointly, producing consistent futures that account for interactions between agents. Inspired by recent language modeling approaches, we use a masking strategy as the query to our model, enabling one to invoke a single model to predict agent behavior in many ways, such as potentially conditioned on the goal or full future trajectory of the autonomous vehicle or the behavior of other agents in the environment. Our model architecture employs attention to combine features across road elements, agent interactions, and time steps. We evaluate our approach on autonomous driving datasets for both marginal and joint motion prediction, and achieve state of the art performance across two popular datasets. Through combining a scene-centric approach, agent permutation equivariant model, and a sequence masking strategy, we show that our model can unify a variety of motion prediction tasks from joint motion predictions to conditioned prediction.

Published as a conference paper at ICLR 2022
SCENE TRANSFORMER: A UNIFIED ARCHITECTURE FOR PREDICTING MULTIPLE AGENT TRAJECTORIES
Jiquan Ngiam"; Benjamin Caine*: Vijay Vasudevan".! Zhengdong Zhang', Hao-Tien Lewis Chiang". Jeffrey Ling". Rebecca Roelofs', Alex Bewley', Chenxi Liu?, Ashish Venugopal", David Weiss?, Ben Sapp". Zhifeng Chen' Jonathon Shlens !Google Research, Brain Team, 2 Waymo (ingiam, bencaine, vrv) @google.com
ABSTRACT
Predicting the motion of multiple agents necessary for planning in dynamic environments. This task is challenging for autonomous driving since agents (e.g., vehicles and pedestrians) and their associated behaviors may diverse and influ- ence one another. Most prior work have focused on predicting independent futures for each agent based on all past motion, and planning against these independent predictions. However, planning against independent predictions make it chal- lenging to represent the future interaction possibilities between different agents, leading sub-optimal planning. In this work, we formulate model for predicting the behavior of all agents jointly, producing consistent futures that account for interactions between agents. Inspired by recent language modeling approaches, we use a masking strategy as the query to our model, enabling one to invoke single model to predict agent behavior in many ways, such as potentially conditioned on the goal or full future trajectory of the autonomous vehicle or the behavior of other agents in the environment. Our model architecture employs attention combine features across road elements, agent interactions, and time steps. We evaluate our approach on autonomous driving datasets for both marginal and joint motion prediction, and achieve state of the art performance across two popular datasets. Through combining a scene-centric approach, agent permutation equivariant model, and sequence masking strategy, show that our model can unify variety of motion prediction tasks from joint motion predictions to conditioned prediction.
INTRODUCTION
Motion planning in dense real-world urban environment is mission-critical problem for deploying autonomous driving technology. Autonomous driving is traditionally considered too difficult for
single end-to-end learned system (Thrun al. 2006). Thus, researchers have opted to split the task into sequential sub-tasks (Zeng et al.) 2019): (i) perception, (ii) motion prediction, and (ii) planning. Perception is the task of detecting and tracking objects in the scene from sensors such as LiDARS and cameras. Motion prediction involves predicting the futures actions of other agents in the scene. Finally, planning involves creating motion plan that navigates through dynamic environments.
Dividing the larger problem into sub-tasks achieves optimal performance when each sub-task is truly independent. However, such strategy breaks down when the assumption of independence does not hold. For instance, the sub-tasks of motion prediction and planning are not truly independent- the autonomous vehicle's actions may significantly impact the behaviors of other agents. Similarly, the behaviors of other agents may dramatically change what is good plan. The goal of this work is to take a step in the direction of unifying motion prediction and planning by developing a model that can exploit varying forms of conditioning information, such as the AV's goal, and produce joint consistent predictions about the future for all agents simultaneously.
While the motion prediction task has traditionally been formulated around per-agent independent predictions, recent datasets (Ettinger et al.) 2021 Zhan et al.) 2019) have introduced interaction prediction tasks that enable us study joint future prediction (Figure These interaction prediction

---

Published as a conference paper at ICLR 2022
marginal prediction
joint prediction
blue inspectory m
orange trajectory
blue scene 0.3
Figure Joint prediction provides consistent motion prediction. Illustration of differences between marginal and joint motion prediction. Each color represents distinet prediction. Left: Marginal prediction for bottom center vehicle. Scores indicate likelihood of trajectory. Note that the prediction is independent of other vehicle trajectories. Right: Joint prediction for three vehicles of interest. Scores indicate likelihood of entire scene consisting of trajectories of all three vehicles.
tasks require models to predict the joint futures of multiple agents: models are expected to produce future predictions for all agents such that the agents futures are consistent with one another.
A naive approach producing joint futures is consider the exponential number of combinations of marginal agent predictions. Many of the combinations are not consistent, especially when agents have overlapping trajectories. We present a unified model that naturally captures the interactions between agents, and can be trained as a joint model to produce scene-level consistent predictions across all agents (Figure T right). Our model uses a ceene-centric representation for all agents (Lee) et al. 2017 Hong et al. 2019;
2019) Casas et al. 2020aa Salzmann et al., 2020) to allow scaling to large numbers of agents in dense environments. We employ a simple variant of self-attention (Vaswani al. 2 in which the attention mechanism is efficiently factorized across the agent-time axes. The resulting architecture simply alternates attention between dimensions representing time and agents uniform, and scalable architecture.
across the scene, resulting in computationalll--fficee
We find that the resulting model, termed Scene Transformer, achieves superior performance on both independent (marginal) and interactive (joint) prediction benchmarks.
Moreover, we demonstrate a novel formulation of the task using a masked sequence model, inspired by recent advances in language modeling (Brown et al. 2020 Devlin et al. (2019), to allow conditioning of the model on the autonomous vehicle (AV) goal state or full trajectory. In this reformulation, a single model can naturally perform tasks such motion prediction, conditional motion prediction, and goal-conditioned prediction simply by changing which data is visible at inference time.
We hope that unified architecture and flexible problem formulation opens up research directions for further combining motion prediction and planning. In summary, our key contributions in this work are:
novel, scene- cenerccnntic approach that allows us gracefully switch training the model produce either marginal (independent) and joint agent predictions in single feed-forwan pass. Our model achieves state-of-the-art on both marginal and joint prediction tasks on both the Argoverse and the Waymo Open Motion Dataset.
A permutation equivariant Transformer- based architecture factored over agents, time, and road graph elements that exploits the inherent symmetries of the problem. The resulting architecture is efficient and integrates the world state in unified way.
A masked sequence modeling approach that enables us to condition on hypothetical agent fu- tures at inference time, enabling conditional motion prediction goal conditioned prediction.
RELATED WORK
Motion prediction architectures. Motion prediction models have flourished in recent years, due to the rise in interest in self-driving applications and the release of related datasets and bench-
Marginal agent predictions may conflict with each other (have overlaps), while consistent joint predictions should have predictions where agents respect each other's behaviors (avoid overlaps) within the same future.
2

---

Published as a conference paper at ICLR 2022
marks (Kesten et al., 2019: Chang et al. 2019 Caesar al., 2020: Ettinger et al. 2021 Successful models must take into account the history of agent motion, and the elements of the road graph (e.g.. lanes, stop lines, traffic light dynamic state). Furthermore, such models must learn the relationships between these agents in the context of the road graph environment.
One class of models draws heavily upon the computer vision literature, rendering inputs multi- et al. 2017, Hong et al.) 2019 Casas et al., 2020aa Salzmann al. [220 Zhao et al. 2019). In this approach, relationships between scene elements are captured via convolutional deep architectures. However, the localized structure of the receptive field makes capturing spatially-
channel rasterized top-down image (Cui et al. 2019 Chai et
al.] |2199]
interactions challenging. A popular alternative is to use an entity-centric approach. With this approach, agent state history is typically encoded via sequence modeling techniques like RNNs (Mercat et al.) 2020 Khandelwal et al. 2020 Lee et al. 2017; [Aaah et al.] 20166] Rhinehart et al. (019) or temporal convolutions (Liang et al. 2020). Road elements are approximated with basic primitives (e.g. piecewise- segments) which encode pose information and semantic type. Modeling relationships between entities is often
presented as an information aggregation process, and models employ pooling (Zhao et al. 2020 Gao
et al. 2020
Lee et al. 2017 Alahi et al. 2016; Gupta et al., 2018). soft-attention (Mercat et al. 2020
Zhao et al.
et al. 2020) as well as graph neural networks (Casas et al. 2020a
Liang
al. 2020
Khandelwal et al. 2020).
Like our proposed method, several recent models use Transformers (Vaswani et al. 2017). composed of multi-headed attention layers. Transformers
a popular state-of-the-an choice for sequence
modeling in natural language processing (Brown et al. 2020 Devlin et al. (211)), and have recently
shown promise in core computer vision tasks such as detection (Bello et al. 2019 Carion et al.. 2020
Srinivas et al. (02T), tracking (Hung et al. 2020) and classification (Ramachandran et al. 2019
Vaswani et al. 2021; Dosovitskiy et al.) [021] Bello [013) Bello et al., 2019). For motion modeling.
recent work has employed variations of self afff-attetioon and Transtormers for modeling different axes:
temporal trajectory encoding and decoding (Yu et al. 2020 Giuliari et al. 2020; Yuan et al. 2021). encoding relationships between agents (Li et al. 2020 Park et al.) 2020; Yuan et al. 202 Yu et al. 2020 Mercat et al., 2020; Bhat et al., 2020), and encoding relationships with road elements. When applying self-attention over multiple axes, past work used independent self-attention for each axis (Yu ct al. 2020), or flattened two axes together into one joint self-attention layer Yuuan et al., (021) by comparison, our method proposes axis axis- factore attention model relationships between time steps, agents, and road graph elements in a unified way.
Scene-centric versus agent-centric representations. Another key design choice is the frame of reference in which the representation is encoded. Some models do a majority of modeling in a global, scene-level coordinate frame, such work that employs rasterized top-down image (Cuil
iet al. 2019 Chai et al. [0199] Lee et al. 2017 Hong et al. 2019 Casas et al. 2020a) Salzmann
al. 2020). This can lead to a more efficient model due to a single shared representation of world
state in common coordinate frame, but comes with the potential sacrifice of
On
the other hand, models that reason in the agent-coordinate frame (Mercat et al. 2020 Zhao et al., 2020: Khandelwal et al. 2020) are intrinsically pose-invariant, but scale linearly with the number of agents, or quadratically with the number of pairwise interactions between agents. Many works employ a mix of top-down raster representation for road representation fused with per-agent representations (Rhinchart et al.) 2019 Tang & Salakhutdinov 20199] Lee et al., (017)). Similar to our own work, LaneGCN (Liang et al. (020) is agent agent-centric yet representations are in global frame- to the best of our knowledge, this is the only other work to do so. This enables efficient reasoning while capturing arbitrarily distant interactions and high-fidelity state representations without rasterization.
Representing multi-agent futures. A common way to represent agent futures is via a weighted set
of trajectories per agent (Alahi et al. 2016
Biktairov al. 2020 Buhet et al.) 22200 Casas et al.
2020a a| Chai et al.] 2019|[Cul et al. 2019: Goo et al.] 2020 Hong et [019 Lee et al. 2017]]
Marchetti et al. 2020 Mercat et al. [220]] Salzmann et al.) 20200] Zhho et al. [020 Chandra et al.
2018). This representation is encouraged by benchmarks which primarily focus on per-agent distance
error metrics (Caesar al. 2020, Chang et al./ 2019 Zhan et al. 2019). We argue in this work that modeling joint jooit futures in a multi-agent environment (Figure I right) is an important concept that has been minimally explored in prior work. Some prior work consider a factorized pairwise joint distribution, where subset of agent futures are conditioned on other agents- informally, modeling P(X) and P(Y[X) for agents X and Y (Khandelwal et al. 2020 Tolstaya et al. 2021 Salzmann et al.
2020) To generalize joint prediction to arbitrary multi-agent settings, other work (Tang
3

---

Published as a conference paper at ICLR 2022
Time
Tertane
Agent Pestues p-1 -1.
den testerr
Agents
Adc ert 2g tirme diventoe
Soan ataenti Ligent neadgrahh
A:1. Jonns
Road
den time:
Do jorm agrins
Calarta den ue agarts)
Motion prediction (MP)
Concitional motion preciction (CMP)
Geallcondiiioned motion prediction (GCP)
Cellatarnenn time)
Dren Ciper smentresh
-Viiible to model
Hidden and be predicted by model
Aguni Proteers
Pockem ZA1S
Figure 2: Single model architecture for multiple motion prediction tasks. Left: Different masking strategies define distinct tasks. The left column represents current time and the top row represents the agent indicating the autonomous vehicle (AV). A single model can be trained for data associated with motion prediction, conditional motion prediction, and goal-directed prediction, by matching the masking strategy each prediction task. Right: iitentio-bbssed encoder-decoder architecture for joint scene modeling. Architecture employs factored attention along the time and agent axes to exploit the dependencies in the data, and cross-attention inject side information.
& Salakhutdinov 2019 Rhinehart et al. 2019 Casas et al., 2020b Suo et al. 2021 Yeh et al.
2019) iteratively roll out samples per-agent, where each agent is conditioned on previously sampled trajectory steps. In contrast, our model directly decodes a set of distinct joint futures with associated likelihoods.
METHODS
The Scene Transformer model has three stages: (i) Embed the agents and the road graph into a high dimensional space, (ii) Employ an attention-ben based network to encode the interactions between agents and the road graph, (iii) Decode multiple futures using an attention-based network. The model takes as input feature for every agent at every time step, and also predicts an output for every agent at every time step. We employ an associated mask, where every agent time step has an associated indicator of (hidden) 0 (visible), indicating whether the input feature is hidden (i.e. removed) from the model. This approach mirrors the approach of masked-language models such as BERT
simultaneously train a single model for motion prediction (MP) (Cui et al. [019 Chai et al. 2019 Lee et al. 2017 Hong et al. 2019 Casas
(Devlin et al. 2019). The approach is flexible, enabling
al. 2020a; Salzmann al.] [2020] Casas al. [020aa] Liang et al. 2020 Khandelwal et al. 2020),
conditional motion prediction (CMP) (Khandel wal et al.
2020 Tolstaya et al. [202 Salzmann et al.
2020) and noalcconditioned prediction (GCP) (Deo & Trivedi 2020) simply by changing what data
is shown to the model (Figure [ left). We summarize the key contributions below, and reserve details
for the Appendix.
Multi-task representation. The key representation in the model is 3-dimensional tensor of A agents with D feature dimensions across T time steps. At every layer within the architecture, we aim to maintain a representation of shape [A, T. D). or when decoding, [F, A,T, D] across F potential futures. Each task (MP, CMP, GCP) can formulated as query with a specific masking strategy by setting the indicator mask to O, thus providing that data to the model (Figure 2 left). The goal of the model is to impute the features for each shaded region corresponding to subsets of time and agents in the scenario that are masked.
3.1 SCENE-CENTRIC REPRESENTATION FOR AGENTS AND ROAD GRAPHS.
We use a scene-centric embedding where we use an agent of interest's position as the origin and encode all roadgraph and agents with respect to it. This contrast to approaches which use an agent-centric representation, where the representations are computed separately for each agent, treating each agent in turn as the origin.
For WOMD, we center the scene with respect to the autonomous vehicle (AV). For Argoverse, we center the scene with respect to the agent that needs to be predicted. Both are centered around what would be the last visible time step in motion prediction setup for all tasks.
4

---

Published as a conference paper at ICLR 2022
In detail, we first generate feature for every agent time step if that time step is visible. Second, generate a set of features for the static road graph. road elements static in space and time, learning one feature vector per polyline (with signs being polylines of length 1) using PointNet (Qi et al. 2017). Last, we generate set of features for the dynamic road graph, which are road elements static in space but dynamic in time (e.g. traffic lights), also one feature vector per object. All three categories have zyz position information, which we preprocess to center and rotate around the agent of interest and then encode with sinusoidal position embeddings (Vaswani et al. 2017
3.2 ENCODING TRANSFORMER
We focus simple encoder-decoder attention- based architecture which maintains representation of [A, T. D] throughout (Figure 2 right). We summarize the architecture briefly, but reserve details for the Appendix and Table The majority of layers are form of the Transformer layer (Vaswani et al. 2017) (Table Attention layers are parameterized as matrices representing the query Q, key
K, and value V, whose output y softmax (Q Each matrix is computed as a learned linear transformation of the underlying representation xr e.g. Q W,, x. Each attention layer is followed by a ferd-forward layer of the same hidden dimension, and a skip connection addition of the result with the input to the whole Transformer layer. All layers of the encoder and decoder employ a D feature dimension. The final layer after the decoder is a 2-layer MLP that predicts 7 outputs. The first 6 outputs correspond the 3-dimensional position of an agent a given time step in absolute coordinates (e.g. meters) with respect to the agent of interest, and the corresponding uncertainty parameterized by a Laplace distribution (Meyer & Thakurdesai 2020). The remaining dimension predicts the heading.
Efficient factorized self-attention. The bulk of the computation is performed with Transformer (Vaswani et al. (017)) 6.) One naive approach to use the Transformer would be to perform attention directly on the entire set of agent and time step features (i.e., attention across AT dimensions). However, this approach is computationally expensive, and also suffers from an identity symmetry challenge: since we do not add any specific agent identity indicator, two agents of the same type with the same masked future time-step will have the same input representation to the transformer, resulting in the same output. Thus, we design a factorized attention based on the time and agents axes (for related ideas, see Wang al. (2020): Szegedy al. (2016): Ho et al. (019)))
Applying attention only across time allows the model to learn smooth trajectories independent of the identity of the agent. Likewise, applying attention only across agents allows the model to learn multi-agent interactions independent of the specific time step. Finally, in order to capture both time and agent dependencies, the model simply alternates attention across agents and time in subsequent layers (Figure [ right panel). The model is also permutation equivariant to the ordering of the agents at input, since the attention operation is permutation equivariant.
Cross-attention. In order to exploit side information, which in our case is road graph, we use cross-attention to enable the agent features to he updated by attending to the road graph. Concretely, we calculate the queries from the agents, but the keys and values come from the embeddings of the road graph. The road graph embeddings are final after the per-polyy per-polyline PointNet, and therefore not updated during these attention layers. This requires that the model learn interactions between the road structure and agents that are independent of the specific time step or agent. We highlight that the and shared across all agents in the scene,
road graph representation is also crrmmuattionequivarttt whereas prior approaches have often used a per-agent road graph representation.
3.3 PREDICTING PROBABILITIES FOR FACH FUTURES.
Our model also needs to predict a probability score for each future (in the joint model) or trajectory (in the marginal model). In order to do we need a feature representation that summarizes the scene and each agent. After the first set of factorized self-attention layers, we compute the mean of the agent features tensor across the agent and time dimension separately, and add these an additional artificial agent and time making our internal representation [A+1,T 1, D] (Figure 2 left panel). This artificial agent and time step propagates through the network, and provides the model with extra capacity for representing each agent, that is not tied to any timestep. At the final layer, we slice out the artificial agent and time step to obtain summary features for each agent (the additional time per
5

---

Published as a conference paper at ICLR 2022
agent), and for the scene (the "corner' feature that is both additional time and agent). This feature is then processed by a 2-layer MLP producing a single logit value that we use with a softmax classifier for permutation equivariant estimate of probabilities for each futures.
3.4 JOINT AND MARGINAL LOSS FORMULATION.
The output of our model is tensor of shape [F, A, T, 7] representing the location and heading of each agent at the given time step. Because the model uses scene-centric representation for the locations through positional embeddings, the model is able to predict all agents simultancously in single feed-forward pass. This design also makes it possible to have straight- forward switch between joint future predictions and marginal future predictions.
To perform joint future prediction, we treat each future (in the first dimension) to be coherent futures across all agents. Thus, we aggregate the displacement loss across all agents and time steps to build a loss tensor of shape [P]. We only propaaate the loss through the individual future that most closely matches the ground-truth in terms of displacement loss (Gupta et al.) 2018 Yeh et al. (019)). For marginal future predictions, each agent is treated independently. After computing the displacement loss of shape [F, A], we do not aggregate across agents. Instead, we select the future with minimum loss for each agent separately, and back-propagate the error correspondingly (Appendix, Figure 8)
Evaluation metrics for motion prediction. We evaluate the quality of & weighted trajectory hy- potheses using the standard evaluation metrics: minADE, minFDE, miss rate, and mAP. Each evaluation metric attempts to measure how close the top h trajectories are to ground truth observation. simple and common distance- metric to measure the L.2 norm between a given trajectory and the ground truth (Alahi et al.) 2016 Pellegrini et al.) 2009). minADE reports the L22 norm of the trajectory with the minimal distance. minFDE likewise reports the Le norm of the trajectory with the smallest distance only evaluated at the final location of the trajectory. We additionally report the miss rate (MR) and mean average precision (mAP) to capture how well a model predicts all of the future trajectories of agents probabilistically (Yeh et al., 2019 Chang et al. (2099] [Etinnerr al. (021). For joint future evaluation settings, we measure the scene-level equivalents (minSADE, minSFDE, and SMR) that evaluate the prediction of the best single consistent future (Casas et al. 2020b).
4 RESULTS
We evaluate the Scene Transformer on motion prediction tasks from the Argoverse dataset (Chang
al. 2019) and Waymo Open Motion Dataset (WOMD) (Ettinger et al.) 2021). The Argoverse dataset consists of 324,000 run segments (each seconds in length) from 290 km of roadway containing 1 11.7 million agent tracks in total, and focuses single agent (marginal) motion prediction. The WOMD dataset consists of 104,000 run segments (each 20 seconds in length) from 1,750 km of roadway containing 7.64 million unique agent tracks. Importantly, the WOMD has two tasks, each with their own set of evaluation metrics: a marginal motion prediction challenge that evaluates the quality of the motion predictions independently for each agent (up to 8 per scene), and a joint motion prediction challenge that evaluates the quality of model's 's joint predictions of exactly 2 agents per scene. We train each model on Cloud TPU (Jouppi et al. 2017) See Appendix for all training details.
First, in Section 4.1 we focus on the marginal prediction task and show that Scene Transformer achieves competitive results on both Argoverse (Chang et al. 2019) and the WOMD (Futinger et al.. 2021). In Section 4.2, we focus on the joint prediction task and train the Scene Transformer with our joint loss formulation (see Section (.44) We show that with a single switch of the loss formula, we can achieve superior joint motion prediction performance. In Section 4.3 we discuss factorizedd versus non-fac con-factorized attention. Finally, in Section 4.4 we show how our masked sequence model formulation allows to train multi-task model capable of motion prediction, conditional motion prediction, and coal-conditioned prediction. In Appendix B.I we discuss the trade-off between marginal and joint models.
Fach dataset identifies subset of agents to be predicted. We only include this subset in our loss calculation. For Argoverse, this 1 agent; for WOMD this is 2-8 agents.

---

Published as a conference paper at ICLR 2022
Method
minFDE minADE
MR
Jean (Mercat al. 2020
0.97
1.42
0.13
WIMP Khandelwal TNT Zhao et al. 2020)
al.
2020
0.90
1.42
0.17
0.94
1.54
0.13
LaneGCN (Liang cf al. 2020
0.87
1.36
0.16
TPCN Ye el al. 202T
0.85
1.35
0.16
mm Trarnnsformer LLu
2021b)
0.84
1.34
0.15
HOME (Gilles et al. Ours (marginal
2021
0.94
0.80
1,23
0.13
Table I: Marginal predictive performance on Argoverse motion prediction. Results reported on
test split for vehicles (Chang al. 2019). minADE, minFDE reported for predictions (Alahi al. 2016 Pellegrini et al. 2009;; Miss Rate (MR) (Chang et al. 2019) within 2 meters of the target. All results are reported for seconds.
Motion Prediction
minADE
minFDE
MR J
mAP
veh
ped
veh
ped
eyc
ped
eye
veh
oyc
valid
baseline (Etinger Ours (margiaa))
1.35
0.20
1.17
0.59
1.15
2.51
1.26
2.44
0.20
0.12
0.24
0.26
0.27
0.20
test
1.STM haseline (Fatinger al.
202
1.34
0.64
1.29
2.83
1.35
2.68
0.24
0.13
0.29
0.24
0.22
0.19
ReCoAt (Hanng et al X02
1.69
0.69
1.47
3.96
1.51
3.30
0.40
0.20
0.37
0.18
0.25
0.17
CrmpleCNNOnkissee
2021
1.47
0.71
1.39
3.18
1.52
2.89
0.16
0.31
0.19
0.14
TNT er 2021
1.35
0.85
2.17
3.35
1.40
2.94
0.20
0.13
0.23
0.28
0.28
0.21
(maggnal))
0.20
Table 2: Marginal predictive performance on Waymo Open Motion Dataset motion prediction.
Results presented the standard splits of the validation and test datasets (Ettinger et al. 2021) evaluated with traditional marginal metrics for t = seconds. minADE, minFDE reported for =
predictions (Alahi al. 2016 Pellegrin al. 2009;; Miss Rate (MR) (Chang et al., (2099) within meters of the target. See Appendix for t = 3 or seconds. We include the challenge winner results in
this table (Waymo] 2021).
4.1 MARGINAL MOTION PREDICTION
We first evaluate the performance of Scene Transformer trained and evaluated as a traditional marginal, per-agent motion prediction model. This is analogous the problem illustrated Figure ! (left). For all results until Section 4.4 we use masking strategy that provides the model with all agents as input, but with their futures hidden. We also mask out future traffic light information.
Argoverse. We evaluate on the popular Argoverse (Chang et al. 2019) benchmark to demonstrate the efficacy of our architecture. During training and evaluation, the model is only required to predict the future of the single agent of interest. Our best Argoverse model uses D 512 feature dimensions and label smoothing for trajectory classification. Our model achieves state-of-the-arr results compared to published, prior work in terms of minADE and minFDE (Table I).
Waymo Open Motion Dataset (WOMD). We next evaluate the performance our model with D = 256 on the recently released WOMD (Ettinger et al., 2021 for the marginal motion prediction task. This task a standard motion prediction task where up to agents per scene are selected to have their top 6 motion predictions evaluated independently. Our model trained with the marginal loss achieves state-of-the-arr results on the minADE, minFDE, and miss rate metrics (Table
4.2 JOINT MOTION PREDICTION
To evaluate the effectiveness of Scene Transformer when trained with a joint loss formulation (Section 3.4). we evaluate our model on the Interaction Prediction challenge in WOMD (Ettinger et al. 2021). This task measures the performance of the model predicting the joint future trajectories of two
We exclude comparing to public leaderboard entries that have not been published since their details are not available, but note that results are competitive the leaderhoard of the submission date.
7

---

Published as a conference paper at ICLR 2022
Interaction Prediction
minSADE
minSFDE J
SMR
MAPT
veh
ped
eye
ped
eye
ped
eye
veh
ped
eye
volid
LSTM baseline (atinger et al. 2021
2.42
2.73
3.16
6.07
4,20
6.46
0.66
1.00
0.83
0.007
0.06
0.02
Ours (margial-&--oon
2.04
1.62
2.28
4.94
3.81
5.67
0.54
0.63
0.72
0.11
0.05
0.03
Ours (joint, MP-caly)
1.72
1.38
1.96
3.98
3.11
4.75
0.49
0.60
0.73
0.11
0.05
0.03
Ours (joint, multi-task)
1.72
1.39
1.94
3.99
3.15
4.69
0.49
0.62
0.71
0.11
0.06
0.04
test
LSTM baseline Ertinger et al. 2021
2.46
2.47
6.22
4.30
6.26
0.67
0.89
0.89
0.06
0.03
0.03
HealRm44 (No et al, 2021
2.93
1.77
2.74
7.20
4.06
6.69
0.80
0.80
0.91
0.97
0.05
0.00
Ours (marginal as joinl)
2.08
0.64
0.73
0.08
0.05
0.03
Ours (joiat, MP-oaly)
1.76
1.38
1.95
4.08
3.19
4.65
0.50
0.62
0.70
0.10
0.05
0.04
Ours (joint, mulli lask))
1.74
1.41
1.95
4.06
3.26
4.68
0.50
0.64
0.71
0.13
0.04
0.03
Table 3: Joint predictive performance on Waymo Open Motion Dataset motion prediction.
Results presented on the interactive splits of the validation and test datasets (Ettinger et al. 202T) evaluated with scene-level joint metrics for I. 8 seconds. minSADE, minSFDE for k = 6 predictions (Alahi et al. 2016; Pellegrini et al. p009); Miss Rate (MR) (Chang et al. (019) within meters of
the target. "S" indicates scene-level joint metric. See Appendix for t = 3 or 5 seconds. We include the challenge winner results in this table (Waymo 2021).
interacting agents (Figure [ right), and employs joint variants of the common minADE, minFDE, and Miss Rate (MR) metries denoted as minSADE, minSFDE, SMR. Note that the "S" indicates "scene-level" metries. These metrics aim to measure the quality and consistency of the two agents joint prediction for example, the joint variant of Miss Rate (SMR) only records a "hit" if both interacting agent's 's predicted trajectories are within the threshold of their respective ground truths.
We find that for the Interaction Prediction challenge our joint model's 's joint predictions easily outperforms the WOMD provided baseline as well as a marginal version of the model converted into joint prediction F into joint predictions. (Table This shows that beyond the strength of our overall architecture and approach, that explicitly training model as a joint model significantly improves joint performance joint metrics. A notable observation is that even though the Interaction Prediction task only requires predicting the joint trajectories of two agents, our method is fully general and predicts joint consistent futures of all agents.
4.3 FACTORIZED AGENTS SELF-ATTENTION
Factorized self-attention confers two benefits to our model: (a) it is more efficient since the attention over smaller set, and (b) it provides an implicit identity to each agent during the attention across time. We ran an experiment where we replaced each axis-factorized attention layer (each pair of time and agent factorized layer) with a non-axis factored attention layer. This increased the computational cost of the model and performed worse on the Argoverse validation dataset: the factorized version achieved a minADE of 0.609 with the factored version, and 0.639 with the non- cantorctrr version.
4.4 ADVANTAGES OF A MASKED SEQUENCE MODELING STRATEGY
Our model is formulated as a masked sequence model (Devlin al. 2019). where training and inference time we specify which agent timesteps to mask from the model. This formulation allows us select which information about any agent at any timestep to supply to the model, and measure how the model exploits or responds to this additional information. We can express several motion prediction tasks at inference time in mask space (Figure 2. in effect providing multi-task model. We can use this unique capability to query the models for counterfactuals. What would the model predict given a subset of agents full trajectories (conditional motion prediction), or given a subset of agents final goals coal-condiiioned motion prediction)? This feature could be particularly useful for autonomous vehicle planning to predict what various future rollouts of the scene would look like given a desired goal for the autonomous vehicle (Figure 3.
$ "Note that the output a joint model can be directly used a marginal evaluation. However, converting the output of a marginal model into a joint evaluation is nuanced because there lacks an association of futures across agents (Figure []) We employ simple heuristic to convert the outputs of a marginal model for joint evaluation: we take the top 6 pairs of trajectories from the combination of both agent's 's trajectories for 36 total pairs, and retain the top pairs with the highest product of probabilities.
8

---

Published as a conference paper at ICLR 2022
Scenario A
Scenario
Scenario C
Figure 3: Gondicond prediction navigates AV to selected goal positions. Rectangles indi- cate vehicles on the road. Lines indicate the road graph (RG) colored by the type of lane marker. Circles indicate predicted trajectory for each agent. Star indicates selected goal for AV. Each column represents a scenario in which the AV is directed to perform one of two actions taking into account the dynamics of other agents. (A) AV instructed either change lanes or remain in the same lane. (B) AV instructed to stop to allow oncoming vehicle to turn left, or adjust to the side in narrow passage so that the oncoming vehicle has enough space pass. (C) AV instructed to either proceed straight or turn right at the intersection.
In the previous results, we used a prefix-mask mask during training and inference, which shows the first few steps and predicts the remaining timesteps for all agents. For this experiment, employ two masking strategies to test on the interactive split of the WOMD, namely "conditional motion prediction" (CMP) where show one of the interacting agents' full trajectory, and "goal- conditioned motion prediction" (GCP) where we show the autonomous vehicle's 's desired goal state (Figure 2) We train model using each strategy (including MP masking) 1/3 of the time, and evaluate the model on each all tasks. We find that across the 44k evaluation segments, the multi-task model matches the performance of our MP-only trained joint model on both joint (Table (3] and marginal metrics (see also Appendix, Table IT: multi-tasking training does not significantly degrade standard motion prediction performance. We qualitatively examine the performance of the multi-tase model in GCP setting, and observe that the joint motion predictions for the AV and non-AV agents flexibly adapt to selected goal points for the AV (Figure
We note that dataset average metrics like ADDE do not capture forecasting subtleties (Ivanovic Pavone 202I) that are critical for measuring progress in the field. Unfortunately, the Waymo Motion Open Dataset Interactive Prediction set does not contain pre-defined metadata necessary to evaluate specific interesting slices using different models. In Appendix we provide some initial analysis of slices that could compute the dataset, which illustrate how joint and marginal model predictions behavior differently as the scene a) seales with more agents complicated scenes, b) differs in the average speed of agents. We also show that joint prediction models demonstrate joint consistent futures via lower inter-prediction overlap compared to marginal models.
joint
DISCUSSION
We propose a unified architecture for autonomous driving that is able model the complex interac- tions of agents in the environment. Our approach enables single model to perform motion prediction, conditional motion prediction, and goal-conditioned prediction. In the field of autonomous driving, elaborations of this problem formulation may result in learning models for planning systems that
quantitatively improve upon existing systems (Buehler et al. 2009: Montemerlo et al. 2008 Ziegler et al. 2014: Zeng et al. 2019 Liu al. 2021a). Likewise, such modeling efforts may be used to directly go after issues of identilying interacting agents in an environment, and potentially provide an important task for identifying causal relationships (Arjovsky al. 2019. SSchölkopt al. 2021)

---

Published as a conference paper at ICLR 2022
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaogiang Zheng. TensorFlow: Large-scale machine learning heterogeneous systems, 2015. URL http: /ttp///nnnnoo..oe/ee//e/ tensorflow.org.
org/ o
Software available from
Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social LSTM: Human trajectory prediction in crowded spaces. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 961-971, 2016.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paze Invariant risk minimization.
arXiv preprini arXiv: 1907. 0028999, 2019.
Irwan Bello. LambdaNetworks: Modeling long-range interactions without attention. International Conference on Leurning Representations (ICLR), 2013.
Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoe V Le. Attention augmented convolutional networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3286-3295, 2019.
Manoj Bhat, Jonathan Francis, and Jean Oh. Trajformer: Trajectory prediction with local self-attentive contexts for autonomous driving. arXiv preprint arXiv:2011.14910 2020.
Yuriy Biktairov, Maxim Stebelev, Irina Rudenko, Oleh Shliazhko, and Boris Yangel. Prank: motion prediction based on ranking. In Advances in neural information processing systems, 2020.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020.
Martin Buchler, Karl lagnemma, and Sanjiv Singh. The DARPA urban challenge: autonomous vehicles in city traffic, volume 56. springer, 2009.
Thibault Buhet, Emilie Wirbel, and Xavier Perrotton. Plop: Probabilistic polynomial objects trajectory planning for autonomous driving. In Conference on Robot Learning, 2020.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 1621-116611, 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pp. 213-229. Springer, 2020.
Sergio Casas, Cole Gulino, Renjie Liao, and Raquel Urtasun. Spagnn: Spatially-aware graph neural networks for relational behavior forecasting from sensor data. In 2020 IEEE International Conference Robotics and Automation, ICRA 2020, Paris, France, May 31 August 31, 2020, pp. 9491-9497. IEEE, 2020a.
Sergio Casas, Cole Gulino, Simon Suo, Katie Luo, Renjie Liao, and Raquel Urtasun. Implicit latent variable model for scene-consistent motion forecasting. In Proceedings of the European Conference on Computer Vision (ECCV). Springer, 2020b.
Yuning Chai, Benjamin Sapp, Mayank Bansal, and Dragomir Anguelov. Multipath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction. In Conference on Robot Learning, 2019.
10

---

Published as a conference paper at ICLR 2022
Rohan Chandra, Uttaran Bhattacharya, Aniket Bera, and Dinesh Manocha. Traphic: Trajectory prediction in dense and heterogeneous traffic using weighted interactions. CoRR, abs/1812.04767, 2018.
1812.04767
Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, et al. Argoverse: 3d tracking and forecasting with rich maps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern 2019.
Recognition, pp.
Henggang Cui, Vladan Radosavljevic, Fang-Chich Chou, Tsung-Han Lin, Thi Nguyen, Tzu-Kuo Huang. Jeff Schneider, and Nemanja Djuric. Multimodal trajectory predictions for autonomous driving using deep convolutional networks. In 2019 International Conference on Robotics and Automation (ICRA), pp. 2090-2096. IEEE, 2019.
Nachiket Deo and Mohan M. Trivedi. Trajectory forecasts in unknown enviromments conditioneed on grid-based plans. CoRR, abs/2001.00733. 2020.
00735
Jacob Devlin, Ming-Was Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles Qi, Yin Zhou, et al. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. arXiv preprint arXiv:2 10133, 2021.
Jiyang Gao, Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguelov, Congcong Li, and Cordelia Schmid. VectorNet: Encoding hd maps and agent dynamics from vectorized representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Thomas Gilles, Stefano Sabatini, Dzmitry Tsishkou, Bogdan Stanciulescu, and Fabien Moutarde. HOME: heatmap output for future motion estimation. CoRR, abs/2105.10968, 2021. URL
https://arxiv. /rg/ atss///...
Francesco Giuliari, Irtiza Hasan, Marco Cristani, and Fabio Galasso. Transformer networks for trajectory forecasting. In International Conference Pattern Recognition, 2020.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings. 2010.
Junru Gu, Chen Sun. and Hang Zhao. Densetnt: End-to-end trajectory prediction from dense goal sets. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social GAN: Socially acceptable trajectories with generative adversarial networks. In CVPR, 2018.
Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidi-
mensional transformers. arXiv preprint 191.v:1112.12/888 2019.
Joey Hong, Benjamin Sapp, and James Philbin. Rules of the road: Predicting driving behavior with convolutional model of semantic interactions. In CVPR, 2019.
Zhiyu Huang, Xiaoyu Mo. and Chen Lv. Recoat: A deep learning framework with attention mechanism for multi-modal motion prediction. In Workshop Autonomous Driving, CVPR, 2021.
Wei-Chih Hung, Henrik Kretzschmar, Tsung- Yi Lin, Yuning Chai, Ruichi Yu, Ming Hsuan Yang, and Drago Anguelov. Soda: Multi-object tracking with soft data association. arXiv preprint arXiv:2008.0777 077255 2020.
11

---

Published as a conference paper at ICLR 2022
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448 456. PMLR. 2015.
Boris Ivanovic and Marco Pavone. Rethinking trajectory forecasting evaluation. arXiv preprint arXiv:2107.10297, 2021.
Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter performance analysis of tensor processing unit. In Proceedings of the 44th annual international symposium computer architecture, pp. 1-12, 2017.
R. Kesten, M. Usman, Houston, T. Pandya, K. Nadhamuni, A. Ferreira, M. Yuan, B. Low, Jain, P. Ondruska, S. Omari, S. Shah, A. Kulkarni, A. Kazakova, C. Tao, L. Platinsky, W. Jiang, and V. Shet. Lyft level 5 pereeption dataset 2020. huttps://leves:. lyft. com/ www/cattaaaa/eeeeccoooo datastt/ 2019.
Siddhesh Khandelwal, William Qi, Jagjeet Singh, Andrew Hartnett, and Deva Ramanan. What-if motion prediction for autonomous driving. arXiv preprint arXiv:2008. 10587, 2020.
Stepan Konev, Kirill Brodt, and Artsiom Sanakoyeu. Motioncan: A strong baseline for motion prediction in autonomous driving. In Workshop on Autonomous Driving. CVPR, 2021.
Namhoon Lee, Wongun Choi, Paul Vernaza, Christopher B Choy, Philip HS Torr, and Manmohan Chandraker. Desire: Distant future prediction in dynamic scenes with interacting agents. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 336-345, 2017.
Lingyun Luke Li, Bin Yang, Ming Liang, Wenyuan Zeng. Mengye Ren, Sean Segal, and Raquel Urta- sun. End-to-end contextual perception and prediction with interaction transformer. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020.
Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, and Raquel Urtasun. Learning lane graph representations for motion forecasting. In European Conference on Computer Vision (ECCV), 2020.
Jerry Liu, Wenyuan Zeng, Raquel Urtasun, and Ersin Yumer. Deep structured reactive planning. arXiv preprini arXiv:2101.0688 06632 2021a.
Yicheng Liu, Jinghuai Zhang, Liangji Fang, Qinhong Jiang, and Bolei Zhou. Multimodal motion prediction with stacked transformers, 2021b.
Francesco Marchetti, Federico Becattini, Lorenzo Seidenari, and Alberto Del Bimbo. Mantra: Memory augmented networks for multiple trajectory prediction. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 7143-7152, 2020.
Jean Mercat, Thomas Gilles, Nicole Zoghby, Guillaume Sandou, Dominique Beauvois, and Guillermo Gil. Multi-head attention for joint multi-modal vehicle motion forecasting. IEEE International Conference on Robotics and Automation, 2020.
Gregory Meyer and Niranjan Thakurdesai. Learning an uncertainty-awar object detector for autonomous driving. In JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 10521-10527, 2020.
Xiaoyu Mo, Zhiyu Huang, and Chen Lv. heterogeneous edge-enhanced graph attention network. In Workshop on Autonomous Driving, CVPR, 2021.
interactive agent trajectory prediction using
Michael Montemerlo, Jan Becker, Suhrid Bhat, Hendrik Dahlkamp, Dmitri Dolgov, Scott Ettinger, Dirk Hachnel, Tim Hilden, Gabe Hoffmaann, Burkhard Huhnke, et al. Junior: The stanford entry in the urban challenge. Journal of field Robotics, 25(9):569
2008
12

---

Published as a conference paper at ICLR 2022
Seong Hyeon Park, Gyubok Lee, Jimin Seo, Manoj Bhat, Minseok Kang, Jonathan Francis, Ashwin Jadhav, Paul Pu Liang, and Louis- Phiii-p Morency. Diverse and admissible trajectory forecasting through multimodal context understanding. In European Conference on Computer Vision, pp. 282-298. Springer, 2020.
Stefano Pellegrini, Andreas Ess, Konrad Schindler, and Luc Van Gool. You'll never walk alone: Modeling social behavior for multi-targe ttrrgt tracking. In 2009 JEEE 12th International Conference on Computer Vision, pp. 261-268. IEEE, 2009.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 652-660, 2017.
Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. Stand-alone eelf-attention in vision models. In Advances in Neural Information Processing Systems, volume 32, 2019.
Nicholas Rhinehart, Rowan McAllister, Kris Kitani, and Sergey Levine. Precog: Prediction con- ditioned on goals in visual multi-agent settings. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp-
2019.
Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++:: Dynamically- feasible trajectory forecasting with heterogeneous data. arXiv preprint arXiv:2001.0309: 2020.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 2021.
Aravind Srinivas, Tsung-Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. In CVPR, 2021.
Simon Suo, Sebastian Regalado, Sergio Casas, and Raquel Urtasun. Trafficsim: Learning to simulate realistic multi-agent behaviors. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818 18-2826, 2016.
Charlie Tang and Russ Salakhutdinov. Multiple futures prediction. In NeurlPS, 2019.
Sebastian Thrun, Mike Montemerlo, Hendrik Dahlkamp, David Stavens, Andrei Aron, James Diebel, Philip Fong, John Gale, Morgan Halpenny, Gabriel Hoffmaann, et al. Stanley: The robot that won the darpa grand challenge. Journal of field Robotics, 23(9):661-69 2006.
Ekaterina Tolstaya, Reza Mahjourian, Carlton Downey, Balakrishnan Vadarajan, Benjamin Sapp, and Dragomir Anguelov. Identifying driver interactions via conditional behavior prediction. 202 IEEE International Conference on Robotics and Automation (ICRA), 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv:1706.000
2017.
Ashish Vaswani, Prajit Ramachandran, Aravind Srinivas. Niki Parmar, Blake Hechtman, and Jonathon Shlens. Scaling local self-attention for parameter efficient visual backbones. In CVPR, 2021.
Huiyu Wang, Yukun Zhu, Bradley Green, Hartwig Adam, Alan Yuille, and Liang-Chich Chen. Axial-deeplab: Stand-alowe axial-attention for panoptic segmentation. European Conference on Computer Vision, pp. 108-126. Springer, 2020.
Waymo. Waymo open dataset challenge 2021 winners. Shallengesswwinee. 2021. Accessed: 2021-10-01.
https://waymo.coo
Maosheng Ye, Tongyi Cao, and Qifeng Chen. Tpen: Temporal point cloud networks for motion forecasting, arXiv preprint arXiv: 103.020603 2021.
13

---

Published as a conference paper at ICLR 2022
Raymond A Yeh, Alexander G Schwing, Jonathan Huang, and Kevin Murphy. Diverse generation for multi-agent sports games. Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition, pp. 4610-4619, 2019.
Cunjun Yu, Xiao Ma, Jiawei Ren, Haiyu Zhao, and Shuai Yi. Spatio-temporal graph transformer networks for pedestrian trajectory prediction. In European Conference on Computer Vision, pp. 507-523. Springer, 2020.
Ye Yuan, Xinshuo Weng, Yanglan Ou, and Kris Kitani. Agentformer: Agent-aware transformers for socio-temporal multi-agent forecasting, arXiv preprint 11KK:::203.1102333 2021.
Wenyuan Zeng, Wenjie Luo, Simon Suo, Abbas Sadat, Bin Yang, Sergio Casas, and Raquel Urtasun. End-to-end interpretable neural motion planner. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8660-8661 2019.
Wei Zhan, Liting Sun, Di Wang, Haojie Shi, Aubrey Clausse, Maximilian Naumann, Julius Küm- merle, Hendrik Königshof, Christoph Stiller, Arnaud de La Fortelle, and Masayoshi Tomizuka. INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps. 190.00098 [cs, eess], 2019.
Hang Zhao, Jiyang Gao, Tian Lan, Chen Sun, Benjamin Sapp, Balakrishnan Varadarajan, Yue Shen, Yi Shen, Yuning Chai, Cordelia Schmid, et al. Tnt: Target-driven trajectory prediction. arXiv preprint ar X00: 200.000080.994, 2020.
Tianyang Zhao, Yifei Xu, Mathew Monfort, Wongun Choi, Chris Baker, Yibiao Zhao, Yizhou Wang, and Ying Nian Wu. Multi-agent tensor fusion for contextual trajectory prediction. In CVPR, pp. 12134, 2019.
Julius Ziegler, Philipp Bender, Thao Dang, and Christoph Stiller. Trajectory planning for bertha local, continuous method. In 2014 IEEE intelligent vehicles symposium proceedings, pp. 450-457. IEEE, 2014.
14

---

Published as a conference paper at ICLR 2022
Appendix
A ARCHITECTURE AND TRAINING DETAILS
Overview. ( additional details about the Scene Transformer architecture and training. The bulk of the computation and parameters reside in the Transformer layers. Table ( lists all of the operations and learned parameters in our implementation of Transformer. MLPs are employed to embed the original data (e.g. layers A, B, C), build F futures from the encoding (layer 7), and predict the final position, headings and uncertainties of all agents (layers Z1 and Z2).. The resulting incention-based architecture using D = 256 feature dimensions contains 15,296. 136 parameters. Nearly all settings used for both WOMD and Argoverse are identical, except that for Argoverse we use D = 512 and label smoothing to get our best marginal results. The resulting model is trained on a TPU custom hardware accelerator (Jouppi et al., (017) and converges in about 3 days of training.
Decoding multiple futures. In order to allow the decoder to output F distinct futures, we perform the following operations. The decoder receives as input a tensor of shape |A, T, D] corresponding to A agents across T time steps and D feature dimensions. The following series of simple operations restructures the representation to predict F futures. First, the representation is tiled F times to generate [F, A, T, D]. We append a one-hot encoding to the final dimension where 1 indicates which the identity of each future, resulting in a tensor of shape [F, A, D + F]. The one-hot encoding allows the network learn embedding for each of the F futures. For computational simplicity, the resulting representation is propagated through a small MLP to produce a return a tensor of the shape [F", A, D].
Padding and Hidden Masks. Padding and hidden masks are important to get right in the implemen- tation of such a model. In particular, we need ensure that the masks do not convey additional future information to the model (e.g., if the model knows which timesteps an agent is visible or occluded based on the padding in the data, may take advantage of this and not be able to generalize). We use the concept of padding to indicate the positions where the imput is absent or provided. This is distinct from the hidden mask of shape [A, T that used for task specification. The hidden mask is used to query the model to inform it on which locations to predict, while the padding tells us which locations have inputs and ground-truth to compute a loss over. All padded positions are set to be hidden during preprocessing, regardless of the masking scheme, so the model tries to predict their values. All layers in our implementation are padding aware, including normalization layers (like batch normalization used in our encoding MLPs) and attention operations. Our attention operations set the attention weights to 0 for padded elements. If after providing our task specific hidden mask, no non-padded (valid) time steps exist for an agent, the whole agent is set to be padded. This prevents agent slots with no valid data from being used in the model.
Predicting uncertainties. For each predicted variate such as position or heading, we additionally predict a corresponding uncertainty. In early experiments we found that predicting paired uncertainty improves the predictive performance of the original variate and provides useful signal for interpreting the fidelity of the predicted value. We employ a loss function that predicts a parameterization of the uncertainty corresponding to a Laplace distribution (Meyer & Thakurdesa] 2020).
Loss in lateral and longitudinal coordinates. We use a Laplace distribution parameterization for the loss, with a diagonal covariance such that each axis is independent. To enable the model to learn meaningful uncertainties, we rotate the scene per box prediction such that each prediction's 's associated ground-truth is axis aligned. This formulation results in uncertainty estimates that correspond to the agent's 's lateral and longitudinal coordinates.
Heading representation: Our output format 7 dimensional, where one of the dimensions, heading, is represented in radians. To supervise the loss on the heading dimension, employ a now standard method of "wrapping" the angle difference between the predicted and groundtruth heading for each agent; see Figure
Marginal predictions from a joint model. The model has the added feature of being easily adapted for making marginal predictions for each agent. To produce per- per-trajectory scores for each agents predictions, we attach an artificial extra time step to the end of the [A,T, D] agent feature matrix to give us matrix of shape [A, T + 1, D). This additional feature provides the model with extra
15

---

Published as a conference paper at ICLR 2022
def WrapAngleRan eRadi (angles_rad, min_val=-np. pi, pox_val-np-pi) max min diff max val min val return min_val + tf.math. floormed (angrmd rad max val, max min diff)
heading_oitt _ifg = pred heading - gt_ ____heiding
hoading_error = WrapAngloRRdiann (heading_diff, min_val=-PI, max_val-PI)
heading_loss HuberLoss (heading_error)
Figure Pseudo-code for heading loss component.
With Augmentation?
Vchicle minADE @8s
Ped minADE @8s
Cyelist minADE @ 8s
No
1.30
0.69
1.29
Yes
1.18
0.59
1.15
Table 4: WOMD validation set performance with and without data augmentation during training.
capacity for representing each agent, that is not tied to any timestep. This additional feature can be used to predict which of the & trajectories most closely matches the ground truth for that agent.
Data augmentation. We use two methods for data augmentation to aid generalization and combat overfitting on both datasets. First, we use agent dropout, where we artificially remove non-predicted agents with some probability. We found probability of 0.1 worked well for both datasets. We also found it beneficial to randomly rotate the entire scene between [- 5] after centering the agent of interest. On Argoverse, this agent of interest is the agent the task designates to be predicted, where as on the WOMD we always center around the autonomous vehicle (AV), even though for some tasks it is not one of the predicted agents. Lastly, each Argoverse scene contains many agents that the model is not required predict; we employ these contextual agents as additional target training data if the contextual agents moved by at least 6m. On WOMD, Table shows data augmentation on the standard validation set does improve minADE modestly, though does not account for the majority of the benefit of the model.
Argoverse classification label smoothing: Our best model on Argoverse uses D 512 feature dimensions, but naively scaling the model this way leads to severe overfitting on the classification subtask. During training of our best model we employ label smoothing (0.1 1/6 for negatives and 0.9 + 1/6 for positive target).
WOMD redundant trajectory combination: The AP evaluation metric on the WOMD expects that exactly one of the trajectories is given high probability and the other trajectories a low probability. For example, the future prediction for a stationary agent expected to have only one future with high probability score, with low score for the rest even though the trajectories may all be the same, in this case, stationary. Our best model combines redundant trajectories together if they are close in spatial location (less than 3.2m) at the final timestep.
Embedding of agents and road graph. To generate input features, we use sinusoidal positional embeddings (Vaswani et al., (017) embed the time (for agents and dynamic roadgraph) and ryz=ccordinatess separately into a D dimensional features per dimension. We encode the type of each object using a one-hot encoding (e.g. object type, lane type, etc), and concatenate any other features provided in the data set such as yaw, width, length, height, and velocity. Dynamic road graphs have second one-hot encoding indicating state, like the traffic light state. Lastly, for agents and the dynamic road graph, we add binary indicator whether the agent is hidden at that time step. If the agent or dynamic road graph element is hidden, all input features (e.g. position, type, velocity, state, etc) are set to 0 before encoding except the time embedding, which are linearly spaced values at the dataset's update rate starting at O, and the hidden indicator.
For agents and the dynamic road graph, we use 2 layer MLP with hidden and output dimension of D to produce a final feature per agent or object and per time step. For the static road graph, we must reduce point cloud of up to 20,000 road graph points, each belonging to smaller set of polylines, to single vector per polyline. Because some lanes can be extremely long, we break up any polyline longer than 20 points into new set of smaller polylines. Then, we apply small PointNet (Qi et al.
16

---

Published as a conference paper at ICLR 2022
2017) architecture with layer MLP with hidden and output dimension of D each point, and use max pooling per polyline to get final feature per element.
Inference latency. Scene Transformer produces prediction for every agent in the scene in single pass. Its inference speed depends on the number agents in the scene. Our preliminary profiling of inference speed ranged from 52 ms (32 agents) to 175 ms (128 agents) the Waymo Open Motion Dataset (128 agents, 91 timesteps for each agent, up to 1400 roadgraph elements). This was measured on an Nvidia V100 using standard TensorFlow inference graph float32 without optimization tricks, optimization tuning, etc. This is in line with the expected linear scaling of the factorized attention modules.
17

---

Published as a conference paper at ICLR 2022
Meta-Arch
Name
Input
Operation
Queries
Kays/Values
Acruss
Alten Matrix
Quiput Sizee
Puram
Encoder
A
Agents
MLP + BN
[A. T. D]
334080
B
Dyna RG
MLP* + BN
[G 2. D
337408
Static
gG
270392
D
B,C
Transformer
Agent
Agents
Time
[4.2.
[A, D] [A,
789824
Agents
Agenes
Agents
[2
F
E
Tranformer Tranffrmerr
Agenis
Agent
Time
[A
[A D [A,T,
789824
Agents
Agones
Agents
H
G
Transformer Transfrmmr
Agents
Agenss
Time
[.
[A,
789824
I
Agents
Agents
Agents
[]
[A,T.
789824
J
I
Transformer Tonsformer
Apenis
Stusc RG
Time
G.
[A,
789824
K:
J
Agents
Dyna RG
Time
[T, AD
[A,
789934
C
K
Transformer Transformer
Agenis
Time
[A.T.T]
[A.
789824
M
Agents
Agents
Agents
AT
[A, DJ
789824
A
Tranformer Transformer
Agents
Sturi RG
A,Gsl
[A,
780834
N
Agents
Dyna RG
Time
[A, D]
789824
Agents Apenis
Agents
[A.2.T]
[A.
Q
Transfrmer
Agens
Agenis
AT
[A
789824
Decoder
R
Q
Tile
A, DI [F, A, D4 [F
8
T
Transformer Tranformer
Agents
Agents
Time
[A.2.T]
[F, T. D
789824
Agenis
Agents
Agents
W
V
Transformer
Agents
Agents
Time
[A.2. [T
[F D [F,
789924
Agents
Agone
Agents
y
X
Layer Norm
[F T.
512
MLP
[F,
Zg
y
MLP
[R,A,T
1799
Optimizer
Adam (or 10- 01 0.9, 92 0.999)
Leaminng Rate Schedule
Toral spocis: 150, Linear mmp-up: D.1 spochs.
Gradient Clipping (norm)
5.0
Weighr initialization
Glonet & Bengio 2010
Weight docay
Position Embeediings
Min Timescale Max Timescale:
Temporal Embeddings
Min Timescule: 6; Max Timescale: 80
Furare chessficatiion weight
classifiation weight
Laplace Target Scale
1.0
Table 5: Scene Transformer architecture and training details. The network receives as input of A agents across time steps and features. is the total number of input features (e.g. 3-D position, velocity, object type, bounding box size). A subset of these inputs masked. G3 and GD the maximum number of road graph (RG) elements and D is the total number of features. MLP and BN denote multilayer perception and batch normalization (loffe & Szegedy 2015), respectively. The output of the network is E1 and Zz. Z1 corresponds to predicting the logits for classifying which one of the F futures most likely. Ze corresponds the predicted zggz-corrdinates with their associated uncertainties, and single value for heading. In our model D = 256, K and F = 6 for total of 15,296, 136 parameters for both datasets. For the Waymo Open Motion Dataset G, 1400, G. 16, T = 91, A 128, and for Argoverse G, = 256, Ga 0, T = 50, and = 64. All layers employ ReLU nonlinearities.
18

---

Published as a conference paper at ICLR 2022
Name
Input
Operation
Parameter Sizes
Output Size
Param
X
X
Layer Norm
[D], [D] H, D B H D [D, Hb. [ D1
[A,T. D
512
K
X
Affine Projection
[A,T, H, D
65792
H
V
X
Affine Projection
[A,T, H. D 65792
Q.
X
Affine Projection
DD DI. [H, M1 [AT
[A, D1
65792
Q
Q.
Rescale
[A,T, D
64
Y
Q,
K,K.V
softmax V
[, D
H
Y2
Y1
Affine Projection
H, H D), D), [D]
D
65792
X,
Sum
D)
F1
V2
MLP
[D, ID). [k:D]
[A,T
263168
F2
S
MLP
[k:D, D). [D]
[I
262400
Z
F2
Layer Norm
[D], [D]
[A,T.D]
512
Total
789824
Table 6: Transformer architecture. The network receives as input X. and outputs Z. All MLP's employ ReLU nonlinearity. D is the number of feature dimensions; H is the number of attention heads. In our model D=256, H-4 and k=4.
19

---

Published as a conference paper at ICLR 2022
B ADDITIONAL MOTION PREDICTION RESULTS
minADE ped
minFDE. ped
MR
mAP1
Motion Prediction
veh
eye
veh
veh
ped
veh
ped
valid
baseline Etringer et ours (marginal)
0.39
0.19
0.41
0.65
0.33
0.27
0.33
0.20
0.39
0.57
0.33
0.67
0.11
0.07
0.21
0.38
0.33
0.28
ours [oint-as-marree as-marginal)
0.42
0.28
0.50
0.78
0.51
0.94
0.19
0.21
0.32
0.34
0.25
0.23
ours lask joint marginal)
0.43
bascline (Etinggr al. 2021
0.39
0.20
0.41
0.65
0.36
0.74
0.14
0.07
0.25
0.34
0.32
0.24
ours (marginal)
0.32
0.20
0.38
0.56
0.33
0.67
0.11
0.07
0.21
0.38
0.32
0.28
ours (joint marginal)
0.49
0.24
ours (multi-task task (maa-ia--aa
0.44
0.29
0.50
0.80
0.53
0.93
0.20
0.22
0.32
0.32
0.24
0.24
Table 7: Marginal predictive performance Waymo Open Motion Dataset motion prediction for t seconds, Please see Table for details.
minSADE ped
minSFDE J.
SMR ped
mAP1
Interaction Prediction
veh
cyc
veh
ped
eyc
veh
eyc
veh
ped
eyc
valid
baseline Ettinger al. 2021
0.58
0.43
1.13
0.86
0.47
0.61
0.06
(margmal jx-jomm))
0.45
0.37
0.52
0.91
0.75
1.04
0.35
0.42
0.52
0.20
0.12
0.09
ours (joint)
0.41
0.34
0.47
0.81
0.65
0.92
0.29
0.38
0.49
0.26
0.14
0.11
ours (multi-task joint)
0.40
0.34
0.47
0.80
0.65
0.91
0.28
0.38
0.49
0.27
0.17
0.13
test
buseline Entineer et ul. 2021
0.58
0.42
0.61
1.14
0.85
1.21
0.45
0.47
0.61
0.16
0.11
0.05
ours (marginal-ds-joun))
0.45
0.36
0.93
0.74
0.36
0.40
0.55
(joint)
0.41
0.33
0.48
0.82
0.64
0.94
0.29
0.36
0.50
0.18
0.12
0.07
(multi-task joint)
0.41
0.34
0.48
0.81
0.65
0.94
0.29
0.37
0.51
0.26
0.15
0.10
Table Joint predictive performance on Waymo Open Motion Dataset motion prediction for I = 3 seconds. Please see Table for details.
Motion Prediction
minADE, ped
minFDE, ped
MR pod
mAP1
veh
eye
veh
oye
eye
veh
ped
eye
baseline Eltinger al. 2021
0.74
0.37
0.75
1.36
0.73
1.43
0.17
0.10
0.25
0.29
0.27
0.26
ours gnal
0.65
0.35
0.69
1.23
0.67
1.30
0.15
0.10
0.22
0.33
0.26
0.25
ours joint-as-marginall
0.83
0.93
0.19
(malti task marginal)
0.84
0.52
0.92
1.72
1.08
1.87
0.24
0.26
0.33
0.26
0.18
0.18
text
bascline Eminger et al. 2021
0.74
0.37
0.76
1.35
0.73
1.43
0.17
0.10
0.25
0.29
0.26
0.23
murs (marginal)
(joint marginal)
0.83
0.52
0.91
1.68
1.10
1.88
0.23
0.26
0.33
0.26
0.19
0.22
ours (multi-task joint-as-marginnl
0.85
0.53
0.93
1.72
1.10
1.91
0.24
0.27
0.34
0.26
0.19
0.21
Table 9: Marginal predictive performance Waymo Open Motion Dataset motion prediction for f = seconds. Please see Table for details.
20

---

Published as a conference paper at ICLR 2022
minSADE. ped
minSHDE. ped
SMR
MAP1
Interaction Prediction
veh
veh
veh
ped
eye
veh
ped
valid
baseline Emtnger et al. 2021
1.19
0.90
1.25
2.64
1.98
2.82
0.55
0.57
0.70
0.13
0.09
0.04
ours (margias as-jomt)
0.96
0.78
1.08
2.15
1.75
2.46
0.44
0.51
0.60
0.13
0.09
0.06
ours (joint)
0.84
1.81
1.45
2.09
0.48
0.59
0.18
0.07
(multi-task joint)
0.83
0.68
0.95
1.79
1.45
2.08
0.38
0.48
0.59
0.19
0.10
0.06
baseline Fitinger al. 2021
1.21
0.89
1.26
2.70
1.96
2.80
0.56
0.59
0.69
0.13
0.07
0.03
ours (marginal-uss -us-oo-)
0.96
0.77
1.07
2.19
1.74
2.45
0.46
0.51
0.62
0.13
0.09
0.06
ours (joint)
0.85
0.67
0.96
1.85
1.44
2.10
0.39
0.50
0.58
0.17
0.10
0.07
ours (multi-task joint)
0.84
0.68
0.97
1.82
1.46
2.09
0.39
0.49
0.59
0.20
0.09
0.08
Table 10: Joint predictive performance on Waymo Open Motion Dataset motion prediction for t = seconds. Please see Table for details.
minADE
minFDE
mAP
Motion Prediction
eye
ped
eye
veh
ped
oye
veh
ped
eye
volld
1.STM haseline (Fitinger et al. 2021
1.34
0.63
1.26
2.85
1.35
2.68
0.25
0.13
0.29
0.23
0.23
0.20
(marginal)
0.20
0.12
0.24
0.26
0.27
0.20
ours (joint-as as- (o-mariinaa)
1.53
0.90
1.63
3.48
2.09
3.77
0.28
0.30
0.37
0.20
0.16
0.13
ours (molti-task joint-as-margina))
1.56
3. 68
0.29
0.29
0.37
0.19
0.15
0.15
LSTM baseline Ettinger et al. 2021
1.34
0.64
1.29
2.83
1.35
2.68
0.24
0.13
0.29
0.24
0.22
0.19
outs mmaginal)
1.17
0.60
1.17
2.48
1.25
2.43
0.19
0.12
0.22
0.27
0.23
0.20
(joint-as- ms--iigial)
1.52
3. 3.68
0.28
0.30
0.37
0.19
0.16
0.19
ours (multi-task joint-as-margina))
1.55
0.91
1.64
3.50
2.08
3.75
0.28
0.29
0.38
0.19
0.16
0.19
Table 11: Marginal predictive performance Waymo Open Motion Dataset motion prediction for 1 = seconds. Please see Table for details. We additionally include "joint-as-marginal" results for standard MP masked joint model, and our multi-task joint model. These are joint models evaluated as if they were marginal models (with no changes in the outputs).
minSFDE J
mAP
Interaction Prediction
veh
veh
eye
veh
pod
сye
veh
ped
сyс
volid
LSTM baseline (Ettinger et al. 2021
2.42
3.16
4.20
6.46
0.66
1.00
0.83
0.07
0.06
0.02
(marggnal ours (joint, mp-only)
2.04
1.62
2.28
4.94
3.81
5.67
0.54
0.63
0.72
0.11
0.05
0.03
1.72
1.38
1.96
3.98
3.11
4.75
0.49
0.60
0.73
0.11
0.05
0.03
ours jjoint, multi-task)
LSTM buseline (itinnger et al. 2021
2.46
2.47
2.96
6.22
4.30
6.26
0.67
0.89
0.89
0.06
0.03
0.03
ours (marginal-as join!)
(joint, mp-only)
1.76
1.38
1.95
4.08
3.19
4.65
0.50
0.62
0.70
0.10
0.05
0.04
ours (joini, multi tusk)
1.74
1.41
1.95
4.06
4.68
0.50
0.64
0.71
0.13
0.04
0.03
Table 12: Joint predictive performance on Waymo Open Motion Dataset motion prediction for 1 = seconds. These results are the same as Table 3 but is included here for completeness with other Appendix tables.
21

---

Published as a conference paper at ICLR 2022
B.1
UNDERSTANDING THE TRADE-OFFS BETWEEN THE JOINT AND MARGINAL MODELS
1.4
minADE
Inter-predicaiy Overiap
(96)
joint marginal
1.2
1
joint
marginal
of Predicted Agents
of Predicted Agents
Figure 5: Quantitative comparison of the marginal and joint prediction models. Left: The marginal minADE broken down function of number of predicted agents. When there are more agents, producing internally consistent predictions is more challenging, and hence the joint model performs slightly worse. Right: Overlap rate between pairs of agent predictions. The joint model produces internally consistent predictions with lower inter-prediction overlap rates.
While the joint mp-only model performs better on the interactive prediction task than the marginal model when evaluated with joint metrics, performs worse on the motion prediction task when evaluated with marginal metrics (see Table ID "ours (joint)"). This is because the joint model is trained to produce predictions that are internally consistent between agents, while the marginal model does not, which is strictly harder task. In this section, we examine this quality difference and the internal consistency of the predictions from both models.
In the WOMD dataset, each example has a different number of agents to be predicted. By slicing the marginal minADE results based on the number of agents to be predicted (Figure left), we find that the joint model performs worse there are more predicted agents, while the marginal model performs the same. This is expected: when there are more agents, the joint model has a more difficult task since needs to produce internally consistent predictions. One may expect joint model need an exponential number of trajectory combinations to perform competitively with the marginal model, but we are encouraged to find that this is not actually the case. We believe this is due to the fact that when many agents are interacting, the number of realistic scenarios is actually heavily constrained by interactions- most combinations of marginal predictions don't 't actually make sense together.
However, when more agents are to be predicted, the possibility of interactions are higher and we would expect that the joint model is able capture these interactions through internally consistent predictions. We measure the models' internal consistency by selecting the best trajectory and measuring the inter-pre mrer-prrdiction overlap rate (details below). We find that the joint model has consistently lower inter- predictiiit overlap rate, showing that it is able to capture the interactions between agents. The ability to model interactions enables the model to suitable for conditional motion prediction, and goal conditioned prediction tasks, which we discuss in Section 4.4
Measuring agent inter-prediction overlap. We measure predicted agent overlap on the best set of trajectory produced by the model. For the joint model, this corresponds to the joint prediction that has the higher probability score. For the marginal model, we take the top scoring trajectory for each agent to obtain the best set. For every predicted agent in the trajectory, we determine if it has an overlap with any other predicted agent by comparing the rotated upright 3D bounding boxes using predicted xyz and heading. The inter-predie prediction overlap rate is the number of predicted agents that are involved in some overlap, divided by the total number of predicted agents. We count two agents overlapping if the intersection over the union (loU) exceeds 0.01.
22

---

Published as a conference paper at ICLR 2022
C
SLICING RESULTS
20000
joint marginal
15000
4
Caunt
minADE
10000
3
5000
10
20
30
0
10
15
20
25
30
Mean AV Speed (m/s)
Mean AV Speed (m/s)
1.4
6000
1.3
Count
minADE
joint
4000
1.2
marginal
2000
1.1
0
2
8
# of Predicted Agents
of Predicted Agents
1.4
15000
1.3
Ca.rtt
minADE
1.2
joint
10000
marginal
1.1
5000
10
25
75 Dynamic Agents
50
100
40
60
80
# of Dynamic Agents
Figure 6: We show marginal minADE on a per scene basis broken down by different scene level statistics. Interestingly, we show that both the marginal and joint models become out of distribution above m/s, where there is minimal training data.
23

---

Published as a conference paper at ICLR 2022
Cumulative Probability
1.0
1.5
0.8
0.6
minADE
1.0
0.4
0.5
0.2
0.0
0.0
2
10 15
20
minADE
Mean AV Speed (m/s)
Figure Analysis of the multi-task model for goal-conditioned motion prediction. Left: Cumu- lative probability of the AV minADE in goal conditioned prediction (blue) and motion prediction (black) masking strategies. Right: AV minADE for goal conditioned prediction (blue) and motion prediction (black) function of AV speed (averaged over the ground truth trajectory).
24

---

Published as a conference paper at ICLR 2022
D LOss IMPLEMENTATION
The Scene Transformer model is both agent permutation equivariant and scene-centric. These properties allow to easily switch between marginal joint loss formulation (Figure 8}. In the marginal formulation, we reduce the loss of the best future for every agent separately; while in the joint formulation, we reduce the loss of the best future for all agents jointly. In practice, this determines when the reduce min operation is performed.
Our final loss composes the regression loss on the trajectory and the classification loss of the best trajectory. A weighted linear combination of the loss terms is used to combine these two losses together. The classification loss weight was set to 0.1, while the regression losses have weight 1.0. The weights were determined using the hold-out validation set.
agent_predicction and agent_ gt are (F, A, I, 71 Tensors with [x, Y, 21, uncertainty terms, and yav.
Get the KL Divergence of the predictions ve ground truth. LapĪaceKb method from (Meyer Thakurdesai, 2020)
loss pareeeeettteeeteeedectttttt
agent_gt)
Now reduce across all timesteps and values to produce tensor shape [E, A] loss tf.uce sum (loss, axis= 3})
The marginal loss, we only apply the loss to the best trajectory per agent (so min across the future dimension). marginal_lcas
wwr.uuceeeiinlssse axis-0)
Then sum over the agent dimension marginal Loss = rt.uce sum (marginal_ loss)
The joint loss, we sum over all agents to get a loss value per future. joint_loss tf. reduce__um (loss, axis-1)
Then only apply the loss the best future prediction joint_loss tÊ. reduce_miin (joint_lcss
Figure 8: Pseudo-code in TensorFlow Abadi et al. 2015) demonstrating the joint versus marginal loss formulation.
25


Reviews:
[{"review_id": "x2O5Uv5pnxx", "reviewer": "Reviewer_ADF1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n1. The scene transformer has its flexibility to switch between different prediction tasks by using different masking strategies.\n2. The model has proved to generate competitive results on both marginal and joint predictions compared to recent SOT algorithms.\n\nWeakness:\nSince model flexibility using the mask is the highlighted benefit of this model, I was expecting more experiments and discussion for the other prediction modes, such as conditional motion prediction and goal conditioned prediction. They are more interesting part of the paper that may lead a new direction for prediction model. However, the current result are still too preliminary. It would be better to provide more quantitative results to justify the advantage of such a unified model on CMP or GCP.\n\nSeveral design choices on components are described, such as factorized self attention and cross attention. An ablation study to summarize these findings would be better to review for the readers.", "summary_of_the_paper": "The paper presents a unified architecture to predict future motion trajectories of all agents in the scene, while interactions between agents are captured. A Transformer based architecture is designed. It also uses a masking strategy as a query, enabling one to condition on hypothetical agent futures during the inference time, and achieves conditional motion prediction or goal conditioned prediction. The model was tested on two standard datasets, and achieves state of the art performance.", "summary_of_the_review": "Overall this is a strong paper with sufficient novelty in its model architecture in the context of motion prediction for autonomous driving. The experiment results on standard prediction tasks on popular benchmarks are convincing. Despite some minor issues in presentation, the paper is above the acceptance bar.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "vrMZ9ERsuy1", "reviewer": "Reviewer_RDXN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "--- Strengths\n\n- This is a very thought-provoking paper with many interesting novel ideas. It uses a transformer architecture to translate the input [A, T, D] tensor into the prediction output [F, A, T, 7] tensor. With this design, it can perform motion prediction, conditioned motion prediction, and goal-conditioned motion prediction all in the same model. It also proposes a factorized self-attention approach to reduce the computational cost.\n\n- The result is quite impressive. It achieves state-of-the-art performance on both Argoverse and Waymo Open Motion Dataset (WOMD).\n\n\n--- Issues and suggestions\n\n- This paper proposes a method to model the joint distribution of the trajectories of all agents in the scene, and this is listed as one of its three contributions. However, I don't think this is a significant contribution of this paper because 1) the idea is fairly straightforward and 2) the performance is questionable. The difference between training a marginal model vs. a joint model is the loss function. The loss of a marginal model is sum_A(min_F(sum_T(error))), while the loss of a joint model is min_F(sum_A(sum_T(error))). This loss function design is actually fairly straightforward, and every model that predicts trajectories of all agents in the scene can do that. What makes joint trajectory prediction hard is not this loss function design but the challenge of scaling with the number of agents in the scene. A crowded scene can easily have up to 100 agents, and simply modeling the joint prediction this way won't work in practice because it requires F to be very large. I expected this paper to tackle this challenge, but I don't see it in the paper. The WOMD Interaction Prediction task only requires the model to predict the joint prediction of two agents, so this simple approach works there, but I just don't see how this approach can work in an actual autonomous driving software stack. Also, in Figure 4 in the appendix, the prediction error of the joint model looks terrible even with a few (e.g., four) agents.\n\n- The appendix mentions that the model was trained with some data augmentation. So it's not very clear to me how much of the performance improvement comes from the Scene Transformer design and how much of it comes from data augmentation. It will be useful to also show the performance numbers without data augmentation.\n\n- It would be helpful to show some inference latency numbers for people to understand whether the proposed model can be used in practice. Also, this paper mentions using the factorized self-attention to save computation, so it will be helpful to show some numbers on that as well.\n\n- A few statements in this paper are pretty confusing and hard to understand. I believe there is some room for writing improvements there.  I will give a few examples below.\n\n- I don't understand why \"two agents of the same type hidden at the same future location will have identical input representations\". Don't they already have different past positions unless they are on top of each other?\n\n- The paper argues that it uses a scene-centric representation while the prediction space is agent-centric, but I find a lot of statements around this to be quite confusing. For example, in Section 2, it says, \"Some models do a majority of modeling in a global, scene-level coordinate frame, such as work that employs a rasterized top-down image\". However, I believe most of the rasterization-based prediction methods are actually agent-centric. They center the raster around the agent-of-interest (e.g., in Cui et al., 2019). Actually, there are a lot more works than just LaneGCN that predict trajectories in agent-centric frame from scene-centric features, such as MultiXNet (Djuric el al. 2020) and SPAGNN (Casas el al. 2020). In Section 3.1, the sentence \"We use a scene-centric embedding where we use an agent of interest’s position as the origin, and encode all roadgraph and agents with respect to it.\" is quite confusing. People will wonder why it is not agent-centric when you use the agent of interest as the origin. I think what the authors are trying to say is that they use the ego vehicle as the origin in Waymo Open Motion Dataset (WOMD), so it's scene-centric. But I think for Argovese dataset, it is still agent-centric. On the WOMD dataset, where the ego vehicle is the origin, do you predict trajectory waypoints of an agent with respect to the ego vehicle or the agent? I think the trajectory waypoints should also be with respect to the ego vehicle if you want to model agent-agent interactions, but this seems to contradict the statement of \"agent-centric representations\". This part is very unclear in the paper, and please clarify this.\n\n- The paper says the 7 output values of each waypoint are x/y position and uncertainty in 3D space and heading. But if we want to model the output waypoints in 3D space, shouldn't the heading be 3D yaw/pitch/roll heading? The appendix says the losses and uncertainties are defined in lateral and longitudinal coordinates, and this will only make sense if it's a 2D space. This really makes me wonder whether the output space is 3D or 2D. Also, I don't think it's a good idea to represent the heading with just one value, because, for example, when the model predicts a 179 deg heading while the ground-truth heading is -179 deg, it will get a very high loss. I doubt how this can work.\n\n- In the appendix, the paper says, \"We append on a one-hot encoding to the final dimension where a 1 indicates which of the F futures is the ground truth trajectory resulting in a tensor of shape [F, A, T, D + F].\" This can only be done at training time, and I don't get how we can get the [F, A, T, D + F] tensor at inference time.\n", "summary_of_the_paper": "This paper proposes a transformer-style architecture, Scene Transformer, for joint trajectory prediction of multiple agents (including the autonomous vehicle). The key representation of Scene Transformer is a 3D tensor [A, T, D], with the dimensions being agent, time step, and feature dimension, and we mask out the entries that to be predicted. The output is [F, A, T, 7], where F is the number of future predictions. The model outputs 7 values for each waypoint, x/y position and uncertainty in 3D space and heading. The model uses factorized self-attention applied on the A axis and T axis separately to transform the [A, T, D] representation tensor, and uses a cross attention to fuse in the static map features. The model can be trained to predict either marginal predictions or joint predictions depending on the loss function. Also, based on what entries in the input tensor are masked out, Scene Transformer can perform motion prediction, conditioned motion prediction, and goal-conditioned motion prediction all in the same model.\n\nThe Scene Transformer model was evaluated on Argoverse and Waymo Open Motion Dataset (WOMD), and its marginal prediction achieves the state-of-the-art performance on both datasets. The evaluation also shows that the model is able to perform joint prediction for the WOMD Interaction Prediction task and achieves better performance on the scene-level metrics compared to the baselines. The evaluation also shows that the Scene Transformer model is able to do conditional motion prediction and goal-conditioned motion prediction.", "summary_of_the_review": "Overall, I think this is a very thought-provoking paper with many interesting novel ideas and impressive results. The performance of the marginal prediction is a bit questionable, and there is also room for some writing improvement. I recommend this paper to be accepted, but I hope the authors can clarify my questions in the final version of the paper.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "qnTkz_TbCj7", "reviewer": "Reviewer_LaWD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Section: Abstract, Introduction: Issue with novelty. The manuscript \"[combines] a scene-centric approach, agent permutation equivariant model, and a sequence masking strategy...\". The first element (specifically, understanding the agent-to-scene interactions, through various attention mechanisms) has been considered, ad nauseum, by several trajectory forecasting works — in the last year alone. The second element is an inherent property of the chosen model class (that remains unsupported/uncited in the text, by the way) and, thus, cannot be used to strengthen the manuscript's contribution. The final element is a well-known practice related, again, to the chosen model architectural class: while it is true that the last element (sequence masking) has not been explored sufficiently in the context of trajectory forecasting (despite the already broad usage, now, of transformer-based architectures), I am hard-pressed to regard this as a strong standalone contribution to the community.\n\nSection: Introduction: The manuscript claims a \"unified architecture for prediction and planning\". I assert that this not appropriate. The manuscript itself cites Liu et al., (2021a) in reference to the goal-conditioned prediction (GCP) sub-task; however, *that* work is actually performs planning: they actually optimize with respect to a cost formulation, their model includes an inductive bias to learn the transition dynamics of the environment (online), and they evaluate their work in CARLA (an urban driving simulator, for evaluation of online planning and control). In this work, planning-related experiments are not shown: it is another conditional trajectory prediction approach with a transformer-based architecture. I would suggest that this manuscript removes all references to the term 'planning' and claims of the two prediction/planning paradigms being 'unified'.\n\nSection 2: The Related Works section is well-written, save for a couple issues. (1) In most places, the manuscript summarizes the related works and neglects to explain the differences between those works and the present manuscript, let alone explain why the differences are significant. For example, the manuscript states that \"LaneGCN (Liang et al., 2020) is agent-centric yet representations are in a global frame – ... the only other work to do so.\" The manuscript should explain, then, the notable differences that makes the present work uniquely more performant. (2) some concepts and properties are left undefined or unsupported. For example, the manuscript states that \"... models that reason in the agent-coordinate frame (Mercat et al., 2020; Zhao et al., 2020; Khandelwal et al., 2020) are intrinsically pose-invariant, but scale linearly with the number of agents, or quadratically with the number of pairwise interactions between agents.\" Through what mechanism does the present manuscript claim pose-invariance, and what scaling order does the approach in the present manuscript enjoy?\n\nSection 2: \"Representing multi-agent futures\": The manuscript states that \"A common way to represent agent futures is via a weighted set of trajectories per agent... We argue in this work that modeling joint futures in a multi-agent environment (Figure 1, right) is an important concept that has been minimally explored in prior work.\" One could argue that the implicit weighting for those weighted set of trajectories (for each agent) is a function of other agents’ historical trajectories and the scene context; and, in the case of flow-based approaches, the weighting is also conditioned on a prior density over diverse and admissible trajectories (see Park et al., 2020). Is the manuscript asserting that the implicit weighting, then, does not capture as effectively the same multi-agent/agent-scene interactions as that which is proposed? More discussion is needed to understand why.\n\nSection 3.2: The manuscript attempts to claim as its main novelty the use of masked self-attention for joint prediction in trajectory forecasting problems. If the manuscript wishes to do this, then this inductive bias should be compared with those of other model classes, such as normalizing flow- and CVAE-based approaches. This would provide the community with insights as to why we might choose one over the other; and this work could then claim as a contribution the investigation of these differences, just as [1] did in their discussion of flow (see Figure 9 for comparisons w.r.t RNN-, CVAE-, and GAIL-based configurations).\n\nSection 3.2: \"Cross-attention\": Not novel. The paper cannot both claim primary novelty from its attention mechanisms and simultaneously ignore similar attention mechanisms as components in other works, as applied to the same tasks. The manuscript should discuss why this \"cross-attention\" mechanism is better than the plethora of other agent-to-scene attention mechanisms, which have been popular in robot navigation, planning, and trajectory forecasting.\n\nSection 3.2: The manuscript says \"Finally, in order to capture both time and agent dependencies, the model simply alternates attention across agents and time in subsequent layers (Figure 2, right panel).\" Why interleave? Why not calculate both attention distributions — across both agent and time dimensions — serially?\n\nSection 3.4: The referenced 'Figure 7' appears to be missing.\n\nSection 4.4: The manuscript asserts that \"...the multi-task model matches the performance of our MP-only trained joint model on both joint (Table 3) and marginal metrics (see also Appendix, Table 10), indicating that the addition of tasks does not degrade standard motion prediction performance.\" This claim is not really apparent from the provided results: throughout the tables, the performance of the multi-task models are always subpar or inconclusive, especially since no standard error calculations are given to contextualize the numerical results.\n\nSections 4, B: Results of the various ablations are very sensitive to the model instance's training objective (marginal, joint, multi-task) and the trajectory lengths (t={3, 5, 8}). To me, this renders such claims as \"we [produce] consistent futures that account for interactions between agents\" (Abstract, Intro) somewhat inconclusive, still.\n\nSection 4: The manuscript states: \"Although no quantitative benchmarks are available in the community for quantifying GCP predictions, we take these results as positive indication that the model is responding appropriately and save further exploration of counterfactual analysis for future work.\" Consider evaluation in online urban-driving settings, like the CARLA benchmarks: Original, NoCrash, Anyweather, CARNOVEL [2], or CARLA Challenge. See also Liu et al., (2021a), which this manuscript even cites.\n\n----\n\n[1] Rhinehart, Nicholas, Kris M. Kitani, and Paul Vernaza. \"R2p2: A reparameterized pushforward policy for diverse, precise generative path forecasting.\" In Proceedings of the European Conference on Computer Vision (ECCV), pp. 772-788. 2018.\n\n[2] Filos, Angelos, Panagiotis Tigkas, Rowan McAllister, Nicholas Rhinehart, Sergey Levine, and Yarin Gal. \"Can autonomous vehicles identify, recover from, and adapt to distribution shifts?.\" In International Conference on Machine Learning, pp. 3145-3153. PMLR, 2020.", "summary_of_the_paper": "The manuscript proposes the use of a factorized attention mechanism and a masked transformer-based architecture, for conditional trajectory forecasting. Experiments on Argoverse and the Waymo Open Motion Dataset are provided.", "summary_of_the_review": "I have some issues with the novelty of the work presented, the accuracy of the claims made, and the somewhat lacking comparison with the related work and key ablation experiments. See main review, above. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "M5POLc_Csik", "reviewer": "Reviewer_evQV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n1. Paper is well written and clear.\n2. The problem statement is justified and relevant in autonomous driving.\n3. Results are good.\n4. I like the flexibility that the model offers in terms of use cases.\n\n**Weaknesses:**\n\nWhile the problem statement is important, the technical novelty and the analysis is lacking. The autonomous driving community has started to move beyond rudimentary metrics like ADE/FDE to include more analysis on “interesting behaviors”. For instance, see [0]. Waymo, in fact, even has included such labels in their Open Motion dataset for more challenging scenarios (See Section 3.1 in https://arxiv.org/pdf/2104.10133.pdf). \n\nFrom a research contribution perspective, it is not enough to simply rack up higher numbers on the leaderboard. There needs to be analysis that advances the field forward. The future of the field lies in challenging scenarios like intersections, unprotected left turns etc. The proposed work currently does not present any such analysis. Therefore, in its present form, the current work in my opinion is not of high technical quality and novelty enough to be published in ICLR.\n\n**What I would like to see in a revised version:**\n\nTo improve its novelty this paper needs to include a section in Section 4 on analysis of the proposed method on more challenging and interesting scenarios in the WOMD. Such a section should include:\n\n1. ADE/FDE results for the scenarios that WOMD labels as challenging or interesting.\n2. Explanations for where the model particularly is successful and also the cases where the model fails (if so).\n3. Furthermore, there is past research on this [1,2,3,4]. The authors should compare and contrast the technical details with these methods in this section.\n\n[0] Ivanovic, Boris, and Marco Pavone. \"Rethinking trajectory forecasting evaluation.\" arXiv preprint arXiv:2107.10297 (2021).\n\n[1]  Deo, Nachiket, and Mohan M. Trivedi. \"Trajectory forecasts in unknown environments conditioned on grid-based plans.\" arXiv preprint arXiv:2001.00735 (2020).\n\n[2] Bhattacharyya, Apratim, Mario Fritz, and Bernt Schiele. \"Long-term on-board prediction of people in traffic scenes under uncertainty.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.\n\n[3] Chandra, Rohan, et al. \"Traphic: Trajectory prediction in dense and heterogeneous traffic using weighted interactions.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n\n[4] Chandra, Rohan, et al. \"Forecasting trajectory and behavior of road-agents using spectral clustering in graph-lstms.\" IEEE Robotics and Automation Letters 5.3 (2020): 4882-4890.\n", "summary_of_the_paper": "The paper proposes a new Transformer-based trajectory forecasting model that can predict multiple agents in a scene. It can be used for goal-directed trajectory forecasting as well. Experiments on Argoverse and Waymo with good results. Related work is sufficient. Technical novelty is low. Paper is clearly written.", "summary_of_the_review": "Without the required analysis as requested in the revised version, the present work lacks research rigor, technical novelty, and is simply an impressive engineering effort. If no revision is made, I would defer it to a workshop publication.\n\nIf authors provide **strong** analysis on challenging, rare, and interesting examples as provided in WOMD, I may recommend acceptance (pending additional review of the provided analysis).\n\n============= POST AUTHOR-DISCUSSION =============================\n\nThe authors have made good faith attempt in addressing my concern and engaging in discussion. \n\nWhile I wouldn't fight for it's acceptance, I have no problems if it is published in ICLR. I have updated my score accordingly. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper shows interesting and discussion inspiring results on multi-agent trajectory prediction, as needed, for instance, in autonomous driving. Among the key technical ideas is a “conditional scene transformer” approach for flexible predictions for different agents.  Results on two  public benchmarks are impressive. Some reviewers are a bit torn about the significance of the technical contributions and the analyses of the results. Nevertheless, on average, the reviewers vote the paper to be above the acceptance threshold.
Decision: Accept (Poster)