Real-Time Motion Prediction via Heterogeneous Polyline Transformer with Relative Pose Encoding
Authors: Zhejun Zhang
Alexander Liniger
Christos Sakaridis
Fisher Yu
Luc Van Gool
Venue: NeurIPS 2023 poster (2023)
Keywords: Motion Prediction
Autonomous Driving
Transformer
TL;DR: We use hierarchical Transformers with pairwise-relative representation to realize accurate and efficient motion prediction in autonomous driving.

Abstract: The real-world deployment of an autonomous driving system requires its components to run on-board and in real-time, including the motion prediction module that predicts the future trajectories of surrounding traffic participants. Existing agent-centric methods have demonstrated outstanding performance on public benchmarks. However, they suffer from high computational overhead and poor scalability as the number of agents to be predicted increases. To address this problem, we introduce the K-nearest neighbor attention with relative pose encoding (KNARPE), a novel attention mechanism allowing the pairwise-relative representation to be used by Transformers. Then, based on KNARPE we present the Heterogeneous Polyline Transformer with Relative pose encoding (HPTR), a hierarchical framework enabling asynchronous token update during the online inference. By sharing contexts among agents and reusing the unchanged contexts, our approach is as efficient as scene-centric methods, while performing on par with state-of-the-art agent-centric methods. Experiments on Waymo and Argoverse-2 datasets show that HPTR achieves superior performance among end-to-end methods that do not apply expensive post-processing or model ensembling. The code is available at https://github.com/zhejz/HPTR.

Real-Time Motion Prediction via Heterogeneous Polyline Transformer with Relative Pose Encoding
Zhejun Zhang Computer Vision Lab
Alexander Liniger Computer Vision Lab ETH Zurich
ETH Zurich
Zurich, Switzerland
Zurich, Switzerland alex. wwwiieerriiiiinn...........ee..e www.ioonneeeeh........
zhejun www.zaaiiiviiiii...... ch
Christos Sakaridis
Fisher Yu
Lue Van Gool CVL, ETH Zurich, CH PSI. KU Leuven, BE INSAIT, Un. Sofia, BU vangool@vision. ee ethz. ch
Computer Vision Lab ETH Zurich
Computer Vision Lab ETH Zurich Zurich, Switzerland 10yf io
Zurich, Switzerland ceakarid@visionn ee. ethz.ch
Abstract
The real-world deployment of an autonomous driving system requires its compo- nents to run on-board and in real-time, including the motion prediction module that predicts the future trajectories of surrounding traffic participants. Existing agent- centric methods have demonstrated outstanding performance public benchmarks. However, they suffer from high computational overhead and poor scalability as the number of agents to be predicted increases. To address this problem, we in- troduce the K-nearest neighbor attention with relative pose encoding (KNARPE), novel attention mechanism allowing the sarwise-reeative representation be used by Transformers. Then, based on KNARPE we present the Heterogencous Polyline Transformer with Relative pose encoding (HPTR), hierarchical frame- work enabling asynchronous token update during the online inference. By sharing contexts among agents and reusing the unchanged contexts, efficient as scene-centric methods, while performing on par with state-of-thh--ar agent-centric methods. Experiments Waymo and Argoverse-2 datasets show that HPTR achieves superior performance among end-to-end methods that do not apply expensive post-processing or model ensembling. The code is available thub. ctw/zhejz/////ccc/eoo http////Rz///HT.....
approach is
https: //githbb/..........//
Introduction
Motion prediction is an important component of modular autonomous driving stack [63]. As the downstream module of perception [22, 42, 48] and the upstream module of planning [3, 39], the task of motion prediction [16, 60) is to predict the multi-modal future trajectories of other agents next the self-driving vehicle (SDV) based the heterogeneous observations, including for example high-definition (HD) maps, traffic lights and other vehicles, pedestrians and cyclists. This is an essential task because without accurate and real-time prediction results [30], the planning module cannot safely and comfortably navigate the SDV through highly interactive driving environments.
To achieve top performance public motion prediction leaderboards [1, 58), state-of-the-ar (SOTA) methods [36, 47, 52, 55] leverage agent-centric vectorized representations and Transformer based network architectures. However, the good performance of these approaches comes at cost of high computational overhead as illustrated Figure 1. The most well-known problem of agent-centric
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(a) Dense traffic scenario
(b) Agent-centric ROIs
(c) Online inference
(d) Trajectory aggregation
Figure 1: To efficiently predict the multi-modal future of numerous agents in dense traffic (la). HPTR minimizes the computational overhead by: (1b) Sharing contexts among target agents. (1c) Reusing
static contexts during online inference. (1d) Avoiding expensive post-processing and ensembling.
approaches is the poor scalability as the number of target agents grows in urban driving environments with dense traffic; for example the busy intersection in Figure la. Although the agent-centric regions of interest (ROI) in Figure lb are largely overlapping, the same context is transformed to the coordinate system of each target agent and independently saved and processed. This causes huge waste of computational resources, which is often neglected by prior works because their experiments focus on offline inference that queries the prediction only once given scenario from the dataset. In contrast to offline inference, the motion prediction module real-world SDV is continuously queried with streaming inputs during online inference. For example, in Figure Ic, the prediction module is queried again shortly after Figure la. Although most contexts remain unchanged after this short period of time, existing methods would start inference from scratch without reusing the encoded features of static contexts. Moreover, prior works often predict a massive number of redundant trajectories and ensemble the outputs of numerous models, such that the final output can be adjusted in favor of prediction diversity during the post-processing, as illustrated in Figure 1d. Although these techniques significantly boost the performance on public benchmarks, they should be sparingly used on real-worts SDV where the computational resources are scarce and the raw predictions, rather than the heuristically aggregated ones, are preferred by the downstream planning module.
To address these problems, our first step is to represent everything as heterogeneous polylines [17]. Then we adopt the pairwise-relative representation [11, 28] and separate the local attribute from the global pose of each polyline. The local attribute specifies what the polyline actually is and will be shared or reused if possible. The global pose specifies where the polyline and it will be used to derive the pairwise-relatiee pose before being processed by the neural networks. Considering polylines as tokens, our second step introduce the K-nearest Neighbor Attention with Relative Pose Encoding (KNARPE) module, which allows Transformers [53] to aggregate the local contexts for each token via the local attributes and relative poses. While KNARPE enables context sharing among agents, we further propose the Heterogeneous Polyline Transformer with Relative pose encoding (HPTR), which emphasizes the heterogeneous nature of polylines in motion prediction tasks, Based on KNARPE, HPTR uses hierarchical Transformer encoders and decoders to separate the intra-class and inter-class attention, such that tokens can be updated asynchronously during online inference. More specifically, for static polylines, such as HD maps, their features will be reused, whereas for dynamic polylines, such as traffic lights and agents, their features will be updated on demand.
Our contributions are summarized as follows: (1) We introduce KNARPE, novel attention mechanism that enables the pairwise-relative representation be used by Transformer- Based on KNARPE we propose HPTR, a hierarchical framework that minimizes the computational overhead via context sharing among agents and asynchronous token update during online inference. (3) Compared to SOTA agent-centric methods, we achieve similar performance while reducing the memory consumption and the inference latency by 80%. By caching the static map features during online inference, HPTR can generate predictions for 64 agents in real time at 40 frames per second. Experiments on Waymo and Argoverse-2 datasets show that our approach compares favorably against other end-to-end methods which do not apply expensive processinn and ensembling.
architectures. (2)

---

2 Related work
Motion prediction is a popular research topic because it is an essential component of modular autonomous driving stacks [63]. The task of motion prediction has been formulated in different ways. In this paper, we focus on the most popular one: the marginal motion prediction [6, 8, 22, 33] where the multi-modal future trajectories are predicted individually for each target agent [1, 38, 58]. In contrast to the marginal formulation, joint motion prediction [19, 20, 35, 49] requires the multi- modal futures to be simultaneously generated for all target agents from the same scenario [2. 57, 66]. Beyond the open-loop motion prediction tasks, behavior simulation [44, 50, 68] is formulated in closed-loop fashion where the future agent trajectories policy [3, 59]. Numerous works have also investigated the possibility to combine motion prediction with other modules, such as perception and planning [7, 15, 23, 29], to learn end-to-end driving policy [24, 25, 67] mapping sensor measurements directly to the actions or motion plans of the SDV,
simulated by rolling out learned
Vectorized representation proposed by VectorNet [17] is widely used in recent works because of its promising performance [47, 52, 55J. Depending on how the coordinate system is selected, the vectorized representation falls into three categories: agent-centric, scene-centric and pairwise-relative. The most popular one is the agent-centric representation [36, 47, 52, 55] that transforms all inputs to the local coordinates of each target agent. Despite its good performance, this approach suffers from poor scalability as the number of target agents grows. To reduce the computational overhead, scene-centric representation [37] shares the contexts among all target agents by transforming all inputs to a global coordinate system, which is not tied to any specific agent but to the whole scene. However, its performance is poor due the lack of rotation and translation invariance. In this paper we use the pairwise-relative representation, which is less often discussed in prior works because it has not demonstrated any clear advantage. To the best of our knowledge, there are three works using the pairwise-relative representation: HDGT [28]. GoRela [11] and HiVT [70]. HDGT and GoRela are based on Graph Neural Networks (GNNs), and their implementation requires message passing and GNN librarics. However, these libraries are often less efficiently implemented on Graphics Processing Units (GPUs) when compared to the basic matrix operations that Transformers rely on. HiVT augments the agent-centric encoders with a pairwise-relative interaction decoder in order to realize multi-agent prediction. In contrast to HiVT which uses agent-centric vectors and the standard Transformer, we formulate all inputs as relative polylines and introduce the KNARPE attention mechanism. As a result, our HPTR demonstrates clear advantages in terms of accuracy and efficiency. Replacing the vanilla attention with our KNARPE, we can borrow ideas from other Transformer-based methods and adapt them to the pairwise-relative representation. For example, the hierarchical architecture of our HPTR is inspired by Wayformer [36]. More sophisticated architectures and techniques, such as goal-based decoding [21, 69I, can also be incorporated into our framework to further boost the performance.
Rasterized representation was widely used in early works on motion prediction [34, 40, 45, 51]. These methods use convolutional neural networks (CNNs) to process the rasterized image of agent- centric ROI. To improve the inference efficiency, some works [5, 6, 10, 65] pre-compute the static map features and use rotated ROI alignment [26] to retrieve the local contexts around the target agent. In this paper, our KNARPE realizes this operation for pure Transformer-basee architectures.
Transformers with attention mechanism [53] have achieved great success in natural language processing [13, 43] and computer vision tasks [4, 14, 64]. Inspired by the vision Transformers, we treat polylines as high-dimensional pixels; the local attribute corresponds to color channels, whereas the global pose corresponds to the pixel-wise location. We further extend the relative position encoding [12, 46] to higher dimensions and rename it to relative pose encoding (RPE) in this paper. Although the benefit of RPE is still controversial for other tasks [54, 61]. its value for motion prediction is demonstrated in this paper. Instead of using RPE, previous motion prediction Transformers use everything as input, which is equivalent to concatenating the pixel-wise location to the RGB channels. This is a very uncommon practice from the perspective of vision Transformers.
3 Method
In Sec. 3.1 we introduce the pairwise-relative polyline representation which enjoys the advantages of both the agent-centric and the scene-centric representation. Then See. 3.2 we present the K-nearest Neighbor Attention with Relative Pose Encoding (KNARPE) which enables the pairwise-
3

---

MP
Polyline
Where is it?
What is it?
(P2 U2)
TL
Normalize
Local Attribute u starts at x
PP un)
(P3. Uz)
w.r.t.
AG
A
r02
Global Pose
y
r03
Fo1
(2, 0,
Relative
(po Un)
Pose
(a) Polylines.
(b) Local attribute represented in the global posc's frame.
(c) Obtain relative poses.
Figure 2: Pairwise-relative polyline representation. MP: Map. TL: Traffic lights. AG: Agents.
relative polyline representation to be used by Transformers. Based on KNARPE, we propose the Heterogeneous Polyline Transformer with Relative pose encoding (HPTR) in Sec. 3.3. HPTR uses hierarchical architecture to prevent redundant computations and to realize the asynchronous update of heterogeneous tokens. Finally in Sec. 3.4 we discuss our output representation and training strategies.
3.1
Pairwise-relative polyline representation
Based on prior works [11, 28], we formulate the pairwise-relative polyline representation as the third type of input representation for motion prediction. As shown in Figure 2a, all data relevant to motion prediction can be represented as an ordered list of consecutive vectors, i.e. polylines. By the coordinate frame of its global pose as done in Figure 26, polyline is
transforming the polyline fully described by a pair of global pose and local attribute. 'he global pose specifies the location and heading of the polyline in the global coordinate system, whereas the local attribute describes the polyline in its local coordinates, for example it is yellow solid lane, 8 meters long, slightly curved to the right. In practice, it does not matter where the lane is on the earth; we only care about Hence, we obtain the relative poses from the global poses as shown
where the lane is relative in Figure 2c, before feeding them to the prediction network. Specifically, the global pose and local attribute of polyline i are denoted (Ps. u,). The global pose p: has degrees of freedom (I, y. 0), ie. the 2D position and the heading yaw angle. The local attribute u, is derived from C₁ the intrinsic characteristics of the polyline, and 1,, the polyline vectors represented in the local coordinate.
The task of marginal motion prediction is to predict the future 2D positions individually for each target agent based on the static HD map (MP), the history trajectories of all agents (AG) and the traffic lights (TL). For each scenario to be predicted, we consider a maximum number of NMP map polylines, NTL traffic lights and NAG agents. We define t to be the current step, {Th 1,.... 0} to be the observed history steps and {I,.... Ti} to be the predicted future steps. The static HD map are spatial polylines (D IMP
c cMP),, E (1, (1,..., ) where MM" E R3 is its starting vector, IM1 E Mm are the 2D positions and directions of the Nusde segments normalized against PMP. and c.MP E RMP the one-hot encoding of CMP different lane types. Polygonal elements, such as crosswalks, are converted to group of parallel polylines across the polygon. Similar to the map, history trajectories of agents are represented as spatial-temporal polylines (pAG 1AG, . A, E (1,..., NAG) where DA IR is the last observed agent pose, 1 E IK (6++3) contains the history 2D positions, directions and velocities normalized against p. as well as the ID speed, yaw rate and acceleration which do not need be normalized, and eº E 1 the 3D agent size and the one-hot encoding of 3 agent types (vehicle, pedestrian and cyclist). Following VectorNet [17], we use PointNet [41] with masked max-pooling to aggregate (1,40, cMP) into UM R and (1A6 ( into wº R where D is the hidden dimension. Since current datasets contain only the detection results rather than the tracking results of traffic lights, we consider only the traffic lights observed t 0 and represent them as singular polylines (p!T ). E (I,..., NNr where p!! R is the pose of the stop point and TL e CC... is the one-hot encoding of On. different states of traffic lights. We use multi-layer perceptron (MLP) to encode c. into u. E R.
Because the local attributes are normalized and the global poses are used to compute the relative poses before being consumed by the network, the pairwise-relative representation preserves the viewpoint invariance of the agent-centric representation. Additionally, since the local attributes have higher dimensions compared to the 3-dimensional global poses, sharing the local attributes enables the pairwise-relative representation maintain the good scalability of the scene-centric representation.

---

However, so far this representation has only be exploited by GNNs, which are not comparable to Transformers terms of accuracy and efficiency on the motion prediction benchmarks [l. 58].
3.2 KNARPE: K-nearest neighbors attention with relative pose encoding
After encoding the local attributes via the polyline-level encoders, the scenario is now described as (pi, u:),i E {1, ................................ ] where N NMM NNGG + NTr. is the total number of polylines. However, this representation cannot be handled by standard self-attention because on the one hand modeling the all-to-all attention prohibitively expensive [36] due to the large N, and on the other hand the performance deteriorates when the input is scene-centric [37]. To address both problems, we introduce the K-nearest Neighbors Attention with Relative Pose Encoding (KNARPE). Similar to MTR [47], KNARPE limits the attention to the K-nearest neighbors of each token. Specifically, R.K (d:1,-.., dix) S [I....,N] .N a set that contains the indices of K tokens closest to token i. For simplicity, we use the L2 distance to measure the distance dy between the global poses of token and j. Instead of directly processing global poses, KNARPE uses the relative pose encoding (RPE), i.e. the positional encoding for pairwise-relative poses. Denoting rij (xij. Mij. 0.j) to be the global pose of token j represented the coordinate token i.e. P, transformed to the coordinate p., the RPE of rs is computed using sinusoidal positional encoding (PE) and angular encoding (AE) [68],
RPE(r:)) ) = concat concat(PE(z.) (PE().))
AB((,)
PEz (4) = sin(x w 3) PE2++1 (x) cos w
AB2; (0) = sin (0 (i (++1) 1)), (E2++1 = cos (0 (i 1)), E -
D/2 1},
where w is the base frequency. Following [46], the RPE is projected and added to the keys and values to obtain 2,, the output of letting token i attend to its K neighbors nK.
2- KNARPE (u,, UJ, Fay 1 (jEN))
Σ O4
W (u, b" b"+RRE(TT)) W + bb))
(4)
JEN
exp(er)) 0i EKERK exp(ex))*
Cij ez== (u W + bu W + b* +R-RE)(T W +6*)
(5)
VD
where Ciy are the attention weights, Eiy are the logits, Wlgkko) bla.k.)) are the learnable projection matrices and biases for query, key and value, and (k(k.,), ((k.e) are the learnable projection matrices and biases for RPE. We do not apply RPE to query because doing this does not boost the performance in our experiments.
Efficient implementation of KNARPE can be achieved using basic matrix operations such as matrix indexing, summation and element-wise multiplication. See the appendix for more details. KNARPE allows the pairwise-relative representation to be used by pure Transformer-based architectures. Self-attention with KNARPE aggregates the local context for each token in the same way as CNN aggregates the context around each pixel via convolutional kernels. The pixel corresponds to the token, whereas the kernel size of a CNN corresponds to the number of neighbors of KNARPE. Cross- attention with KNARPE enables rotated ROI alignment of pre-computed features, e.g. the static map features. Previously, this was only possible for CNN- bbsed [6, 10] and GNN-based [11, 28] methods.
3.3 HPTR: Heterogeneous polyline transformer with relative pose encoding
By replacing the standard multi-head attention [53] with our KNARPF, we construct Transformer encoders and decoders which can model the interactions between heterogencous polylines via relative poses and local attributes. To address the marginal motion prediction task involving HD map, traffic lights and agents, we propose the Heterogeneous Polyline Transformer with Relative pose encoding (HPTR) as shown in Figure 3. Since the temporal dimension is eliminated by the polyline-level encoder [17], HPTR models only the spatial relationship between tokens from different classes.
Firstly in Figure 3a, the intra-class Transformer encoders build block diagonal attention matrix that models the interactions within each class of tokens. Then in Figure 3b, the inter-class Transformer decoders enhance the traffic lights tokens by making them attend to the map tokens, whereas the agent tokens are enhanced by attending to both the traffic lights and map tokens. The intuition behind this is the hierarchical nature of traffic; first there is only the map. then the traffic lights are added
5

---

MP
TL
AG
MP
TL
AG
MP
TL
AG
AG
Anchors
MP
Intra
MP
MP
MP
TL
Intra
Enhance-Ti TL
TL
All-o-All Self-Attention
TL
Cross-Altn
Self-Altn
All Tokens
AG
Intra
AG
Enhance-AG
AG
MP
AG
AG
(a) Intra-class TF encoder (b) Inter-class TF decoder
(c) All-to-all TF encoder
(d) AC-to-all TF decoder
Figure 3: The hierarchical architecture of HPTR. Transformers are applied in sequential order from left to right, top to down. Some attentions are redundant and can be skipped for better efficiency. We propose the lower triangular attention matrix, which excludes the intra-TL. intra-AG and all-to-all self-attentions (the transparent parts). By removing the redundant attentions, this specific lower triangular architecture enables asynchronous token update during online inference. TF: Transformer. Attn: Attention. MP: Map. TL: Traffic lights. AG: Agents.
and finally the agents join. Intuitively, the map influences the interpretation of traffic lights but not vice versa, whereas map and traffic lights together influence the behavior of agents but not vice versa. After the inter-class Transformer decoders, the all-to-all Transformer encoder in Figure 3c builds full attention matrix allowing tokens attend each other irrespective of their class. Finally, each agent token is concatenated with NAC learnable anchors, which are shared among each type of agent. Since the task is marginal motion prediction, we batch over agents and anchors, i.e. now the number of anchor tokens is NAG NAc- Then the anchor-to-all Transformer decoder in Figure 3d lets each of these anchor tokens attend to all tokens so as aggregate more contextual information. The final output is denoted as 2,6 i {1, NAG NAc). and it is used to generate the multi-modal future trajectories. We set the number of neighbors to K for the intra-class and all-to-all Transformers. This number K is multiplied by M.. 7AG and "ACC respectively for the enhance- TL, enhance-AG and AC-to-all Transformers, such that these Transformers can have a larger receptive field.
HPTR organizes the Transformers in a hierarchical way such that some of the intermediate results can be cached and reused during the online inference; for example the outputs of the intra-MP Transformer, iee the static map features. This allows tokens from different classes to be updated asynchronously. which significantly reduces the online inference latency. We can further improve the efficiency without sacrificing the performance by trimming the redundant attentions. The attention matrices shown in Figure 3a, 3b and 3c are overlapping, which means some relationships are repeatedly modeled, such as the agent-to-agent self-attention. We propose to remove the intra-TL, intra-AG and all-to-all Transformer, while keeping the intra-MP, enhance-TL and enhance-ACC AG Transformer. These three Transformers together build a lower triangular attention matrix which is necessary and sufficient to model the relationship between map, traffic lights and agents. Our approach can be seen extension Wayformer [36] which does not introduce the inter-class Transformer decoders. We can trim Figure 3 differently and cast HPTR into the Wayformer. Specifically. Wayformer with late fusion corresponds to HPTR with diagonal attention matrix (only intra-class TF). Wayformer with early fusion corresponds to HPTR with full attention matrix (only all-to-all TF) and Wayformer with hierarchical fusion corresponds HPTR with the diagonal followed by the full attention matrix.
3.4 Output representation and training strategies
We follow the common practice [36, 47, 52] represent the outputs as a mixture of Gaussians and train with hard assignment. The multi-modal future trajectories for each agent are generated by decoding 2,1G i E [1,.... AA ]NAc via two MLPs; the confidence head and the trajectory head. The confidence head predicts a scalar confidence of each trajectory. Besides the Gaussian parameters of 2D positions (izz Myy az. P) at each future time step, our trajectory head predicts also the yaw angles, speeds and 2D velocities. We use the cross entropy loss for confidences, negative log- likelihood loss for 2D positions, negative cosine loss for yaw angles and Huber loss for speeds and 2D velocities. The final training loss is the unweighted sum of all losses. Following the hard-assignment strategy, for each agent we optimize only the predicted trajectory that is closest to the ground truth in terms of 2D average displacement error. Please refer to the appendix for more details.

---

4 Experiments
4.1 Experimental setup
Benchmarks. We benchmark our method on the two most popular datasets: the Waymo Open Motion Dataset (WOMD) [16] and the Argoverse-2 motion forecasting dataset (AV2) [60]. Both datasets have an online leaderboard for marginal motion prediction. However, their task descriptions are slightly different. For each scenario, WOMD evaluates the predictions of up to 8 agents, whereas AV2 evaluates only one agent. Both datasets require exactly 6 futures be predicted for each target agent. The sampling time is 0.1 seconds for both datasets. The history length is 11 steps for WOMD and 50 steps for AV2. whereas the future length is 80 steps for WOMD and 60 steps for AV2. We use the official evaluation tool of the leaderboards to compute the metrics. For the WOMD leaderboard [58]. the ranking metric is soft mAP; for the AV2 leaderboard [1]. it is brier-minFDE. Please refer to the leaderboard homepages for more details about the dataset and evaluation metrics.
Implementation details. We use Transformer with pre-layer normalization [62] for HPTR. For each episode, we consider NMP 1024 map polylines, NT.. = 40 traffic light stop points and NAG = 64 agents. Each polyline contains up Nnode 20 one-meter-tong segments. The base number of neighbors considered by KNARPE is K = 36. This number is multiplied by Yn. = 2, TAG and T 10 respectively for the enhance-TL, enhance-AG and AC-to-all Transformer. We set NAC = 6 to predict exactly 6 futures as specified by the leaderboard. We do not apply ensembling or expensive post-processing such as trajectory aggregation. Our post-processing manipulates only the confidences. To improve the soft mAP, we use the mon-mamimum suppression of MPA [31] for the WOMD leaderboard. To improve the brier-minFDE, we set the softmax temperature to 0.5 for the AV2 leaderboard. More details are provided in the appendix.
Training details. Thanks to the viewpoint invariance of the sairwise-relative representation, no data augmentation or input permutation is needed for the training of HPTR. We use Adam W optimizer with an initial learning rate of le-4 and decaying by 0.5 every 25 epochs. We train with total batch size of 12 episodes on 4 RTX 2080Ti GPUs. For WOMD, we randomly sample 25% from all training episodes at each epoch; for AV2 we use 50%. Our final models are trained for 120 epochs for WOMD and 150 epochs for AV2. The complete training takes 10 days. For WOMD, the SDV agents, agents of interest and agents be predicted are used for optimization. For AV2, the SDV agents, scored agents and focal agents are used for optimization. To address the imbalance between agent types, for WOMD we additionally optimize for pedestrians and cyelists which are tracked for at least seconds.
4.2 Benchmark results
In Table I benchmark our approach on the public leaderboards of WOMD and AV2. From the leaderboard we observe that using ensemble and predicting redundant trajectories can increase the prediction diversity and hence improve the soft mAP, brier-minF-E and miss rate. For example, MTR-Ady-ens aggregates the outputs of 7 ablation models, each predicting 64 futures. Besides the exaggerated large number of redundant predictions, is also non-trivia to aggregate them into pre- dictions. Some works use K-means clustering while the others use non-maxinn maximmmximm suppression; both involve heuristic parameter fine-tuning, Since our framework dedicated real-world autonomous driving, applying these computationally expensive techniques is contradictory to our motivation. For a fair comparison, we focus on end-to-end methods which do not apply these techniques. On the WOMD dataset, we achieve SOTA performance among the end-to-end methods, including MTR-e26 the end-to-end version of MTR, and MPA, the end-to-end version of MultiPath++. Specifically, HPTR outperforms the pairwise-relative HDGT and the scene-centric SceneTransoonmte Transformer by large margin all metrics. We also show competitive performance on the AV2 leaderboard. We achieve better performance in all metrics except the representation. Since there exists a performance gap while adapting from
compared to GoRela, which uses GNNs to
tackle the pairwise-relatte one dataset to another and we choose WOMD to be our main benchmark, our performance the AV2 leaderboard could be further improved by fine-tuning the hyper-paramettn parameters for the AV2 dataset.
4.3 Ablation study
In Table we ablate different input representations and hierarchical architectures. The scene- centric baseline, HPTR SC, uses the architecture of our HPTR and the input representation of
7

---

Table Results on the marginal motion prediction leaderboards of WOMD and AV2. Both tables
are sorted according to the ranking metric such that the best performing method is on the top. The
ranking metric is soft mAP for WOMD, and brier-minFDE, for AV2. f denotes ensemble. * denotes
predicting more futures than required. SC: scene-centric. AC: agent-centric. PR: pairwise-relative
WOMD test
repr.
soft mAP f
mAP1 t
minADE +
minFDE +
miss rate !
-TR--Advenns [47]
AC
0.4594
0.4492
0.5640
1.1344
0.1160
"1 Wayformer [36]
AC
0,4335
0,4190
0.5454
1.1280
0,1228
MMT [47]
AC
0.4216
0.4129
0.6050
1.2207
0.1351
MMultiPath++ [52]
AC
N/A
0.4092
0.5557
1.1577
0.1340
HPTR (Ours)
PR
0.3968
0.3904
0.5565
1.1393
0.1434
MPA [31] (MultiPath++))
AC
0.3930
0.3866
0.5913
1.2507
0.1603
HDGT [28]
PR
0.3709
0.3577
0.7676
1.1077
0.1325
Gnet [18]
AC
0.3396
0.3259
0.6207
1.2391
0.1718
[37]
SC
N/A
0.2788
0.6117
1.2116
0.1564
WOMD valid
repr.
soft mAP 1
mAP 1
minADE +
minFDE +
miss rate J
HPTR (Ours)
PR
0.4222
0.4150
0.5378
1.0923
0.1326
MTR-e2e
AC
N/A
0.3245
0.5160
1.0404
0. 0.1234
AV2 test
repr.
brier-minFDE,
minFDE, 4
FinFDE
minADE,
miss rate;
Prophhet [55]
AC
1.88
1.33
4.74
0.68
0.18
Gnet [18]
AC
1.90
1.34
4.40
0.69
0.18
TTENET [56]
AC
1.90
1.38
4.69
0.70
0.19
"MTR [47]
AC
1.98
1.44
4.39
0.73
0.15
GoRela [11]
PR
2.01
1.48
4.62
0.76
0.22
HPTR (Ours)
PR
2.03
1.43
4.61
0.73
0.19
THOMAS [19]
AC
2.16
1.51
4.71
0.88
0.20
HDGT [28]
PR
2.24
1.60
5.37
0.84
0.21
SceneTransformer [37]. The agent-centric baseline, WF baseline, our reimplementation of the Wayformer [36] with multi-axis attention and early fusion. Both baselines achieve their expected performances compared to their original implementations. The large performance gap between HPTR SC and other models confirms the disadvantage of scene-centric representation. The agent- centric baseline requires more training iterations. After convergence, its performance is on par with HPTR. To ablate the hierarchical architecture, implement three variations of HPTR; each corresponds to a fusion strategy investigated by Wayformer. The full attention corresponds to the early fusion, diagonal corresponds late fusion, and HPTR with diagonal followed by full attention corresponds to Wayformer with hierarchical fusion. While the early fusion performs the best for Wayformer, for HPTR the lower triangular attention proposed outperforms both the early and the late fusion by a significant margin. The performance of hierarchical fusion is slightly worse than ours, but its inference latency is significantly longer because of the redundancy in its attention matrices.
4.4 Efficiency analysis and qualitative results
In Figure 4 we compare the computational efficiency of the ablation models presented in Table 2. As discussed in prior works [36, 47], one of the major drawbacks of agent-centric approaches is the poor scalability, which is reflected by the large slope of the memory and latency curves of our WF baseline. On the RTX 2080Ti, it can handle only up 48 agents while the inference latency is 140ms. On the contrary, the scene-centric baseline is extremely efficient, but it suffers from poor accuracy. Our HPTR takes the best of both approaches. In terms of efficiency, our HPTR comparable to the scene-centric approaches, while our performance is on par with the agent-centric approaches. Compared the scene ceeneic baseline, the GPU memory consumption and the inference latency of HPTR are slightly higher because for each new agent, we have to compute its relative pose to all existing tokens. Compared the hierarchical architectures introduced by Wayformer, HPTR with our lower triangular attention achieves the best trade-off between accuracy and latency. By reusing the static map features during the online inference, our HPTR predicts the multi-modal futures for 64 agents in 37ms without the use of inference libraries. Narrowing the receptive field, for example by

---

Table Ablation on the valid split of WOMD. The table is sorted according to the soft mAP such that the best- perffrrnnnn method is on the top. Performances are reported as the mean plus-minus standard deviations over training seeds. Models are trained for 60 epochs not otherwise mentioned. WF: Wayformer. SC: scene-centric, AC: agent-centric, PR: pairwise-relative
input repr. MP
intra
intra TI/AG
enhance all
WOMD valid
minFDE 4
soft mAP ?
TIJAG
2all
HPTR (Ours)
PR
X
1.145 - 0.016
0.399 = 0.010
WF baseline (100-epoch)
AC
X
X
X
V
1,161 0.006
0.397 = 0.007
HPTR diag+full (WF hier.)
PR
X
V
1.156 0.014
0.391 = 0.002
HPTR diag (WF late)
PR
J
V
X
X
1.169 + 0.013
0 0.012
HPTR full (WF early)
PR
X
X
X
1.158 + 0.093
0.386 = 0.041
WF baseline
AC
XX
X
V
1.212 + 0.019
0.378 = 0.014
HPTR SC
SC
X
V
1.687 + 0.046 X
0.246 = 0.005
WF baseline
HPTR SC
HPTR full
HPTR diag+full
HPTR Aac
x
OOM Error
HPTR
HPTR diag
HPTR half prec.
10
Offline Inference (ms/lt)
140
Online Inference (ms/it)
50
GPU Memory (GB)
120
8
40
100
80
30
4-
60
20
40
10
20
0
0
1
8 16 32 48
64
1
8
16 32
48
64
1
16
32
48
64
Number of Predicted Agents
Number of Predicted Agents
Number of Predicted Agents
Figure 4: HPTR is as efficient as scene-centric methods in terms GPU memory consumption and inference latency, while being accurate agent-centric methods. We use standard Ubuntu, Python and Pytorch without optimizing for real-time deployment. We predict one scenario at each inference time one 2080Ti, i.e. batch scenarios and the batch size is 1. The number of context agents is the same as the number of predicted agents for all experiments. During offline inference, every inference starts from scratch, while during online inference, we cache and features. Specifically, we repeat the inference of the same scenario for 100 times to simulate online inference, where the static map features are computed at the first step and reused for the next 99 steps.
the static map
reducing AAc from 10 to 8, can improve the inference speed but the accuracy might be affected in some cases. Using half precision at inference time can reduce the latency 25ms (40 fps) without affecting the accuracy. Compared to the most efficient agent-centric method Wayformer, we reduce the memory consumption and the online inference latency by 80% without sacrificing the accuracy.
In Figure 5 we compare the computational efficiency of our method with the GNN-based pairwise- relative methods. Currently there are two such methods, GoRela [11] and HDGT [28]. While HDGT does not match the prediction accuracy of GoRela or our method, it is worth noting that GoRela is not open-sourced, whereas HDGT is. Therefore, we use HDGT in this efficiency comparison. The left plot of Figure 5 confirms the good scalability of HDGT in terms of GPU memory. This is expected because it uses the pairwise-relative representation. In the middle plot, we can observe that HDGT slower than both our HPTR and our agent-centric Wayformer baseline in terms of offline inference speed. To confirm that HDGT runs correctly on our setup, in the right plot we reproduce the inference time of HDGT on the complete WOMD validation split with different validation batch size and compare the reproduce numbers with the reported number in the HDGT paper. The slow inference speed of GNN-based methods such HDGT is mainly because GNN libraries cannot utilize the GPU as efficiently as the basic matrix operations do. Our KNARPE is implemented with the most basic matrix operations, hence it is better suited for real-time and on-board applications.
Figure illustrates the qualitative results of our HPTR. For each type of agent, we select a successful case where the most confident prediction matches the ground truth, and a failure case to show the limitation our method. In the successful cases, we observe the vehicle stops at the red light, the pedestrian walks along the road edge with another person, and the cyclist rides across the road via the

---

WF baseline
HPTR SC
HDGT on our 2080TI
OOM Error
HPTR
HDGT original paper
10
Offline Inference (ms/t)
2000
300
GPU Memory (GB)
Validation Time
2500
6
200
100
2
1000
1 16 32
64
48
1
32 64
128
Number Predicted Agents
Number Predicted Agents
Validation Batch Size
Figure 5: Efficiency comparison with HDGT. We run their official repository on our machine with single 2080Ti. The left plot shows that HDGT achieves good scalability in terms of GPU memory consumption as expected. The middle plot shows that the offline inference latency (with batch size 1) of HDGT scales well, but it is significantly larger than that of other Transformer- baser methods. The right plot shows the inference times for the complete WOMD validation split (64 agents per episode) with different validation batch sizes. It confirms the inference speed reported in the original HDGT
paper correctly reproduced on our setup. Our setup is faster because has a more powerful CPU.
X-
(a) Vehicle
(b) Pedestrian
(c) Cyclist
Figure 6: Qualitative results of HPTR. For each type of agent we show a successful case (top) and failure case (bottom). The intersections cluttered because visualize traffic lights by overlaying the lanes they control with their color. The ground truth is in orange. The target agent and the predictions are in cyan. The most confident prediction has the least transparent color, the thickest line and the biggest cross. Please refer to the appendix for a detailed explanation of the visualization.
crosswalk. Although in the failure cases the most confident prediction deviates from the ground truth, encouraged to that our predictions still reasonable. The failure cases can be addressed by improving the prediction diversity via goal-conditioning which is orthogonal to our contributions.
Conclusion
In this paper we introduce novel attention module, KNARPE, that allows the pairwise-relative representation be used by Transformers. Based on KNARPE, we present pure Transformer-bant framework called HPTR, which uses hierarchical architecture to enable asynchronous token update and avoid redundant computations. While agent-centric methods suffer from poor scalability and scene-centric methods suffer from poor accuracy, HPTR gets the best from both worlds. Experi- ments on the two most popular benchmarks show that our approach achieves superior performance, while satisfying the real-time and on-board requirements of real-world autonomous driving.
Limitations. The poses in this work reside on 2D plane, which can be extended the 3D space in the future. For simplicity, we use L2 distance to obtain the K-nearest neighbors. Future works can explore more sophisticated distances which involve map topology. Currently we use the most basic anchor-based decoder which has limited diversity. This can be improved by using more advanced decoding techniques, such as goal-conditioning In this paper we focus on marginal motion prediction. It would be interesting investigate the potential of our approach in other prediction tasks.
10

---

Acknowledgments and Disclosure of Funding
This work funded by Toyota Motor Europe via the research project TRACE-Zürich.
References
[I] Argoverse. Argoverse 2:
Motion forecasting competition. http:://eraa..aa////bbbllllee
challenge-page/ 1tt1://///eeberr//////o 2022. Accessed: 2023-05-10.
[2] Argoverse. Argoverse 2: Motion forecasting competition. https://eval. wwwweb///wweeeea/eeeeeeeeeeeee http//challlnggs/// http:/eppgge/119//eeeeeeeee
2023. Accessed: 2023-05-10.
[3] Holger Caesar, Juraj Kabzan, Kok Seang Tan, Whye Kit Fong, Eric Wolff, Alex Lang, Luke Fleteher, Oscar Beijbom, and Sammy Omari. nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles, arXiv preprint arXiv:2106. 1810, 2021.
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[5] Sergio Casas, Cole Gulino, Renjic Liao, and Raquel Urtasun. Spagmn: Spatially-aware graph neural networks for relational behavior forecasting from sensor data. In ICRA, 2020.
[6] Sergio Casas, Cole Gulino, Simon Suo, Katie Luo, Renjie Liao, and Raquel Urtasun. Implicit latent variahle model for scene-consistent motion forecasting. In ECCV, 2020.
[7] Sergio Casas, Abbas Sadat, and Raquel Urtasun. Mp3: A unified model to map, perceive, predict and plan. In CVPR, 2021.
[8] Yuning Chai, Benjamin Sapp. Mayank Bansal, and Dragomir Anguelov. Multipath: Multiple probabilistic anchor trajectory hypotheses for hehavior prediction. CoRI., 2019.
[9] Laurene Claussmann, Mare Revilloud, Dominique Gruyer, and Sébastien Glaser. A review of motion planning for highway autonomous driving. IEEE Transactions on Intelligent Transportation Systems, 2019.
[10] Alexander Cui, Sergio Casas, Abbas Sadat, Renjie Liao, Raquel Urtasun. Lookout: Diverse multi-future prediction and planning for seifidriving. In ICCV, 2021.
[11] Alexander Cui, Sergio Casas, Kelvin Wong, Simon Suo, and Raquel Urtasun. Gorela: Go relative for viewpoint-invaite invariant motion forecasting, In IEEE International Conference on Robotics and Automation (ICRA), 2023.
[12] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Lec, and Ruslan Salakhutdinov. Transformer-Al: Attentive language models beyond fixed-length context. In ACL, 2019.
[13] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint 1rx0v:1810.04805, 2018.
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenhorn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. /9999 2020.
[15] Jose Luis Vazquez Espinoza, Alexander Liniger, Wilko Schwarting, Daniela Rus, and Luc Van Gool. Deep interactive motion prediction and planning: Playing games with motion prediction models. In Learning for Dynamics and Control Conference, 2022.
[16] Scott Ettinger. Shuyang Cheng, Benjamin Caine, Chenxi Liu. Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp. Charles R Qi, Yin Zhou, et al. Large scale interactive motion forecasting for autonomous driving:
The waymo open motion dataset. In /CCV, 2021.
[17] Jiyang Gao, Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguelov, Congcong Li, and Cordelia Schmid. Vectomet:: Encoding hd maps and agent dynamics from vectorized representation. In CVPR, 2020.
[18] Xing Gao, Xiaogang Jia, Yikang Li, and Hongkai Xiong. Dynamic scenario representation learning for motion forecasting with heterogeneous graph convolutional recurrent networks. RA-L, 2023.
[19] Thomas Gilles, Stefano Sabatini, Dzmitry Tsishkou, Bogdan Stanciulescu, and Fabien Moutarde. Thomas: Trajectory heatmap output with learned multi-agent sampling. In ICLR, 2022.

---

[20] Roger Girgis. Florian Golemo, Felipe Codevilla, Martin Weiss, Jim Aldon D' D'Suuza, Samira Ebrahimi Kahou, Felix Heide, and Christopher Pal. Latent variable sequential transformers for joint multi-agent motion prediction. ICLR, 2022.
[21] Junru Gu, Chen Sun, and Hang Zhao. Densetut: End-to-ena trajectory prediction from dense goal sets, In ICCV, 2021.
[22] Abhishek Gupta, Alagan Anpalagan, Ling Guan, and Ahmed Shaharyar Khwaja. Deep learning for object detection and scene perception in self-driving cars: Survey, challenges, and open issues. Array, 2021.
[23] Anthony Hu. Zak Murez, Nikhil Mohan, Sofía Dudas, Jeffrey Hawke, Vijay Badrinarayanan, Roberto Cipolla, and Alex Kendall. Fiery: future instance prediction in bird's-eye view from surround monocular cameras. In ICCV, 2021.
[24] Anthony Hu, Gianluca Corrado, Nicolas Griffiths, Zachary Murez, Corina Gurau, Hudson Yeo, Alex Kendall, Roberto Cipolla, and Jamie Shotton. Model-based imitation learning for urban driving. NeurlPS, 2022.
[25] Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai Senyao Du, Tianwei Lin, Wenhai Wang, Lewei Lu, Xiaosong Jia, Qiang Liu, Jifeng Dai, Yu Qiao, and Hongyang Li. Planning- oriented autonomous driving. In CVPR, 2023.
[26] Jing Huang. Viswanath Sivakumar, Mher Mnatsakanyan, and Guan Pang. Improving rotated text detection with rotation region proposal networks. arXiv preprint arXiv: 1tt::/8/11.0700 1107031,, 2018.
[27] Boris Ivanovic and Marco Pavone. dynamic spatiotemporal graphs. In ICCV, 2019.
The trajectron: Probabilistic multi-agent trajectory modeling with
[28] Xiaosong Jia, Penghao Wu, Li Chen, Yu Liu, Hongyang Li, and Junchi Yan. Hdgt: Heterogencous driving graph transformer for multi-agent trajectory prediction via scene encoding. IEEE Transactions on Puttern Analysis and Machine Intelligence (TPAMI), 2023.
[29] Alexey Kamenev, Lirui Wang. Ollin Boer Bohan, Ishwar Kulkarni. Bilal Kartal, Artem Molchanov, Stan Birchfield, David Nistér, and Nikolai Smolyanskiy. Predictionnet: Real-time joint probabilistic traffic prediction for planning, control, and simulation. In /CRA, 2022.
[30] Christos Katrakazas, Mohammed Quddus, Wen-Hua Chen, and Lipika Deka. Real-time motion planning methods for autonomous on-road driving: State-of-the-art and future research directions. Transportation Research Part C: Emerging Technologies, 2015.
[31] Stepan Konev. Mpa: Multipath+ based architecture for motion prediction. arXiv preprint arXiv:2206. 10041, 2022.
[32] Namhoon Lee, Wongun Choi, Paul Vernaza, Christopher B Choy, Philip HS Torr, and Manmohan Chan- draker. Desire: Distant future prediction dynamic scenes with interacting agents. CVPR, 2017.
[33] Ming Liang, Bin Yang. Rui Hu, Yun Chen, Renjie Liao, Song Feng, and Raquel Urtasun. Learning lane graph representations for motion forecasting. In ECCV, 2020.
[34] Francesco Marchetti, Federico Becattini, Lorenzo Seidenari, and Alberto Del Bimbo. Mantra: Memory augmented networks for multiple trajectory prediction. In CVPR, 2020.
[35] Xiaoyu Mo, Yang Xing, and Chen Lv. Heterogencous wew.ounhnnnedeeecncecceccccoc graph attention network for multi- agent trajectory prediction. arXiv preprint arXi::2/06.07661, 2021.
[36] Nigamaa Nayakanti. Rami Al-Rfou, Aurick Zhou, Kratarth Goel, Khaled S Refaat, and Benjamin Sapp. Wayformer: Motion forecasting via simple & efficient attention networks. In ICRA, 2023.
[37] Jiquan Ngiam, Vijay Vasudevan, Benjamin Caine, Zhengdong Zhang, Hao-Tien Lewis Chiang, Jeffrey Ling, Rebecca Roclofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, et al. Scene transformer: unified architecture for predicting future trajectories of multiple agents. In ICLR, 2021.
[38] nuScenes. nuscenes prediction challenge. http///addebbrrrr/6551153 2020. Accessed: 2023-05-10.
aw//aa/ hhtp///abccaaalleggecchcaaeee changeess/aaallnegeeeee page//
[39] Brian Paden, Michal Cáp. Sze Zheng Yong. Dmitry Yershov, and Emilio Frazzoli. A survey of motion planning and control techniques for self- urban vehicles. IEEE Transactions on intelligent vehicles. 2016.
12

---

[40] Seong Hyeon Park, Gyubok Lee, Jimin Seo, Manoj Bhat, Minseok Kang, Jonathan Francis, Ashwin Jadhav, Paul Pu Liang, and Louis-Philippe Morency. Diverse and admissible trajectory forecasting through multimodal context understanding. ECCV, 2020.
[41] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, 2017.
[42] Charles R Qi. Yin Zhou, Mahyar Najibi, Pei Sun, Khoa Vo, Boyang Deng, and Dragomir Anguelov. Offboard object detection from point cloud sequences. In CVPR, 2021.
[43] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 2019.
[44] Davis Rempe, Jonah Philion, Leonidas J Guibas, Sanja Fidler, and Or Litany, Generating useful accident- prone driving scenarios via a learned traffic prior. In CVPR, 2022.
[45] Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically- feasible trajectory forecasting with heterogeneous data. In ECCV, 2020.
[46] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In ACL, 2018.
[47] Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schicle. Motion transformer with global intention localization and local movement refinement. In NeurlPS, 2022.
[48] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR, 2020.
[49] Qiao Sun, Xin Huang, Junru Gu, Brian C Williams, and Hang Zhao. M2i: From factored marginal trajectory prediction to interactive prediction. In CVPR, 2022.
[50] Simon Suo. Sebastian Regalado, Sergio Casas. and Raquel Urtasun. Trafficsim: Learning to simulate realistic multi-agent behaviors. In CVPR, 2021.
[51] Charlie Tang and Russ R Salakhutdinov. Multiple futures prediction. In NeurlPS, 2019.
[52] Balakrishnan Varadarajan, Ahmed Hefny, Avikalp Srivastava, Khaled S Refaat, Nigamaa Nayakanti, Andre Coruman, Kan Chen, Bertrand Douillard, Chi Pang Lam, Dragomir Anguelov, et al. Multipath++:: Efficient information fusion and trajectory aggregation for behavior prediction. In ICRA, 2022.
[53] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention all you need. In NewrIPS, 2017.
[54] Benyou Wang, Lifeng Shang, Christina Lioma, Xin Jiang, Hao Yang. Qun Liu, and Jakob Grue Simonsen. On position embeddings in bert. ICLR, 2021.
[55] Xishun Wang. Tong Su, Fang Da. and Xiaodong Yang. Prophnet: Efficient agent-centric motion forecasting with anchor-informed proposals. In CVPR, 2023.
[56] Yuting Wang, Hangning Zhou, Zhigang Zhang, Chen Feng, Huadong Lin, Chaofei Gao, Yizhi Tang. Zhenting Zhao, Shiyu Zhang, Jie Guo, et al. Tenet: Transformer encoding network for effective temporal flow on motion prediction. arXiv preprint arXiv:2207.001770 2022.
[57] Waymo. Waymo open dataset interaction prediction challenge 2021. https://waymo.. http:///wayooocco/pppeeeee open/ challenges/22111 htt.0////tteaationnnoom
wtt.teractiooopeeiiiiioooo///o/a prediction/ 2021. Accessed: 2023-05-10.
[58] Waymo. Waymo open dataset motion prediction challenge 2022.
httpp:///ayyn http://aaamcco//ppeee/o/ee http://ttooooppeiiion.ll prediction/, 2022. Accessed: 2023-05-10.
[59] Waymo. Waymo open datasct sim agent challenge 2023. https://waymo. cwm////moocooo./aaooooama/a/ae/t wwwpa/////////paaaappeaaaaeeeeee challenges/
2023/sim-agente 2023. Accessed: 2023-05-10.
[60] Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagject Singh, Siddhesh Khandelwal, Bowen Pan, Ramesh Kumar, Andrew Harteett, Jhony Kaesemodel Pontes, et al. Argovorse 2: Next generation datasets for self-driving perception and forecasting, In NewrlPS, 2021.
[61] Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, and Hongyang Chao. Rethinking and improving relative position encoding for vision transformer. In ICCV, 2021.
13

---

[62] Ruibin Xiong, Yunchang Yang. Di He. Kai Zheng, Shuxin Zheng. Chen Xing, Huishuai Zhang. Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In JCML, 2020.
[63] Ekim Yurtsever, Jacoh Lambert, Alexander Carballo, and Kazuya Takeda. A survey of autonomous driving: Common practices and emerging technologies. IEEE access, 2020.
[64] Fangao Zeng, Bin Dong, Yuang Zhang, Tiancai Wang, Xiangyu Zhang, and Yichen Wei. Motr: End-to-end multiple-object tracking with transformer. In ECCV, 2022.
[65] Wenyuan Zeng, Ming Liang, Renjie Liao, and Raquel Urtasun. Lanerenn: Distributed representations for graph-centric motion forecasting. In IROS, 2021.
[66] Wei Zhan, Liting Sun, Hengho Ma, Chenran Li, Xiaosong Jia, Masayoshi Tomizuka, et al. Inter- pret: Interaation-ddttsse--aadddd prediction challenge. http://challenge http://cccallleeeeenttraraaaaaaa interacttion-datast.. com/leader-Bel board, 2021. Accessed: 2023-05-10.
[67] Zhejun Zhang, Alexander Liniger, Dengxin Dai, Fisher Yu, and Luc Van Gool. End-to-end urban driving by imitating reinforcement learning coach. /CCV, 2021.
[68] Zhejun Zhang, Alexander Liniger, Dengxin Dai, Fisher Yu, and Lue Van Gool. Traffichots: Towards world models for autonomous driving simulation and motion prediction. ICRA, 2023.
[69]
Hang Zhao, Jiyang Gao, Tian Lan, Chen Sun, Benjamin Sapp. Balakrishnan Varadarajan, Yue Shen, Yi Shen, Yuning Chai, and Cordelia Schmid. Tnt: Target-driven trajectory prediction. In CoRL, 2021.
[70] Zikang Zhou, Luyao Ye, Jianping Wang, Kui Wu, and Kejie Lu. Hivt: Hicrarchical vector transformer for multi-agent motion prediction. CVPR, 2022.
14

---

A Output representation and training strategies
For each anchor token 2,0 i {I,..., NNac), the confidence head predicts the logits pk {I...., 6}, whereas the trajectory head predicts 6 trajectories, each of which is represented as (pl Hig, log r². logo oy p¹, u.g vyy, 0¹ s'), (T.....TTy).... i.e. the mean of the Gaussian in z, y, the log standard deviation of the Gaussian in æ, y, the correlation of the Gaussians, the velocity in z, the heading angle and the speed. We denote the ground truth as (i, U, ua my 0, 8). The negative log-likelihood loss for position is formulated as
Lpps (á; 0 Maas My Ox. oy: P)-
(6)
The negative cosine loss for the heading angle is formulated as
Lix cos(@ - 8).
(7)
The Huber loss for velocities and speed is formulated as
L = Lz(0z Uz) + La - vyy + Lo s),
(8)
where Ls is the Huber loss. We use g for all Huber losses. The final regression loss for trajectory is the unweighted sum
Taja
Tepus
Laa Liyel:
which is averaged over the future time steps where the ground truth is available. We use a hard assignment strategy, i.e. among the predictions each agent we select the one that is closest the ground truth in terms of average displacement error and optimize only for that prediction. Denoting the index of this prediction as k, we train the confidence head via the cross entropy loss by taking as the ground truth:
exp(pt)
Lonnt log
(10)
> Sim exp(p:)
The final training loss for the complete model is the unweighted sum
L Luy
Leont-
(11)
Our output representation and training strategies are the same as prior works [36, 47, 52], except for the auxiliary losses on velocities, speeds and heading angles.
B
Implementation details
B.1 KNARPE implementation
Figure 7 shows how KNARPE is implemented with the most basic matrix operations, i.c. matrix indexing, summation and element- multiplication.
B.2 Network architectures
We use ReLU activation and set the hidden dimension D 256. Our KNARPE implemented with multi-head attention with 4 heads. We use Transformer with pre-layer normalization [62] with dropout rate of 0. The feed-forward hidden dimension of Transformers set to 1024. The base frequency of the sinusoidal positional encoding is set to 1000. We train single model for all types agents, while each type of agent has its own anchors. The polyline-line level PointNet and MLPs have layers, the intra-MP Transformer encoder has 6 layers, and the inter-class well the AC-to-all Transformer decoders, have 2 layers. Our HPTR has 15.2M trainable parameters in total. The same setup is used for both the WOMD dataset and the AV2 dataset.
In the following, we report the configuration of ablation models. The HPTR with diagonal attention has 6 layers of intra-MP, layers of intra-TL and 3 layers of intra-AG Transformer. It has 15.4M trainable parameters. The HPTR with full attention has 6 layers of all-to-all and 6 layers AC-to-all Transformer. It has 15.2M parameters. The HPTR with diagonal followed by full attention has layers of intra-MP, layers of intra-TI.. layers of intra-AG, 2 layers of all-to-all and layers of AC-to-all Transformer. It has 15.4M trainable parameters. Both the HPTR with full attention
15

---

softmaz
[B,
P(ka)) MBM,KKK
Elementwiie Multiplicttion
Tile D
11(8% [D, D]
[8, M. K. D
A: J,M,K,KK D
Til K
Elementwise Muttipiiaation
Sum Over K
Pt
p: K,
[B M,M, K, D
uo: B. D,
+ps
Sinusoidal
Postiional Embedding
[8, M.
W [D,
W DD
out D.
W (D,
Relative pose: M, K, (ar, 8) between and K nearest tgt
Egfem
[B, M, K,
dist: [B, M [B, JB,,D
K nearast neighbor
[B, M, DI
Figure 7: In contrast to most GNNs, KNARPE is implemented with the most basic matrix operations. The tensor size is shown in the brackets, where / is the batch size, M is the source (src q) sequence length, N is the target (tgt = k = v) sequence length, K in the number neighbors, D is the hidden dimension.
and the HPTR with diagonal followed by full attention have be trained on GPUs with 24GB of VRAM (RTX 3090 in our case) because they require more GPU memory at training time. The scene-centric baseline uses the scene-centric representation and the standard Transformer. Following SceneTransformer [37]. the input 2D positions and 2D directions
pre-processed using sinusoidal positional encoding. The base frequency is set to 1000 for 2D positions and 10 for 2D directions.
The output dimension of the positional encoding is 256. This model has 13M trainable parameters.
The agent-centric baseline closely follows Wayformer [36]. It has 6 layers of all-to-all Transformer and layers of AC-to-all Transformer. The number of latent queries is 192. The learning rate starts 2e-4 and it is multiplied by 0.5 every 20 epochs. The training of the agent-centric baseline takes 100 epochs converge. We do not auxiliary losses velocities, speeds and yaw angles to train this model. This model has 15.6M trainable parameters.
B.3 Pre-processing and post-processing
Our pre-processing and post-processing closely follow MPA [31]. The post-processing manipulates only the confidences via greedy non-maximum suppression. The distance threshold is 2.5m for vehicles, Im for pedestrians and 1.5m for cyclists. We use the average displacement error compute the distance between predicted trajectories. For AV2 we simply use softmax with a temperature of 0.5 instead of doing mon-maximum suppression.
B.4 Training details
Due to the large size of motion prediction datasets, each epoch would take a very long time if trained on the complete training split. In order to track losses more frequently, we randomly sample fraction of all training data each epoch. This equivalent to using the complete training dataset if the training runs for many epochs. We observe a statistically significant correlation between the model performance and the initialization of anchors. We recommend to use large variance for the initialization distributions. Specifically, we use Xavier initialization and multiply the initialized values by Our final models for the leaderboard submission are trained for 10 days and models for ablation and development are trained for days. This long training time is because on the one hand WOMD very targe-scale dataset, and on the other hand we only use 4 RTX 2080Ti GPUs for the training. While comparing the wall time duration of training. the computational resources should be taken into consideration. As reference, HDGT [28] uses 8 V100 and trains for 4-5 days, GoRela [11] uses 16 GPUs (model not specified but most likely 1100/V100) MTR [47] uses 8 RTX 8000, Wayformer [36] uses 16 TPU v3 cores and ProphNet [55] uses 16 V100. All of these methods use much higher number of more powerful GPUs than we use. Given comparable computational resources, the training time of our method could be reduced to 1-2 days. In terms of sample efficiency, method is on par
16

---

waymo_pred/mean_awe aeeerageeeaaaaeeeeeeeeeee o
waymo_pred/mmtn
scr_wond, Aarrd__pr4___t_ scr_ Aprr
scr_wond_Apr4_cto scr_wond__pp
opoch
cpoch
0 20
60 80
60 100
Figure 8: The validation mAP and minFDE logged during the training of our HPTR. The training runs RTX 2080Ti GPUs. The gray curve is trained for 5 days and the green curve continues the training for another days. At each epoch, we sample 25% of the complete training split so as to do validation and log metrics more frequently. As consequence, the effective epoch of these plots should be divided by 4, i.c. in the first 5 days we actually go through the complete WOMD training split 15 times. As can see, training models for 5 days is enough for development. Only the models for the final leaderboard submission are trained for 10 days. The training can be speeded up given more computational resources.
with other methods. As shown in Figure our HPTR converges after epochs (5 days) and our final model is trained for 30 epochs (10 days) WOMD, As reference, HDGT is trained for 30 epochs, MTR is trained for 30 epochs and ProphNet trained for 60 epochs on WOMD.
C Explanation of the visualization
We use white line for lane centers of freeway, aluminium line for lane centers of surface street, dark orange line for stop sign, chocolate line for lane centers of bike lane, dark blue line for road edges boundary, dark plum line for road edges median, butter line for all types of broken road lines, magenta line for all types of solid single road lines, scarlet red line for types of double road lines, chameleon line for speed bumps and entrances driveways, sky blue line for crosswalks.
The intersections are cluttered because we visualize traffic lights by overlaying the lanes they control with their color: red line for stop light state, yellow line for caution light state, green line for go light state, light aluminum line for unknown light state, violet line for flashing light state.
The ground truth is orange. The target agent and the predictions are in cyan. Confidence are reflected by the transparency and thickness of the trajectory. The most confident prediction has the least transparent color, the thickest line and the biggest cross.
D Additional ablation studies
In Table 3 ablate different ways incorporate RPE into the dot-product attention. The differences are insignificant in terms of performance. However, our approach, i.e. adding projected RPE to projected key and value, consumes less memory training time. We this setup in main paper because it can be trained on the RTX 2080 Ti GPUs (12GB VRAM), which are more accessible than the RTX 3090 GPUs (24GB VRAM) in practice.
E Additional results
In Tables 4, 5, and 6, we provide the complete results of our HPTR on the WOMD test split, the WOMD valid split, and the AV2 valid split, respectively.
In Figures 9, 10, and 11, we provide more qualitative results on WOMD valid of our HPTR predicting vehicles, pedestrians, and cyclists, respectively.
17

---

Table 3: Ablation on WOMD valid split. We study different ways to incorporate the RPE into the dot-product attention. Performance is reported the mean plus-minus 3 standard deviations over training seeds. Models are trained for 60 epochs. OOM: out of memory. query. k: key. e: value.
concat query train on mem% RPE
Min
Soft
model description
RPE
2080ti on 3090
FDE
mAP
ours (add proj. RPE to proj. k v)
58.2
1.143 = 0.039 0.401 = 0.007
ours without q, k v bias add proj. RPE to proj- 9,
X
X
58.2
1.140 0.021 0.396 0.009
V
OOM
66.2
1.144 + 0.036 0.397 + 0.006
concat. RPE to k e concat. RPE q, k
X
OOM
71.6
1.138 0.026 0.395 0.006
OOM
90.5
1.133 + 0.024 0.396 0.006
Table 4: Complete results of our HPTR on the WOMD rest split.
Object Type
Measurement
Soft mAP
mAP
Min
Min
Miss
Overlap
Time (s)
f
ADE 4
FDE
Rate
Rate |
Vehicle
0.5631
0.5475
0.2795
0.4997
0.0927
0.0190
Vehicle
5
0.4687
0.4623
0.5714
1.1020
0.1297
0.0415
Vehicle
8
0.3697
0.3664
1.0739
2.2753
0.1787
0.0915
Vehicle
Avg
0.4671
0.4587
0.6416
1,2923
0.1337
0.0507
Pedestrian
0.4534
0.4427
0.1637
0.3111
0.0676
0.2408
Pedestrian
5
0.3422
0.3370
0.3220
0.6616
0.0938
0.2648
Pedestrian
0.2792
0.2751
0.5722
1.2778
0.1248
0.2952
Pedestrian
Avg
0.3582
0.3516
0.3526
0.7502
0.0954
0.2669
Cyclist
3
0.4334
0.4267
0.3266
0.6078
0.1859
0.0494
Cyclist
0.3587
0.3552
0.6166
1,2085
0.1922
0.0900
Cyelist
0.3025
0.3006
1.0825
2.3096
0.2250
0.1369
Cyclist
Avg
0.3649
0.3608
0.6752
1.3753
0,2011
0.0921
Avg
3
0,4833
0.4723
0.2566
0,4729
0.1154
0.1030
Avg
0.3899
0.3848
0.5033
0.9907
0.1386
0.1321
Avg
8
0.3171
0.3140
0.9095
1.9543
0.1762
0.1745
Avg
Avg
0.3968
0.3904
0.5565
1.1393
0.1434
0.1366
Table 5: Complete results of our HPTR on the WOMD valid split.
Object
Measurement
Soft mAP f
mAP
Min
Min
Miss
Overlap
Type
Time (s)
ADE I
FDE
Rate
Rate +
Vehicle
3
0.5611
0.5451
0.2796
0.4988
0.0934
0.0186
Vehicle
0.4704
0.4637
0.5698
1.0986
0.1297
0.0405
Vehicle
8
0.3678
0.3644
1.0731
2.2909
0.1824
0.0909
Vehicle
Avg
0.4664
0.4577
0.6408
1.2961
0.1352
0.0500
Pedestrian
0.4923
0.4802
0.1454
0,2674
0.0478
0.2358
Pedestrian
5
0.4055
0.3993
0.2782
0.5488
0.0661
0.2605
Pedestrian
0.3639
0.3577
0.4834
1.0157
0.0813
0.2901
Pedestrian
Avg
0.4206
0.4124
0.3023
0.6106
0.0651
0.2621
Cyclist
0.4606
0.4519
0.3309
0.6021
0.1779
0.0523
Cyclist
5
0.3822
0.3788
0.6136
1.1843
0.1905
0.0897
Cyclist
0.2962
0.2943
1.0661
2.3242
0.2239
0.1433
Cyclist
Avg
0.3797
0.3750
0.6702
1.3702
0.1974
0.0951
Avg
3
0.5047
0.4924
0.2519
0,4561
0.1064
0.1022
Avg
0.4194
0.4139
0.4872
0.9439
0.1288
0.1302
Avg
8
0.3427
0.3388
0.8742
1.8769
0.1626
0.1748
Avg
Avg
0,4222
0.4150
0.5378
1.0923
0.1326
0.1357
Table 6: Complete results of our HPTR on the AV2 test split.
minFDE&
minFDE, minADEş1 minADE, Miss RatesJ. Miss Rate, brier-minFDEs +
1.43
4.61
0.73
1.84
0.19
0.61
2.03
18

---

(a) Change lane.
Reverse.
(c) U-Turn
(d) Multi molti-modal futures.
Figure 9: Qualitative results of HPTR predicting vehicles. Scenarios are selected from the WOMD validation dataset. The ground truth in orange. The target agent and the predictions are in cyan. The most confident prediction has the least transparent color, the thickest line and the biggest cross.
(a) Interact with parked cars.
(b) On the crosswalk with other people.
(d) Cross the street without using crosswalk. Figure 10: Qualitative results of HPTR predicting pedestrians. Read as Figure
Walk alone the road edge.
(a) Ride on the bike lane (in brown).
(b) Ride road without bike lane.
(c) Ride on the road edge.
(d) Stop red light.
Figure 11: Qualitative results of HPTR predicting cyclists. Read as Figure
19


Reviews:
[{"review_id": "CwPTU1VeyO", "reviewer": "Reviewer_yHiV", "summary": "--", "questions": "--", "limitations": "--", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "--", "weaknesses": "--", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Since this review does not provide any detailed comments, we will omit the rebuttal in this case."}, {"review_id": "jXT0GtWcJP", "reviewer": "Reviewer_Gjvi", "summary": "This paper introduces several ideas to boost the efficiency of marginal motion prediction: (1) represent all input entities as polylines without global pose attributes, (2) use transformer architectures but limit attention to K nearest neighbors, (3) directly use relative pose in transformer computations, (4) apply full self-attention only to map tokens, which can be cached during online inference, (5) obtain traffic light and agent features hierarchically, with cross-attention, and (6) use a final cross-attention block for all agent-anchor pairs to directly decode trajectories without any clustering or ensembling. All of these ideas are intuitive and an ablation study discusses some of their individual contributions. The final model obtains reasonable performance on WOMD and Argoverse 2 while scaling to dense traffic much more feasibly than one of the existing SoTA methods, Wayformer.", "questions": "1. Is the training time of HTPR similar to Wayformer for the same number of training epochs?\n2. How is the KNARPE operation implemented in practice? Do you still compute and mask a dense attention matrix, or implement custom kernels to only compute attention where needed?\n3. How important is the post-processing described in L254-257? Are these techniques commonly applied by methods on these leaderboards?\n4. Could you please elaborate on L261-262, what does sampling 25% and 50% mean in this context?\n5. Would it be possible to compare the inference time (Fig. 4) to a GNN method with relative pose encodings?\n\nMinor:\n\n1. Have you tried adding the blue dots from Fig. 1b to Fig. 1a as well? This could make it clearer to understand which agents are being used in Fig. 1b.\n2. Could Fig. 5 be simplified, in particular by removing the striking colors for the map elements? An alternative option would be to add a legend describing all colors.\n\nUpdate:\n\nThank you for the detailed responses to all questions. The rebuttal addresses all of my concerns, and I would like to maintain my positive rating.", "limitations": "Limitations are discussed in Section 5.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The key contribution of this work lies in clearly highlighting of some problematic practices which are still commonly used in most research on motion forecasting in autonomous driving (heavy emphasis on the offline setting), and bringing efficiency for online inference to the forefront. The ideas presented to improve efficiency are not all new, but in combination interesting and well-motivated. Despite the large number of complex technical concepts covered in the draft, the presentation is clear and it is possible to follow and understand all components.", "weaknesses": "While all the proposed ideas are simple and intuitive, putting them all together yields a complex architecture with a large space of design choices and hyper-parameters. This model trains for 10 days, despite the efficient vectorized input and hierarchical architecture focused on efficiency. The runtime analysis only presents a comparison to an agent-centric baseline Wayformer, which fails to provide evidence for whether the proposed model is efficient among relative pose based forecasting methods (e.g., no evidence for the claim made in L092 that GNNs are more demanding than transformers in the online inference setting).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you very much for your helpful comments and suggestions!\nWe kindly ask you to read our global response which discusses the comparison with GNN-based pairwise-relative methods and the long training time of our models.\nNow in this post we answer your questions as follows.\n\n---\n\n> **Q1**: Is the training time of HTPR similar to Wayformer for the same number of training epochs?\n\n**A1**:\nNo.\nEach training epoch of HPTR takes longer time than Wayformer does.\nGiven 5 days of training time, we can train our reimplementation of Wayformer for 110 epochs, whereas for HPTR it is 60 epochs.\nHowever, HPTR is more sample efficient.\nAs shown in Table 2 of our submission, the performance of Wayformer at epoch 100 is roughly the same as HPTR at epoch 60.\nSo as a result, HPTR and Wayformer converge roughly at the same speed if measured in wall time.\nThe training of HPTR could be further accelerated by pre-computing and saving the relative poses during the dataset pre-processing.\n\n---\n\n> **Q2**: How is the KNARPE operation implemented in practice? Do you still compute and mask a dense attention matrix, or implement custom kernels to only compute attention where needed?\n\n**A2**:\nWe implement our custom multi-head attention with matrix indexing, summation and element-wise multiplication.\nGiven $src\\in \\mathbb{R}^{B\\times M\\times D}$ and $tgt\\in \\mathbb{R}^{B\\times N\\times D}$, where $B$ is the batch size, $M$ is the length of $src$, $N$ is the length of $tgt$ and $D$ is the hidden dimension.\nThe first step is to get the K-nearest-neighbor $tgt$ for each $src$, i.e. get $\\text{tgt}_{knn} \\in \\mathbb{R}^{B\\times M \\times K \\times D}$ by indexing $tgt$ based on the L2 distances $dist \\in \\mathbb{R}^{B\\times M\\times N}$ between $src$ and $tgt$.\nAfter that, we use element-wise multiplication instead of matrix multiplication to compute the attention matrix $A \\in \\mathbb{R}^{B\\times M\\times K}$.\nAll tensors have fixed shape in our implementation and we use masking to address missing tokens.\nFigure 1 in the PDF of the global response illustrates in detail how KNARPE is implemented in practice.\nWe will add this figure and the implementation details of KNARPE to our appendix.\n\n---\n\n> **Q3**: How important is the post-processing described in L254-257? Are these techniques commonly applied by methods on these leaderboards?\n\n**A3**: \nThe non-maximum suppression post-processing can improve the mAP and soft mAP significantly.\nIt is used by almost all methods submitted to the WOMD leaderboard.\nOur specific implementation follows MPA [1], which is one of the winners of the WOMD challenge 2022.\n\n---\n\n> **Q4**: Could you please elaborate on L261-262, what does sampling 25\\% and 50\\% mean in this context?\n\n**A4**:\nWe apologize for the confusion.\nThis sampling is just because we want to perform validation, metrics logging and checkpoint saving more frequently.\nBy sampling 50\\% of the training dataset at each epoch, we effectively reduce the duration of each training epoch by 50\\% but still ensure that all data from the training split is used for training.\nThis is not a necessary step.\nTraining for 60 epochs while sampling 50\\% of the training dataset at each epoch is equivalent to training for 30 epochs while using the complete training split at each epoch.\nThe difference is negligible in our case because our training runs for many epochs and all samples from the dataset are alike (they are from the same domain).\nThe only effective difference is that the former logs the validation metrics twice as often as the latter, and the parameters of the learning rate scheduler should be changed accordingly if it schedules based on epoch numbers.\n\n---\n\n> **Q5**: Would it be possible to compare the inference time (Fig. 4) to a GNN method with relative pose encodings?\n\n**A5**: Please refer to the global response for this question.\n\n---\n\n**Others**:\nThanks for the suggestions.\nWe will add the blue dots from Fig. 1b to Fig. 1a in the camera-ready.\nWe will also try to make Fig. 5 more readable in the camera-ready.\nDue to the limited space in the main paper, we will add the explanation of the visualization of Fig. 5 to the appendix.\n\n---\n[1] Stepan Konev. Mpa: Multipath++ based architecture for motion prediction. arXiv preprint arXiv:2206.10041, 2022"}, {"review_id": "OX8zZD9LLA", "reviewer": "Reviewer_Ccz6", "summary": "This work proposes a novel method for motion forecasting which uses an efficient attention mechanism with pairwise relative representation and asynchronous updates for the static & dynamic parts of the scene. Extensive experiments on Waymo and Argoverse datasets show competitive performance to existing methods while being more efficient than agent-centric methods.", "questions": "The main concern is that the performance benefits of the proposed approach are not clear. MTR-e2e and GoRela have similar performance to the proposed approach and efficiency comparisons with these methods are not provided. Other clarifications required are mentioned in the weaknesses above.", "limitations": "Limitations are discussed.\n\n---\nI have read the rebuttal, other reviews, and discussion. I appreciate the additional ablations and clarifications provided by the authors. While I agree that efficiency is an important consideration, I am not convinced about the effectiveness of the proposed approach. Since efficiency is the central claim, I would expect to see clear gains over baselines that use some combination of vectorized inputs, pairwise relative information, and/or transformer in their architecture. Looking at the results in Wayformer & HiVT papers, they report latency in the range of 30-60ms for different variants which is in the similar range as HPTR. I think the results should be shown in the form of performance vs latency vs capacity plots (similar to Fig. 4,5,6 in Wayformer paper) while comparing to different baselines to show the benefits of HPTR. So, I am retaining my rating of `Borderline reject`.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This work incorporates a pairwise relative representation in a k-nearest neighborhood for attention mechanism which is more efficient than quadratic attention.\n- The proposed method minimizes redundancies in computation by sharing context among agents and using asynchronous updates for static and dynamic tokens. This makes it as efficient as scene-centric methods.\n- The proposed method is competitive to existing approaches (Table 1) on Waymo and Argoverse datasets which do not use ensembles, while more efficient in terms of memory consumption and inference latency.\n- The ablations (Table 2) are helpful in understanding the benefits of different components in the proposed method. Also, mean & standard deviation over 3 runs are reported to account for the randomness in training.", "weaknesses": "- For pairwise relative poses, it'd be useful to compare with a simpler alternative of using relative distances directly (instead of sinusoidal encoding), as done in Interaction Transformer (eq 4. in [1]).\n- How does pairwise-relative representation retain the good scalability of scene-centric representation? Doesn't the scalability come from k-nearest neighbors which reduces the number of agents considered for attention?\n- L165 states that the performance deteriorates if the input is scene-centric. Why is this the case? The pairwise relative representation should help with standard self-attention as well.\n- What is the difference between the middle and right plots in Fig. 4?\n- It'd be useful to have efficiency comparison against other approaches, eg. GoRela since it also uses pairwise relative representation and achieves similar performance. \n- MTR-e2e and GoRela have similar performance to the proposed approach in Table 1. Without efficiency comparisons with these methods, the performance benefits are not clear.\n- Does the WF baseline in Table 2 use ensembles? \n\n[1] Li et al. End-to-end Contextual Perception and Prediction with Interaction Transformer. IROS 2020", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you very much for your helpful comments and suggestions!\nWe kindly ask you to read our global response which discusses the comparison with GNN-based pairwise-relative methods and the long training time of our models.\nNow in this post we answer your questions as follows.\n\n---\n> **Q1**: For pairwise relative poses, it'd be useful to compare with a simpler alternative...\n\n**A1**:\nUnfortunately we cannot provide additional experimental results during the rebuttal phase because the cluster of our institution has been undergoing maintenance.\nHowever, we can discuss this idea from a theoretical perspective.\nIn contrast to the relative pose, the relative distance does not contain the necessary information for making driving decisions.\nIt only tells us how far away an object is, but not at which direction it is located with respect to the agent of interest.\nTherefore, replacing relative poses with relative distances would not work in our case.\nGiven distances and orientations, it is common practice to use sinusoidal positional encoding (such as Eq. 1-3 in our paper) to pre-process the inputs to Transformer blocks.\nNowadays, almost all SOTA Transformers have the positional encoding in their design.\nNevertheless, we think this is an interesting idea and we will add this discussion to the camera-ready.\n\n---\n> **Q2**: How does pairwise-relative representation retain the good scalability...\n\n**A2**:\nLet us take a lane segment as an example.\nSay we have N agents to be predicted.\nAgent-centric methods will transform this single lane segment to the coordinate of each of the N agents.\nEffectively, we will have N copies of the same lane segment and we will encode the same segment N times.\nSene-centric methods present the lane segment in the global coordinate, hence it is encoded only once.\nPairwise-relative methods decompose the lane segment into a high-dimensional local attribute ($dim\\gg3$), and a low-dimensional global pose ($dim=3$).\nThe high-dimensional local attribute is encoded only once and shared by all agents, hence the good scalability.\nThe 3D global poses will be used to compute the 3D relative poses between the lane segment and N agents.\nTo conclude, both the KNN and the pairwise-relative representation contributes to the scalability.\nConsidering $N$ tokens and hidden dimension $D$, KNN reduces the complexity from $\\mathcal{O}(N^2D)$ to $\\mathcal{O}(NKD)$ by restricting the attention of each token to its K-nearest neighbors.\nThe pairwise-relative representation reduces the complexity from $\\mathcal{O}(N^2D)$ to $\\mathcal{O}(N^2\\cdot 3+ND)=\\mathcal{O}(N^2\\cdot 3)$ by sharing the high-dimensional local attribute.\n\n---\n> **Q3**: L165 states that the performance deteriorates if...\n\n**A3**: \nThis question includes two parts.\n\nFirstly, why the performance deteriorates if the input is scene-centric?\nThe representation $(p_i,u_i)$ in L165 is scene-centric because the global pose $p_i$ is in the global coordinate.\nUsing $(p_i,u_i)$ directly as input to the standard self-attention is investigated in SceneTransformer, which is outperformed by other methods by a large margin because scene-centric representation is not rotation and translation invariance.\n\nSecondly, shouldn't the pairwise-relative representation help with standard self-attention as well?\nIn Eq. 1-5 we propose a new attention mechanism to process $(r_{ij}, u_i)$ .\nIn Sec. C of our appendix we have ablated other variations of attention mechanism for the pairwise-relative representation.\nSince the standard self-attention cannot process the pairwise relative representation, we experimented with some variations which are very similar to the standard self-attention.\nThe results show that using pairwise relative representation with (a moderately modified version of) standard self-attention outperforms scene-centric methods, but it is not as good as the KNARPE we proposed.\n\n---\n> **Q4**: What is the difference between the middle and right plots in Fig. 4?\n\n**A4**: \nThe middle plot is offline inference and the right plot is online inference.\nOffline means doing inference with datasets, whereas online means doing inference with streaming inputs, as if on a real car.\nDuring offline inference, for each episode we will inference only once at $t=0$.\nDuring online inference, for each episode we will inference consecutively at $t=\\{0,1,\\dots,T\\}$, where $T=99$ in Fig. 4.\nOur HPTR allows the encoded static map features to be reused across these time steps during the online inference, hence the latency of our models in the right plot is lower than in the middle plot.\nWe will try to improve the clarity of the caption of Fig. 4 in the camera-ready.\n\n---\n> **Q5**: It'd be useful to have efficiency comparison against other approaches, eg. GoRela...\n\n**A5**: Please refer to the global response.\n\n---\n> **Q6**: MTR-e2e and GoRela have similar performance to the proposed approach in Table 1...\n\n**A6**:\nAs shown in Table 1 (WOMD valid), our method outperforms MTR-e2e substantially in mAP (0.415 vs. 0.3245), which is the major metric considered by the WOMD leaderboard.\nOn the AV2 dataset, our method is on a par with GoRela.\nHowever, GoRela focuses on AV2 and does not provide any results on WOMD, whereas we focus on WOMD and tune all hyperparameters for WOMD.\nWe believe our performance on the AV2 leaderboard could be further improved given sufficient tuning on the AV2 dataset.\nIn terms of efficiency, Wayformer is one of the most efficient agent-centric methods.\nSince MTR-e2e is also agent-centric and it is slower than Wayformer (cf. the appendix of MTR and the results section of Wayformer) which we clearly outperform in terms of efficiency, we believe an efficiency comparison with MTR-e2e is unnecessary.\nFor the efficiency comparison with GNN-based pairwise-relative methods such as GoRela, please refer to the global response.\n\n---\n> **Q7**: Does the WF baseline in Table 2 use ensembles?\n\n**A7**: No, none of the models in Table 2 uses ensembles.\n"}, {"review_id": "kqI15W5ud0", "reviewer": "Reviewer_753q", "summary": "This paper proposes a motion prediction framework HPTR. As agent-centric presentation usually has a high computational cost and poor scalability, the paper uses the transformer to encode pairwise-relative representation with K-nearest neighbor attention and relative pose encoding. It proposes a hierarchical transformer-based framework to efficiently encodes intra-class and inter-class information which allows asynchronous update for better online inference efficiency. Experiments on Waymo Open Motion Dataset and Argoverse-2 motion dataset show its superior performance and good efficiency.", "questions": "As the proposed method is built upon transformer blocks and the transformer is well-recognized for its good scalability, have the authors tried to scale up the model to see its performance improvements?", "limitations": "The authors have well discussed the limitation in Sec5.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed hierarchical transformer-based framework is efficient and enables asynchronous token update which is usually ignored in other motion prediction works.\n2. The paper adopts relative pose encoding to better unleash the expressiveness of the pairwise-relative representation. \n3. Experiment results have shown that the proposed method has achieved a good balance between performance and efficiency.\n4. The overall writing is clear and easy to follow.", "weaknesses": "1. It is claimed in the paper that this paper uses transformer and pairwise-relative representation which is less computational than GNN. However, GNN and transformer can be viewed as equivalent if attention is used to aggregate and update information among the nodes as in HDGT. A more in-depth analysis should be provided to better clarify the difference and advantages of the proposed method over GNN based method with pairwise-relative representation like HDGT.\n2. ProphNet also achieves very good results in terms of both performance and efficiency. And in ProphNet's paper, the single model (without ensembling) has achieved 1.89 brier-min FDE6 on the AV2 dataset. The author should include this result and compare the proposed with it in the efficiency part as well. And the analysis of the comparison is also important.\n3. As previous pairwise-relative representation methods do not use pairwise-relative representation, the ablation experiment on RPE should be included to analyze how much performance gain is due to RPE.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you very much for your helpful comments and suggestions!\nWe kindly ask you to read our global response which discusses the comparison with GNN-based pairwise-relative methods and the long training time of our models.\nNow in this post we answer your questions as follows.\n\n---\n>**Q1**: It is claimed in the paper that this paper uses transformer and pairwise-relative representation which is less computational than GNN. However...\n\n**A1**:\nWe agree that Transformer can be formulated as a special case of GNN and theoretically attention mechanism is not computationally more efficient than message passing.\nHowever, in practice, most of the time Transformers are more efficiently implemented on GPUs compared to GNNs.\nAs shown in Fig. 1 of the global response PDF, KNARPE is implemented with the most basic matrix operations (matrix indexing, summation and element-wise multiplication).\nBased on KNARPE, our HPTR uses only these basic matrix operations which are easier to be efficiently deployed than most of the GNNs.\nIn terms of performance, our method outperforms HDGT by a large margin (cf. Table 2 and the WOMD leaderboard).\nIn terms of efficiency, our method is faster than HDGT by an order of magnitude (cf. Fig. 3 of the global response PDF).\nPlease refer to the global response for the detailed discussion.\n\n---\n>**Q2**: ProphNet also achieves very good results in terms of both performance and efficiency. And in ProphNet paper, the single model...\n\n**A2**:\nWe agree that ProphNet (CVPR 2023) as a concurrent work achieves excellent performance and efficiency on AV1 and AV2 datasets.\nAccording to Table 3 of the ProphNet paper, it achieves 1.89 brier-min FDE6 on the AV2 dataset with a single model.\nAccording to Sec. 4.3 of the ProphNet paper, that single model \"uses 3 heads and each with 6 output trajectories\".\nAccording to Sec. 3.6 of the ProphNet paper, ProphNet \"generates more proposals than the required number of output modality\".\nTherefore, we believe it is fair to claim ProphNet predicts more futures than required, whereas it is controversial to claim ProphNet uses ensembling because the single model ensembles only part of its network, i.e. the hydra heads.\nThe performance of ProphNet in Table 1 of our paper was obtained from the AV2 leaderboard accessed early this year when we wrote our paper.\nThat submission entry is now removed from the AV2 leaderboard, but the performance can still be found in the appendix of the ProphNet paper (Table 7).\nAccording to the appendix A of the ProphNet paper, they \"train three different models for ensembling\".\nSo the ProphNet in the Table 1 of our submission actually used ensembling.\nIn the camera-ready we will update Table 1 and replace the old ProphNet with ensembling by the new single-model ProphNet.\nWe will also remove the dagger in front of ProphNet as the single-model ProphNet is not exactly an ensembling.\n\nSince ProphNet is a concurrent work and it is not open-sourced, we don't have enough time to reproduce it and provide a detailed efficiency comparison in our submission.\nHowever, since ProphNet is still agent-centric, the scalability problem is inevitable.\nWe hope the comparison with Wayformer (ICRA 2023), another SOTA agent-centric method, should be enough to demonstrate the advantage of our method over general agent-centric methods.\nMoreover, according to Table 4 and Sec. 4.5 of the ProphNet paper, they have 28ms latency per agent (64 agents in total), which means per episode the latency is 1792ms.\nThis is more than a magnitude slower than our approach which has 60ms latency during the offline inference with 64 agents.\n\n---\n>**Q3**: As previous pairwise-relative representation methods do not use pairwise-relative representation, the ablation experiment on RPE should be included to analyze how much performance gain is due to RPE.\n\n**A3**:\nWe think there is a typo in this question, it should be \"As previous pairwise-relative representation methods do not use ~pairwise-relative representation~ RPE, the ablation experiment on RPE should be included to analyze how much performance gain is due to RPE.\".\n\nOur answer is as follows.\nRPE can be ablated from different aspects.\nFirstly, we can ablate the \"relative\" aspect, i.e. instead of the pairwise-relative representation we use the agent-centric or scene-centric representation.\nThis ablation has been done in Table 2 of our paper.\nNow given the pairwise-relative representation, there are different ways to use it.\nOur specific methods RPE and KNARPE are defined respectively in Eq. 1-3 and Eq. 4-5 of our paper.\nEq. 1-3 are the standard positional encoding which is applied to almost all Transformer-based methods.\nSince it is a common practice to use positional encoding to pre-process the inputs to Transformer blocks, we omit its ablation in our paper.\nIn Table 1 of our appendix we have provided detailed ablations on Eq. 4-5, i.e. different attention mechanisms for RPE.\nWe show that our KNARPE achieves the best performance and efficiency.\nBack to the question \"previous pairwise-relative representation methods do not use RPE\", this is because all prior works are based on GNNs and our RPE is specifically designed for Transformers.\nCombining GNNs with RPE (Eq. 1-3) is an interesting idea but it is out of the scope of our paper which focuses on Transformers.\n\n---\n>**Q4**: As the proposed method is built upon transformer blocks and the transformer is well-recognized for its good scalability, have the authors tried to scale up the model to see its performance improvements?\n\n**A4**:\nWe do observe a proportional relationship between the performance and the model size.\nHowever, due to the limited computational resources (4 RTX 2080Ti), we cannot experiment with a larger model size without further reducing the batch size, which is already rather small (B=12) at the moment."}, {"review_id": "B8KFNfndDb", "reviewer": "Reviewer_LV25", "summary": "This paper proposes a method to do future motion prediction for autonomous driving agents with the focus of having a computational complexity that is suited for real-time deployment. To do so, they propose a new attention mechanism, called KNARPE, and a hierarchical transformer architecture, called HPTR. By leveraging these components, they achieve SOTA performance (found in agent-centric models) while being close to maintaining the efficiency of scene-centric models. They empirically show this by comparing it to relevant models on both the Waymo and Argoverse 2 datasets.", "questions": "Q1) What is Figure 1 contributing to? In my opinion, it is just confusing and lacks proper motivation.\n\nQ2) At l139, should not $t=0$ be included in the history?\n\nQ3) The training takes around 10 days. How is the model convergence, in terms of losses and final KPIs?\n\nQ4) Do other approaches train as long as the proposed approach?\n\nQ5) During experimentation, it is key to revise the method swiftly. If training takes 10 days, did the development of this approach use a shorter schedule for experimentation?\n\nQ6) Based on the development of this approach, do the results of a shorter training schedule seem to correlate well with the results of a longer training schedule?\n\nQ7) How computationally efficient is the proposed approach compared to GoRela? Also see W1.\n\nQ8) See W2.\n\nQ9) The attention to all (all2all) should be a superset of the proposed HPTR architecture. However, as indicated in Table 2, the all2all model is not as good as the HPTR model. Why is this the case? Is there some intuition on this?\n\nQ10) When caching the map features during online inference, these are cached for one timestep, right? I.e., the next timestep the map features are recomputed? Or, are these cached temporally in some way as well?", "limitations": "There is a discussion on limitations that provides some additional clarity and insight.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "S1) The paper is well-written, has good notation, and is easy to understand. Especially compared to many of the prior works.\n\nS2) Agent-centric approaches may be infeasible for real-time systems because they need one forward-pass per agent. The proposed approach mitigates this issue and reaches the good performance of agent-centric approaches while having a lower computational cost.\n\nS3) The computational cost is rather thoroughly measured, as it looks at GPU memory consumption, offline inference time, and online inference time.\n\nS4) The proposed approach obtains good performance on two standard benchmarks.\n\nS5) There is a theoretical comparison to WayFormer (besides the empirical results).\n\nAs a \"side-note strength\", the authors state an intention to release the code publicly, which should be valuable to the research community.", "weaknesses": "W1) The proposed approach is claimed to reach SotA performance at a computational cost that scales in a nice way with the number of agents for which predictions are to be made. There is a thorough comparison to WayFormer, but not to other approaches. For instance, GoRela seems to (very slightly) outperform the proposed approach while also scaling gracefully with additional agents.\n\nW2) The performance of WayFormer in table 2 seems low. Is this some variant of it? In the WayFormer paper, no results are provided on the valid-set, which is what is used for table 2. However, the valid-set seems to be a bit easier than the test-set, so one would expect WayFormer to land a bit above 0.4335 soft mAP, instead of the provided 0.397. Why is this the case?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you very much for your helpful comments and suggestions!\nWe kindly ask you to read our global response which discusses the comparison with GNN-based pairwise-relative methods and the long training time of our models.\nNow in this post we answer your questions as follows.\n\n---\n>**Q1**: What is Figure 1 contributing to? In my opinion, it is just confusing and lacks proper motivation.\n\n**A1**: \nWe use Figure 1 to demonstrate the limitations of agent-centric approaches and introduce the problem of online inference, both of which are addressed by the approach proposed in our paper.\nAt the moment this figure seems to help the understanding of other reviewers.\nWe will modify it in the camera-ready if other reviewers have the same concern.\n\n---\n>**Q2**: At l139, should not t=0 be included in the history?\n\n**A2**:\nThanks for pointing out! We will fix this in the camera-ready.\n\n---\n>**Q3**: The training takes around 10 days. How is the model convergence, in terms of losses and final KPIs?\n\n**A3**: \nThe training takes 10 days because we have limited GPUs (4 RTX 2080Ti).\nThe training time can be reduced given more computational resources.\nIn Fig. 2 of the global response PDF, we provided the curves of validation metrics.\nAs shown in this figure, the model has largely converged after 5 days of training.\nTherefore, only the models for the final leaderboard submission are trained for 10 days, whereas models for development and ablation are trained for 5 days.\n\n---\n>**Q4**: Do other approaches train as long as the proposed approach?\n\n**A4**:\nPlease refer to the global response for this question.\n\n---\n>**Q5**: During experimentation, it is key to revise the method swiftly. If training takes 10 days, did the development of this approach use a shorter schedule for experimentation?\n\n**A5**:\nOnly the models for final submission are trained for 10 days.\nOther models are trained for 5 days.\nIn practice, 5 days are still too long for development.\nBut for our cluster, a 4-GPU 5-day job is far easier to be scheduled than an 8-GPU 2-day job.\nTherefore we use this 4-GPU 5-day setup for experimentation.\n\n---\n>**Q6**: Based on the development of this approach, do the results of a shorter training schedule seem to correlate well with the results of a longer training schedule?\n\n**A6**:\nFortunately yes.\nIn our case training for 5 days is enough for telling the performance of a model as shown in Fig. 2 of the global response PDF.\n\n---\n>**Q7**:\nHow computationally efficient is the proposed approach compared to GoRela? Also see W1.\n\n**A7**: \nPlease refer to the global response for this question.\n\n---\n>**Q8**: See W2. The performance of WayFormer in table 2 seems low. Is this some variant of it? In the WayFormer paper, no results are provided on the valid-set, which is what is used for table 2. However, the valid-set seems to be a bit easier than the test-set, so one would expect WayFormer to land a bit above 0.4335 soft mAP, instead of the provided 0.397. Why is this the case?.\n\n**A8**:\nThere are multiple potential reasons.\nFirstly, our model does not apply ensembling, which affects the soft mAP significantly.\nSecondly, Wayformer is not open-sourced, so it could be that our reimplementation is not perfect.\nIn fact, our reimplementation has a smaller model size due to the limited computational resources (we use 4 RTX 2080Ti, Waymo uses 16 TPU).\nWe will open-source our Wayformer reimplementation such that the community can improve it.\nThirdly, all models in Table 2 are trained for less epochs because they are meant for ablation studies, not for the final submission.\nAnd finally, in the Wayformer paper we can still find the performance of their ablation models on the validation split, not in the Tables or in the text but in the Figures (on the y-axis of Fig. 4,5,6 in the Wayformer paper).\nWe can see their minADE is never below 0.9, which is way larger compared to their test split submission.\n\n---\n>**Q9**: The attention to all (all2all) should be a superset of the proposed HPTR architecture. However, as indicated in Table 2, the all2all model is not as good as the HPTR model. Why is this the case? Is there some intuition on this?\n\n**A9**: \nThis is because for a fair comparison, the number of layers of models in Table 2 are selected such that their total numbers of learnable parameters are roughly the same (15M in our case).\nAs a consequence, the all2all model in Table 2 is not as deep as other models.\nGiven more layers and longer training time, the all2all model can reach the same performance as our HPTR.\nIt is intuitive that the all2all model is more difficult to train because without any inductive bias it allows all possible attentions.\nOur point here is to show the hierarchical architecture we proposed can achieve the same performance with less parameters and training time.\n\n---\n>**Q10**: When caching the map features during online inference, these are cached for one timestep, right? I.e., the next timestep the map features are recomputed? Or, are these cached temporally in some way as well?\n\n**A10**:\nDuring the online inference, the encoded map features are computed only once at the beginning $t=0$ and cached.\nAt $t=0$ we will predict the future trajectories at $t={1,2,\\dots, T}$.\nAt the next time step $t=1$ we will reuse the cached map features and predict the future trajectories at $t={2,3,\\dots, T+1}$.\nThe cached map features can be reused because we assume the map is static, i.e. it does not change from $t=0$ to $t=1$.\nIn our online inference experiments (right plot Fig. 4), the map features cached at $t=0$ are reused for 100 times (i.e. 10 seconds) and we compute the average latency over these 100 inferences.\nIn this case we assume the map does not change within 10 seconds.\nIn the real world we can reuse the cached map features for a longer period of time, say days or maybe weeks.\nRecomputing the map features is necessary when the HD maps are changed, which does not happen very frequently."}]

Author rebuttal:
## Global Response

We thank all reviewers for their helpful feedback.
We are glad that the reviewers appreciate our technical contributions, specifically the KNARPE attention mechanism, the asynchronous token update and the efficiency comparison.
Moreover, we are happy to see that the reviewers agree with us on the importance of identifying and solving the efficiency problems of motion prediction in autonomous driving.

In this global response, we will address two concerns raised by multiple reviewers.

---

### 1. The efficiency comparison with other methods based on GNN and pairwise-relative representation.

To the best of our knowledge, currently there are only two such methods, GoRela and HDGT.
GoRela is published on ICRA 2023 and it is not open-sourced, so we would need to reimplement it.
However, the hyperparameters of its network, such as the hidden dimension, number of layers of each component etc, are not disclosed in the GoRela paper.
Without this information, we cannot reproduce its performance and conduct a fair efficiency comparison.
After receiving the NeurIPS reviews, we immediately sent an Email to the authors of GoRela asking for these implementation details, but we haven't received any response yet.
As such, we cannot provide an efficiency comparison with GoRela during this rebuttal phase.
In terms of performance, our method is on par with GoRela on the AV2 dataset.
However, GoRela focuses on AV2 and it does not provide any results on WOMD, whereas we focus on WOMD and tune all our hyperparameters on WOMD.
We believe our performance on the AV2 leaderboard could be further improved given sufficient tuning on the AV2 dataset.

Fortunately, two weeks ago on 2023-07-20 HDGT was open-sourced.
HDGT is published in CoRL 2022 and TPAMI 2023.
It does not perform as good as our method and GoRela, but since it is based on GNN and pairwise-relative representation, we believe the efficiency comparison with HDGT should address the reviewers' concern effectively.
As shown in the left plot of Fig. 3 in the global response PDF, HDGT demonstrates good scalability in terms of GPU memory consumption because it uses the pairwise-relative representation.
As shown in the middle plot of Fig. 3, in terms of offline inference speed, HDGT is slower than our HPTR, and it is actually even slower than our agent-centric Wayformer baseline.
To confirm that HDGT runs correctly on our setup, in the right plot of Fig. 3 we reproduce the inference time of HDGT on the complete WOMD validation split with different validation batch size and we compare the reproduce numbers with the reported number in the TPAMI paper.
The slow inference speed of GNN-based methods such as HDGT is mainly because the GNN libraries cannot utilize the GPU as efficiently as the basic matrix operations do.
As shown in Fig. 1 of the global response PDF, our KNARPE is implemented with the most basic matrix operations (matrix indexing, summation and element-wise multiplication), hence it is better suited for real-time and on-board applications.

---

### 2. The long training time of our models.

Our final models for the leaderboard submission are trained for 10 days and models for ablation and development are trained for 5 days.
This long training time is because on the one hand WOMD is a very large-scale dataset, and on the other hand we only use 4 RTX 2080Ti GPUs for the training.
While comparing the wall time duration of training, the computational resources should be taken into consideration.
As a reference, HDGT uses 8 V100 and trains for 4-5 days, GoRela uses 16 GPUs (model not specified but most likely A100/V100), MTR uses 8 RTX 8000, Wayformer uses 16 TPU v3 cores and ProphNet uses 16 V100.
All of these methods use a much higher number of more powerful GPUs than we use.
If we had their computational resources, the training time of our method could be reduced to 1-2 days.
In terms of sample efficiency, our method is on par with other methods.
As shown in Fig. 2 of the global response PDF, our HPTR converges after 15 epochs (5 days) and our final model is trained for 30 epochs (10 days) on WOMD.
As a reference, HDGT is trained for 30 epochs, MTR is trained for 30 epochs and ProphNet is trained for 60 epochs on WOMD.

Meta-review: The submission received three weak accept (WA-6), one borderline reject (BR-4) and one reject (R-3) recommendations from the reviewers. The authors’ responses seems to be convincing for the most reviewers engaged in discussion, three 6 ratings favour acceptance. After considering the paper, rebuttal and the reviews, the AC concurred with the acceptance recommendation. Please consider to incorporate all the additional responses & experiments in the final version. The AC also recommend the authors to consider acknowledging many missing seminal, but relevant, literature in the final version.
Decision: Accept (poster)