Title: Real-Time Motion Prediction via Heterogeneous Polyline Transformer with Relative Pose Encoding
Venue: NeurIPS 2023
Accepted: yes
Keywords: Motion Prediction

Autonomous Driving

Transformer
Abstract: The real-world deployment of an autonomous driving system requires its components to run on-board and in real-time, including the motion prediction module that predicts the future trajectories of surrounding traffic participants. Existing agent-centric methods have demonstrated outstanding performance on public benchmarks. However, they suffer from high computational overhead and poor scalability as the number of agents to be predicted increases. To address this problem, we introduce the K-nearest neighbor attention with relative pose encoding (KNARPE), a novel attention mechanism allowing the pairwise-relative representation to be used by Transformers. Then, based on KNARPE we present the Heterogeneous Polyline Transformer with Relative pose encoding (HPTR), a hierarchical framework enabling asynchronous token update during the online inference. By sharing contexts among agents and reusing the unchanged contexts, our approach is as efficient as scene-centric methods, while performing on par with state-of-the-art agent-centric methods. Experiments on Waymo and Argoverse-2 datasets show that HPTR achieves superior performance among end-to-end methods that do not apply expensive post-processing or model ensembling. The code is available at https://github.com/zhejz/HPTR.

Real-Time Motion Prediction via Heterogeneous Polyline Transformer with Relative Pose Encoding

Abstract

The real-world deployment of an autonomous driving system requires its components to run on-board and in real-time, including the motion prediction module that predicts the future trajectories of surrounding traffic participants. Existing agentcentric methods have demonstrated outstanding performance on public benchmarks. However, they suffer from high computational overhead and poor scalability as the number of agents to be predicted increases. To address this problem, we introduce the K-nearest neighbor attention with relative pose encoding (KNARPE), a novel attention mechanism allowing the pairwise-relative representation to be used by Transformers. Then, based on KNARPE we present the Heterogeneous Polyline Transformer with Relative pose encoding (HPTR), a hierarchical framework enabling asynchronous token update during the online inference. By sharing contexts among agents and reusing the unchanged contexts, our approach is as efficient as scene-centric methods, while performing on par with state-of-the-art agent-centric methods. Experiments on Waymo and Argoverse-2 datasets show that HPTR achieves superior performance among end-to-end methods that do not apply expensive post-processing or model ensembling. The code is available at https://github.com/zhejz/HPTR .

1 Introduction

Motion prediction is an important component of modular autonomous driving stack [63]. As the downstream module of perception [22, 42, 48] and the upstream module of planning [3, 9, 39], the task of motion prediction [16, 60] is to predict the multi-modal future trajectories of other agents next to the self-driving vehicle (SDV) based on the heterogeneous observations, including for example high-definition (HD) maps, traffic lights and other vehicles, pedestrians and cyclists. This is an essential task because without accurate and real-time prediction results [30], the planning module cannot safely and comfortably navigate the SDV through highly interactive driving environments.

To achieve top performance on public motion prediction leaderboards [1, 58], state-of-the-art (SOTA) methods [36, 47, 52, 55] leverage agent-centric vectorized representations and Transformer-based network architectures. However, the good performance of these approaches comes at a cost of high computational overhead as illustrated in Figure 1. The most well-known problem of agent-centric

(a) Dense traffic scenario

(b) Agent-centric ROIs

(c) Online inference

(d) Trajectory aggregation

8

8

三

1

approaches is the poor scalability as the number of target agents grows in urban driving environments with dense traffic; for example the busy intersection in Figure 1a. Although the agent-centric regions of interest (ROI) in Figure 1b are largely overlapping, the same context is transformed to the coordinate system of each target agent and independently saved and processed. This causes a huge waste of computational resources, which is often neglected by prior works because their experiments focus on offline inference that queries the prediction only once given a scenario from the dataset. In contrast to offline inference, the motion prediction module of a real-world SDV is continuously queried with streaming inputs during online inference. For example, in Figure 1c, the prediction module is queried again shortly after Figure 1a. Although most contexts remain unchanged after this short period of time, existing methods would start inference from scratch without reusing the encoded features of static contexts. Moreover, prior works often predict a massive number of redundant trajectories and ensemble the outputs of numerous models, such that the final output can be adjusted in favor of prediction diversity during the post-processing, as illustrated in Figure 1d. Although these techniques significantly boost the performance on public benchmarks, they should be sparingly used on a real-world SDV where the computational resources are scarce and the raw predictions, rather than the heuristically aggregated ones, are preferred by the downstream planning module.

To address these problems, our first step is to represent everything as heterogeneous polylines [17]. Then we adopt the pairwise-relative representation [11, 28] and separate the local attribute from the global pose of each polyline. The local attribute specifies what the polyline actually is and it will be shared or reused if possible. The global pose specifies where the polyline is and it will be used to derive the pairwise-relative pose before being processed by the neural networks. Considering polylines as tokens, our second step is to introduce the K -nearest N eighbor A ttention with R elative P ose E ncoding (KNARPE) module, which allows Transformers [53] to aggregate the local contexts for each token via the local attributes and relative poses. While KNARPE enables context sharing among agents, we further propose the H eterogeneous P olyline T ransformer with R elative pose encoding (HPTR), which emphasizes the heterogeneous nature of polylines in motion prediction tasks. Based on KNARPE, HPTR uses hierarchical Transformer encoders and decoders to separate the intra-class and inter-class attention, such that tokens can be updated asynchronously during online inference. More specifically, for static polylines, such as HD maps, their features will be reused, whereas for dynamic polylines, such as traffic lights and agents, their features will be updated on demand.

Our contributions are summarized as follows: (1) We introduce KNARPE, a novel attention mechanism that enables the pairwise-relative representation to be used by Transformer-based architectures. (2) Based on KNARPE we propose HPTR, a hierarchical framework that minimizes the computational overhead via context sharing among agents and asynchronous token update during online inference. (3) Compared to SOTA agent-centric methods, we achieve similar performance while reducing the memory consumption and the inference latency by 80%. By caching the static map features during online inference, HPTR can generate predictions for 64 agents in real time at 40 frames per second. Experiments on Waymo and Argoverse-2 datasets show that our approach compares favorably against other end-to-end methods which do not apply expensive post-processing and ensembling.

2 Related work

Motion prediction is a popular research topic because it is an essential component of modular autonomous driving stacks [63]. The task of motion prediction has been formulated in different ways. In this paper, we focus on the most popular one: the marginal motion prediction [6, 8, 27, 32, 33] where the multi-modal future trajectories are predicted individually for each target agent [1, 38, 58]. In contrast to the marginal formulation, joint motion prediction [19, 20, 35, 49] requires the multimodal futures to be simultaneously generated for all target agents from the same scenario [2, 57, 66]. Beyond the open-loop motion prediction tasks, behavior simulation [44, 50, 68] is formulated in a closed-loop fashion where the future agent trajectories are simulated by rolling out a learned policy [3, 59]. Numerous works have also investigated the possibility to combine motion prediction with other modules, such as perception and planning [7, 15, 23, 29], or to learn an end-to-end driving policy [24, 25, 67] mapping sensor measurements directly to the actions or motion plans of the SDV.

Vectorized representation proposed by VectorNet [17] is widely used in recent works because of its promising performance [47, 52, 55]. Depending on how the coordinate system is selected, the vectorized representation falls into three categories: agent-centric, scene-centric and pairwise-relative. The most popular one is the agent-centric representation [36, 47, 52, 55] that transforms all inputs to the local coordinates of each target agent. Despite its good performance, this approach suffers from poor scalability as the number of target agents grows. To reduce the computational overhead, scene-centric representation [37] shares the contexts among all target agents by transforming all inputs to a global coordinate system, which is not tied to any specific agent but to the whole scene. However, its performance is poor due to the lack of rotation and translation invariance. In this paper we use the pairwise-relative representation, which is less often discussed in prior works because it has not demonstrated any clear advantage. To the best of our knowledge, there are three works using the pairwise-relative representation: HDGT [28], GoRela [11] and HiVT [70]. HDGT and GoRela are based on Graph Neural Networks (GNNs), and their implementation requires message passing and GNN libraries. However, these libraries are often less efficiently implemented on Graphics Processing Units (GPUs) when compared to the basic matrix operations that Transformers rely on. HiVT augments the agent-centric encoders with a pairwise-relative interaction decoder in order to realize multi-agent prediction. In contrast to HiVT which uses agent-centric vectors and the standard Transformer, we formulate all inputs as pairwise-relative polylines and introduce the KNARPE attention mechanism. As a result, our HPTR demonstrates clear advantages in terms of accuracy and efficiency. Replacing the vanilla attention with our KNARPE, we can borrow ideas from other Transformer-based methods and adapt them to the pairwise-relative representation. For example, the hierarchical architecture of our HPTR is inspired by Wayformer [36]. More sophisticated architectures and techniques, such as goal-based decoding [21, 69], can also be incorporated into our framework to further boost the performance.

Rasterized representation was widely used in early works on motion prediction [34, 40, 45, 51]. These methods use convolutional neural networks (CNNs) to process the rasterized image of agentcentric ROI. To improve the inference efficiency, some works [5, 6, 10, 65] pre-compute the static map features and use rotated ROI alignment [26] to retrieve the local contexts around the target agent. In this paper, our KNARPE realizes this operation for pure Transformer-based architectures.

Transformers with attention mechanism [53] have achieved great success in natural language processing [13, 43] and computer vision tasks [4, 14, 64]. Inspired by the vision Transformers, we treat polylines as high-dimensional pixels; the local attribute corresponds to color channels, whereas the global pose corresponds to the pixel-wise location. We further extend the relative position encoding [12, 46] to higher dimensions and rename it to relative pose encoding (RPE) in this paper. Although the benefit of RPE is still controversial for other tasks [54, 61], its value for motion prediction is demonstrated in this paper. Instead of using RPE, previous motion prediction Transformers use everything as input, which is equivalent to concatenating the pixel-wise location to the RGB channels. This is a very uncommon practice from the perspective of vision Transformers.

3 Method

In Sec. 3.1 we introduce the pairwise-relative polyline representation which enjoys the advantages of both the agent-centric and the scene-centric representation. Then in Sec. 3.2 we present the K -nearest N eighbor A ttention with R elative P ose E ncoding (KNARPE) which enables the pairwise-

MP

TL

AG

(a) Polylines.

Local Attribute

starts at

Normalize

w.r.t.

Polyline

Where

is it?

What

is it?

(b) Local attribute represented in the global pose's frame.

Relative

Pose

(

,

)

(

,

)

(

,

)

(

,

)

(c) Obtain relative poses.

relative polyline representation to be used by Transformers. Based on KNARPE, we propose the H eterogeneous P olyline T ransformer with R elative pose encoding (HPTR) in Sec. 3.3. HPTR uses a hierarchical architecture to prevent redundant computations and to realize the asynchronous update of heterogeneous tokens. Finally in Sec. 3.4 we discuss our output representation and training strategies.

3.1 Pairwise-relative polyline representation

Based on prior works [11, 28], we formulate the pairwise-relative polyline representation as the third type of input representation for motion prediction. As shown in Figure 2a, all data relevant to motion prediction can be represented as an ordered list of consecutive vectors, i.e. polylines. By transforming the polyline to the coordinate frame of its global pose as done in Figure 2b, a polyline is fully described by a pair of global pose and local attribute. The global pose specifies the location and heading of the polyline in the global coordinate system, whereas the local attribute describes the polyline in its local coordinates, for example it is a yellow solid lane, 8 meters long, slightly curved to the right. In practice, it does not matter where the lane is on the earth; we only care about where the lane is relative to us. Hence, we obtain the relative poses from the global poses as shown in Figure 2c, before feeding them to the prediction network. Specifically, the global pose and local attribute of polyline i are denoted as ( p i , u i ) . The global pose p i has 3 degrees of freedom ( x, y, θ ) , i.e. the 2D position and the heading yaw angle. The local attribute u i is derived from c i , the intrinsic characteristics of the polyline, and l i , the polyline vectors represented in the local coordinate.

The task of marginal motion prediction is to predict the future 2D positions individually for each target agent based on the static HD map (MP), the history trajectories of all agents (AG) and the traffic lights (TL). For each scenario to be predicted, we consider a maximum number of N MP map polylines, N TL traffic lights and N AG agents. We define t = 0 to be the current step, { T h -1 , . . . , 0 } to be the observed history steps and { 1 , . . . , T f } to be the predicted future steps. The static HD map are spatial polylines ( p MP i , l MP i , c MP i ) , i ∈ { 1 , . . . , N MP } where p MP i ∈ R 3 is its starting vector, l MP i ∈ R N node × 4 are the 2D positions and directions of the N node segments normalized against p MP i , and c MP i ∈ R C MP is the one-hot encoding of C MP different lane types. Polygonal elements, such as crosswalks, are converted to a group of parallel polylines across the polygon. Similar to the map, history trajectories of agents are represented as spatial-temporal polylines ( p AG i , l AG i , c AG i ) , i ∈ { 1 , . . . , N AG } where p AG i ∈ R 3 is the last observed agent pose, l AG i ∈ R T h × (6+3) contains the history 2D positions, directions and velocities normalized against p AG i as well as the 1D speed, yaw rate and acceleration which do not need to be normalized, and c AG i ∈ R 6 is the 3D agent size and the one-hot encoding of 3 agent types (vehicle, pedestrian and cyclist). Following VectorNet [17], we use PointNet [41] with masked max-pooling to aggregate ( l MP i , c MP i ) into u MP i ∈ R D and ( l AG i , c AG i ) into u AG i ∈ R D , where D is the hidden dimension. Since current datasets contain only the detection results rather than the tracking results of traffic lights, we consider only the traffic lights observed at t = 0 and represent them as singular polylines ( p TL i , c TL i ) , i ∈ { 1 , . . . , N TL } where p TL i ∈ R 3 is the pose of the stop point and c TL i ∈ R C TL is the one-hot encoding of C TL different states of traffic lights. We use a multi-layer perceptron (MLP) to encode c TL i into u TL i ∈ R D .

Because the local attributes are normalized and the global poses are used to compute the relative poses before being consumed by the network, the pairwise-relative representation preserves the viewpoint invariance of the agent-centric representation. Additionally, since the local attributes have higher dimensions compared to the 3-dimensional global poses, sharing the local attributes enables the pairwise-relative representation to maintain the good scalability of the scene-centric representation.

However, so far this representation has only be exploited by GNNs, which are not comparable to Transformers in terms of accuracy and efficiency on the motion prediction benchmarks [1, 58].

3.2 KNARPE: K-nearest neighbors attention with relative pose encoding

After encoding the local attributes via the polyline-level encoders, the scenario is now described as ( p i , u i ) , i ∈ { 1 , . . . , N } where N = N MP + N AG + N TL is the total number of polylines. However, this representation cannot be handled by standard self-attention because on the one hand modeling the all-to-all attention is prohibitively expensive [36] due to the large N , and on the other hand the performance deteriorates when the input is scene-centric [37]. To address both problems, we introduce the K -nearest N eighbors A ttention with R elative P ose E ncoding (KNARPE). Similar to MTR [47], KNARPE limits the attention to the K-nearest neighbors of each token. Specifically, κ K i ( d i 1 , . . . , d iN ) ⊆ { 1 , . . . , N } is a set that contains the indices of K tokens closest to token i . For simplicity, we use the L2 distance to measure the distance d ij between the global poses of token i and j . Instead of directly processing global poses, KNARPE uses the relative pose encoding (RPE), i.e. the positional encoding for pairwise-relative poses. Denoting r ij = ( x ij , y ij , θ ij ) to be the global pose of token j represented in the coordinate of token i , i.e. p j transformed to the coordinate of p i , the RPE of r ij is computed using sinusoidal positional encoding (PE) and angular encoding (AE) [68],

where ω is the base frequency. Following [46], the RPE is projected and added to the keys and values to obtain z i , the output of letting token i attend to its K neighbors κ K i ,

where α ij are the attention weights, e ij are the logits, W { q,k,v } , b { q,k,v } are the learnable projection matrices and biases for query, key and value, and ˆ W { k,v } , ˆ b { k,v } are the learnable projection matrices and biases for RPE. We do not apply RPE to query because doing this does not boost the performance in our experiments.

Efficient implementation of KNARPE can be achieved using basic matrix operations such as matrix indexing, summation and element-wise multiplication. See the appendix for more details. KNARPE allows the pairwise-relative representation to be used by pure Transformer-based architectures. Self-attention with KNARPE aggregates the local context for each token in the same way as CNN aggregates the context around each pixel via convolutional kernels. The pixel corresponds to the token, whereas the kernel size of a CNN corresponds to the number of neighbors of KNARPE. Crossattention with KNARPE enables rotated ROI alignment of pre-computed features, e.g. the static map features. Previously, this was only possible for CNN-based [6, 10] and GNN-based [11, 28] methods.

3.3 HPTR: Heterogeneous polyline transformer with relative pose encoding

By replacing the standard multi-head attention [53] with our KNARPE, we construct Transformer encoders and decoders which can model the interactions between heterogeneous polylines via relative poses and local attributes. To address the marginal motion prediction task involving HD map, traffic lights and agents, we propose the H eterogeneous P olyline T ransformer with R elative pose encoding (HPTR) as shown in Figure 3. Since the temporal dimension is eliminated by the polyline-level encoder [17], HPTR models only the spatial relationship between tokens from different classes.

Firstly in Figure 3a, the intra-class Transformer encoders build a block diagonal attention matrix that models the interactions within each class of tokens. Then in Figure 3b, the inter-class Transformer decoders enhance the traffic lights tokens by making them attend to the map tokens, whereas the agent tokens are enhanced by attending to both the traffic lights and map tokens. The intuition behind this is the hierarchical nature of traffic; first there is only the map, then the traffic lights are added

Intra

MP

Intra

TL

Intra

AG

MP

AG

TL

MP

AG

TL

(a) Intra-class TF encoder

Enhance-TL

Enhance-AG

MP

AG

TL

MP

AG

TL

(b) Inter-class TF decoder

All-to-All

Self-Attention

MP

AG

TL

MP

AG

TL

(c) All-to-all TF encoder

AG

Anchors

1

2

3

4

5

6

MP

TL

AG

Self-Attn

Cross-Attn

All Tokens

(d) AC-to-all TF decoder

and finally the agents join. Intuitively, the map influences the interpretation of traffic lights but not vice versa, whereas map and traffic lights together influence the behavior of agents but not vice versa. After the inter-class Transformer decoders, the all-to-all Transformer encoder in Figure 3c builds a full attention matrix allowing tokens to attend to each other irrespective of their class. Finally, each agent token is concatenated with N AC learnable anchors, which are shared among each type of agent. Since the task is marginal motion prediction, we batch over agents and anchors, i.e. now the number of anchor tokens is N AG · N AC. Then the anchor-to-all Transformer decoder in Figure 3d lets each of these anchor tokens attend to all tokens so as to aggregate more contextual information. The final output is denoted as ˆ z AG i , i ∈ { 1 , . . . , N AG · N AC } , and it is used to generate the multi-modal future trajectories. We set the number of neighbors to K for the intra-class and all-to-all Transformers. This number K is multiplied by γ TL, γ AG and γ AC respectively for the enhance-TL, enhance-AG and AC-to-all Transformers, such that these Transformers can have a larger receptive field.

HPTR organizes the Transformers in a hierarchical way such that some of the intermediate results can be cached and reused during the online inference; for example the outputs of the intra-MP Transformer, i.e. the static map features. This allows tokens from different classes to be updated asynchronously, which significantly reduces the online inference latency. We can further improve the efficiency without sacrificing the performance by trimming the redundant attentions. The attention matrices shown in Figure 3a, 3b and 3c are overlapping, which means some relationships are repeatedly modeled, such as the agent-to-agent self-attention. We propose to remove the intra-TL, intra-AG and all-to-all Transformer, while keeping the intra-MP, enhance-TL and enhance-AG Transformer. These three Transformers together build a lower triangular attention matrix which is necessary and sufficient to model the relationship between map, traffic lights and agents. Our approach can be seen as an extension to Wayformer [36] which does not introduce the inter-class Transformer decoders. We can trim Figure 3 differently and cast HPTR into the Wayformer. Specifically, Wayformer with late fusion corresponds to HPTR with diagonal attention matrix (only intra-class TF), Wayformer with early fusion corresponds to HPTR with full attention matrix (only all-to-all TF) and Wayformer with hierarchical fusion corresponds to HPTR with the diagonal followed by the full attention matrix.

3.4 Output representation and training strategies

We follow the common practice [36, 47, 52] to represent the outputs as a mixture of Gaussians and train with hard assignment. The multi-modal future trajectories for each agent are generated by decoding ˆ z AG i , i ∈ { 1 , . . . , N AG · N AC } via two MLPs; the confidence head and the trajectory head. The confidence head predicts a scalar confidence of each trajectory. Besides the Gaussian parameters of 2D positions ( µ x , µ y , σ x , σ y , ρ ) at each future time step, our trajectory head predicts also the yaw angles, speeds and 2D velocities. We use the cross entropy loss for confidences, negative loglikelihood loss for 2D positions, negative cosine loss for yaw angles and Huber loss for speeds and 2D velocities. The final training loss is the unweighted sum of all losses. Following the hard-assignment strategy, for each agent we optimize only the predicted trajectory that is closest to the ground truth in terms of 2D average displacement error. Please refer to the appendix for more details.

4 Experiments

4.1 Experimental setup

Benchmarks. We benchmark our method on the two most popular datasets: the Waymo Open Motion Dataset (WOMD) [16] and the Argoverse-2 motion forecasting dataset (AV2) [60]. Both datasets have an online leaderboard for marginal motion prediction. However, their task descriptions are slightly different. For each scenario, WOMD evaluates the predictions of up to 8 agents, whereas AV2 evaluates only one agent. Both datasets require exactly 6 futures to be predicted for each target agent. The sampling time is 0.1 seconds for both datasets. The history length is 11 steps for WOMD and 50 steps for AV2, whereas the future length is 80 steps for WOMD and 60 steps for AV2. We use the official evaluation tool of the leaderboards to compute the metrics. For the WOMD leaderboard [58], the ranking metric is soft mAP; for the A V2 leaderboard [1], it is brier-minFDE. Please refer to the leaderboard homepages for more details about the dataset and evaluation metrics.

Implementation details. We use Transformer with pre-layer normalization [62] for HPTR. For each episode, we consider N MP = 1024 map polylines, N TL = 40 traffic light stop points and N AG = 64 agents. Each polyline contains up to N node = 20 one-meter-long segments. The base number of neighbors considered by KNARPE is K = 36 . This number is multiplied by γ TL = 2 , γ AG = 4 and γ AC = 10 respectively for the enhance-TL, enhance-AG and AC-to-all Transformer. We set N AC = 6 to predict exactly 6 futures as specified by the leaderboard. We do not apply ensembling or expensive post-processing such as trajectory aggregation. Our post-processing manipulates only the confidences. To improve the soft mAP, we use the non-maximum suppression of MPA [31] for the WOMD leaderboard. To improve the brier-minFDE, we set the softmax temperature to 0.5 for the AV2 leaderboard. More details are provided in the appendix.

Training details. Thanks to the viewpoint invariance of the pairwise-relative representation, no data augmentation or input permutation is needed for the training of HPTR. We use AdamW optimizer with an initial learning rate of 1e-4 and decaying by 0.5 every 25 epochs. We train with a total batch size of 12 episodes on 4 RTX 2080Ti GPUs. For WOMD, we randomly sample 25% from all training episodes at each epoch; for A V2 we use 50%. Our final models are trained for 120 epochs for WOMD and 150 epochs for AV2. The complete training takes 10 days. For WOMD, the SDV agents, agents of interest and agents to be predicted are used for optimization. For A V2, the SDV agents, scored agents and focal agents are used for optimization. To address the imbalance between agent types, for WOMDweadditionally optimize for pedestrians and cyclists which are tracked for at least 4 seconds.

4.2 Benchmark results

In Table 1 we benchmark our approach on the public leaderboards of WOMD and AV2. From the leaderboard we observe that using ensemble and predicting redundant trajectories can increase the prediction diversity and hence improve the soft mAP, brier-minFDE and miss rate. For example, MTR-Adv-ens aggregates the outputs of 7 ablation models, each predicting 64 futures. Besides the exaggerated large number of redundant predictions, it is also non-trivial to aggregate them into 6 predictions. Some works use K-means clustering while the others use non-maximum suppression; both involve heuristic parameter fine-tuning. Since our framework is dedicated to real-world autonomous driving, applying these computationally expensive techniques is contradictory to our motivation. For a fair comparison, we focus on end-to-end methods which do not apply these techniques. On the WOMDdataset, we achieve SOTA performance among the end-to-end methods, including MTR-e2e, the end-to-end version of MTR, and MPA, the end-to-end version of MultiPath++. Specifically, HPTR outperforms the pairwise-relative HDGT and the scene-centric SceneTransformer by a large margin in all metrics. We also show competitive performance on the AV2 leaderboard. We achieve better performance in all metrics except the brier-minFDE compared to GoRela, which uses GNNs to tackle the pairwise-relative representation. Since there exists a performance gap while adapting from one dataset to another and we choose WOMD to be our main benchmark, our performance on the AV2 leaderboard could be further improved by fine-tuning the hyper-parameters for the AV2 dataset.

4.3 Ablation study

In Table 2 we ablate different input representations and hierarchical architectures. The scenecentric baseline, HPTR SC, uses the architecture of our HPTR and the input representation of

SceneTransformer [37]. The agent-centric baseline, WF baseline, is our reimplementation of the Wayformer [36] with multi-axis attention and early fusion. Both baselines achieve their expected performances compared to their original implementations. The large performance gap between HPTR SC and other models confirms the disadvantage of scene-centric representation. The agentcentric baseline requires more training iterations. After convergence, its performance is on par with our HPTR. To ablate the hierarchical architecture, we implement three variations of HPTR; each corresponds to a fusion strategy investigated by Wayformer. The full attention corresponds to the early fusion, diagonal corresponds to late fusion, and HPTR with diagonal followed by full attention corresponds to Wayformer with hierarchical fusion. While the early fusion performs the best for Wayformer, for HPTR the lower triangular attention we proposed outperforms both the early and the late fusion by a significant margin. The performance of hierarchical fusion is slightly worse than ours, but its inference latency is significantly longer because of the redundancy in its attention matrices.

4.4 Efficiency analysis and qualitative results

In Figure 4 we compare the computational efficiency of the ablation models presented in Table 2. As discussed in prior works [36, 47], one of the major drawbacks of agent-centric approaches is the poor scalability, which is reflected by the large slope of the memory and latency curves of our WF baseline. On the RTX 2080Ti, it can handle only up to 48 agents while the inference latency is 140ms. On the contrary, the scene-centric baseline is extremely efficient, but it suffers from poor accuracy. Our HPTR takes the best of both approaches. In terms of efficiency, our HPTR is comparable to the scene-centric approaches, while our performance is on par with the agent-centric approaches. Compared to the scene-centric baseline, the GPU memory consumption and the inference latency of HPTR are slightly higher because for each new agent, we have to compute its relative pose to all existing tokens. Compared to the hierarchical architectures introduced by Wayformer, HPTR with our lower triangular attention achieves the best trade-off between accuracy and latency. By reusing the static map features during the online inference, our HPTR predicts the multi-modal futures for 64 agents in 37ms without the use of inference libraries. Narrowing the receptive field, for example by

WOMD

valid

input

repr.

intra

MP

intra

TL/AG

enhance

TL/AG

all

2all

minFDE

↓

soft mAP

↑

HPTR (Ours)

PR

✓

×

✓

×

1

.

145

±

0

.

016

0

.

399

±

0

.

010

WF baseline (100-epoch)

AC

×

×

×

✓

1

.

161

±

0

.

006

0

.

397

±

0

.

007

HPTR diag+full (WF hier.)

PR

✓

✓

×

✓

1

.

156

±

0

.

014

0

.

391

±

0

.

002

HPTR diag (WF late)

PR

✓

✓

×

×

1

.

169

±

0

.

013

0

.

387

±

0

.

012

HPTR full (WF early)

PR

×

×

×

✓

1

.

158

±

0

.

093

0

.

386

±

0

.

041

WF baseline

AC

×

×

×

✓

1

.

212

±

0

.

019

0

.

378

±

0

.

014

HPTR SC

SC

✓

×

✓

×

1

.

687

±

0

.

046

0

.

246

±

0

.

005

1

8 16

32

48

64

Number of Predicted Agents

0

2

4

6

8

10

1

8 16

32

48

64

Number of Predicted Agents

0

20

40

60

80

100

120

140

Offline Inference (ms/it)

1

8 16

32

48

64

Number of Predicted Agents

0

10

20

30

40

50

Online Inference (ms/it)

WF baseline

OOM Error

HPTR SC

HPTR

HPTR full

HPTR diag

HPTR diag+full

HPTR half prec.

HPTR

AC

=8

reducing λ AC from 10 to 8, can improve the inference speed but the accuracy might be affected in some cases. Using half precision at inference time can reduce the latency to 25ms (40 fps) without affecting the accuracy. Compared to the most efficient agent-centric method Wayformer, we reduce the memory consumption and the online inference latency by 80% without sacrificing the accuracy.

In Figure 5 we compare the computational efficiency of our method with the GNN-based pairwiserelative methods. Currently there are two such methods, GoRela [11] and HDGT [28]. While HDGT does not match the prediction accuracy of GoRela or our method, it is worth noting that GoRela is not open-sourced, whereas HDGT is. Therefore, we use HDGT in this efficiency comparison. The left plot of Figure 5 confirms the good scalability of HDGT in terms of GPU memory. This is expected because it uses the pairwise-relative representation. In the middle plot, we can observe that HDGT is slower than both our HPTR and our agent-centric Wayformer baseline in terms of offline inference speed. To confirm that HDGT runs correctly on our setup, in the right plot we reproduce the inference time of HDGT on the complete WOMD validation split with different validation batch size and we compare the reproduce numbers with the reported number in the HDGT paper. The slow inference speed of GNN-based methods such as HDGT is mainly because GNN libraries cannot utilize the GPU as efficiently as the basic matrix operations do. Our KNARPE is implemented with the most basic matrix operations, hence it is better suited for real-time and on-board applications.

Figure 6 illustrates the qualitative results of our HPTR. For each type of agent, we select a successful case where the most confident prediction matches the ground truth, and a failure case to show the limitation of our method. In the successful cases, we observe the vehicle stops at the red light, the pedestrian walks along the road edge with another person, and the cyclist rides across the road via the

1

8 16

32

48

64

Number of Predicted Agents

0

2

4

6

8

10

1

8 16

32

48

64

Number of Predicted Agents

0

100

200

300

Offline Inference (ms/it)

1 8 16

32

64

128

1000

1500

2000

2500

3000

Validation Time (s)

WF baseline

OOM Error

HPTR SC

HPTR

HDGT on our 2080Ti

HDGT original paper

(a) Vehicle

(b) Pedestrian

(c) Cyclist

K

K

+

crosswalk. Although in the failure cases the most confident prediction deviates from the ground truth, we are encouraged to see that our predictions are still reasonable. The failure cases can be addressed by improving the prediction diversity via goal-conditioning, which is orthogonal to our contributions.

5 Conclusion

In this paper we introduce a novel attention module, KNARPE, that allows the pairwise-relative representation to be used by Transformers. Based on KNARPE, we present a pure Transformer-based framework called HPTR, which uses hierarchical architecture to enable asynchronous token update and avoid redundant computations. While agent-centric methods suffer from poor scalability and scene-centric methods suffer from poor accuracy, our HPTR gets the best from both worlds. Experiments on the two most popular benchmarks show that our approach achieves superior performance, while satisfying the real-time and on-board requirements of real-world autonomous driving.

Limitations. The poses in this work reside on a 2D plane, which can be extended to the 3D space in the future. For simplicity, we use L2 distance to obtain the K-nearest neighbors. Future works can explore more sophisticated distances which involve map topology. Currently we use the most basic anchor-based decoder which has limited diversity. This can be improved by using more advanced decoding techniques, such as goal-conditioning. In this paper we focus on marginal motion prediction. It would be interesting to investigate the potential of our approach in other prediction tasks.

Acknowledgments and Disclosure of Funding

This work is funded by Toyota Motor Europe via the research project TRACE-Zürich.

References

Argoverse. Argoverse 2: Motion forecasting competition. https://eval.ai/web/challenges/ challenge-page/1719/leaderboard/4098 , 2022. Accessed: 2023-05-10.

Argoverse. Argoverse 2: Motion forecasting competition. https://eval.ai/web/challenges/ challenge-page/1719/leaderboard/4761 , 2023. Accessed: 2023-05-10.

Holger Caesar, Juraj Kabzan, Kok Seang Tan, Whye Kit Fong, Eric Wolff, Alex Lang, Luke Fletcher, Oscar Beijbom, and Sammy Omari. nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles. arXiv preprint arXiv:2106.11810 , 2021.

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV , 2020.

Sergio Casas, Cole Gulino, Renjie Liao, and Raquel Urtasun. Spagnn: Spatially-aware graph neural networks for relational behavior forecasting from sensor data. In ICRA , 2020.

Sergio Casas, Cole Gulino, Simon Suo, Katie Luo, Renjie Liao, and Raquel Urtasun. Implicit latent variable model for scene-consistent motion forecasting. In ECCV , 2020.

Sergio Casas, Abbas Sadat, and Raquel Urtasun. Mp3: A unified model to map, perceive, predict and plan. In CVPR , 2021.

Yuning Chai, Benjamin Sapp, Mayank Bansal, and Dragomir Anguelov. Multipath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction. In CoRL , 2019.

Laurene Claussmann, Marc Revilloud, Dominique Gruyer, and Sébastien Glaser. A review of motion planning for highway autonomous driving. IEEE Transactions on Intelligent Transportation Systems , 2019.

Alexander Cui, Sergio Casas, Abbas Sadat, Renjie Liao, and Raquel Urtasun. Lookout: Diverse multi-future prediction and planning for self-driving. In ICCV , 2021.

Alexander Cui, Sergio Casas, Kelvin Wong, Simon Suo, and Raquel Urtasun. Gorela: Go relative for viewpoint-invariant motion forecasting. In IEEE International Conference on Robotics and Automation (ICRA) , 2023.

Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. In ACL , 2019.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Jose Luis Vazquez Espinoza, Alexander Liniger, Wilko Schwarting, Daniela Rus, and Luc Van Gool. Deep interactive motion prediction and planning: Playing games with motion prediction models. In Learning for Dynamics and Control Conference , 2022.

Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles R Qi, Yin Zhou, et al. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In ICCV , 2021.

Jiyang Gao, Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguelov, Congcong Li, and Cordelia Schmid. Vectornet: Encoding hd maps and agent dynamics from vectorized representation. In CVPR , 2020.

Xing Gao, Xiaogang Jia, Yikang Li, and Hongkai Xiong. Dynamic scenario representation learning for motion forecasting with heterogeneous graph convolutional recurrent networks. RA-L , 2023.

Thomas Gilles, Stefano Sabatini, Dzmitry Tsishkou, Bogdan Stanciulescu, and Fabien Moutarde. Thomas: Trajectory heatmap output with learned multi-agent sampling. In ICLR , 2022.

Roger Girgis, Florian Golemo, Felipe Codevilla, Martin Weiss, Jim Aldon D'Souza, Samira Ebrahimi Kahou, Felix Heide, and Christopher Pal. Latent variable sequential set transformers for joint multi-agent motion prediction. In ICLR , 2022.

Junru Gu, Chen Sun, and Hang Zhao. Densetnt: End-to-end trajectory prediction from dense goal sets. In ICCV , 2021.

Abhishek Gupta, Alagan Anpalagan, Ling Guan, and Ahmed Shaharyar Khwaja. Deep learning for object detection and scene perception in self-driving cars: Survey, challenges, and open issues. Array , 2021.

Anthony Hu, Zak Murez, Nikhil Mohan, Sofía Dudas, Jeffrey Hawke, Vijay Badrinarayanan, Roberto Cipolla, and Alex Kendall. Fiery: future instance prediction in bird's-eye view from surround monocular cameras. In ICCV , 2021.

Anthony Hu, Gianluca Corrado, Nicolas Griffiths, Zachary Murez, Corina Gurau, Hudson Yeo, Alex Kendall, Roberto Cipolla, and Jamie Shotton. Model-based imitation learning for urban driving. NeurIPS , 2022.

Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, Lewei Lu, Xiaosong Jia, Qiang Liu, Jifeng Dai, Yu Qiao, and Hongyang Li. Planningoriented autonomous driving. In CVPR , 2023.

Jing Huang, Viswanath Sivakumar, Mher Mnatsakanyan, and Guan Pang. Improving rotated text detection with rotation region proposal networks. arXiv preprint arXiv:1811.07031 , 2018.

Boris Ivanovic and Marco Pavone. The trajectron: Probabilistic multi-agent trajectory modeling with dynamic spatiotemporal graphs. In ICCV , 2019.

Xiaosong Jia, Penghao Wu, Li Chen, Yu Liu, Hongyang Li, and Junchi Yan. Hdgt: Heterogeneous driving graph transformer for multi-agent trajectory prediction via scene encoding. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) , 2023.

Alexey Kamenev, Lirui Wang, Ollin Boer Bohan, Ishwar Kulkarni, Bilal Kartal, Artem Molchanov, Stan Birchfield, David Nistér, and Nikolai Smolyanskiy. Predictionnet: Real-time joint probabilistic traffic prediction for planning, control, and simulation. In ICRA , 2022.

Christos Katrakazas, Mohammed Quddus, Wen-Hua Chen, and Lipika Deka. Real-time motion planning methods for autonomous on-road driving: State-of-the-art and future research directions. Transportation Research Part C: Emerging Technologies , 2015.

Stepan Konev. Mpa: Multipath++ based architecture for motion prediction. arXiv preprint arXiv:2206.10041 , 2022.

Namhoon Lee, Wongun Choi, Paul Vernaza, Christopher B Choy, Philip HS Torr, and Manmohan Chandraker. Desire: Distant future prediction in dynamic scenes with interacting agents. In CVPR , 2017.

Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, and Raquel Urtasun. Learning lane graph representations for motion forecasting. In ECCV , 2020.

Francesco Marchetti, Federico Becattini, Lorenzo Seidenari, and Alberto Del Bimbo. Mantra: Memory augmented networks for multiple trajectory prediction. In CVPR , 2020.

Xiaoyu Mo, Yang Xing, and Chen Lv. Heterogeneous edge-enhanced graph attention network for multiagent trajectory prediction. arXiv preprint arXiv:2106.07161 , 2021.

Nigamaa Nayakanti, Rami Al-Rfou, Aurick Zhou, Kratarth Goel, Khaled S Refaat, and Benjamin Sapp. Wayformer: Motion forecasting via simple & efficient attention networks. In ICRA , 2023.

Jiquan Ngiam, Vijay Vasudevan, Benjamin Caine, Zhengdong Zhang, Hao-Tien Lewis Chiang, Jeffrey Ling, Rebecca Roelofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, et al. Scene transformer: A unified architecture for predicting future trajectories of multiple agents. In ICLR , 2021.

nuScenes. nuscenes prediction challenge. https://eval.ai/web/challenges/challenge-page/ 591/leaderboard/1659 , 2020. Accessed: 2023-05-10.

Brian Paden, Michal ˇ Cáp, Sze Zheng Yong, Dmitry Yershov, and Emilio Frazzoli. A survey of motion planning and control techniques for self-driving urban vehicles. IEEE Transactions on intelligent vehicles , 2016.

Seong Hyeon Park, Gyubok Lee, Jimin Seo, Manoj Bhat, Minseok Kang, Jonathan Francis, Ashwin Jadhav, Paul Pu Liang, and Louis-Philippe Morency. Diverse and admissible trajectory forecasting through multimodal context understanding. In ECCV , 2020.

Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR , 2017.

Charles R Qi, Yin Zhou, Mahyar Najibi, Pei Sun, Khoa Vo, Boyang Deng, and Dragomir Anguelov. Offboard 3d object detection from point cloud sequences. In CVPR , 2021.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 2019.

Davis Rempe, Jonah Philion, Leonidas J Guibas, Sanja Fidler, and Or Litany. Generating useful accidentprone driving scenarios via a learned traffic prior. In CVPR , 2022.

Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamicallyfeasible trajectory forecasting with heterogeneous data. In ECCV , 2020.

Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In ACL , 2018.

Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Motion transformer with global intention localization and local movement refinement. In NeurIPS , 2022.

Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR , 2020.

Qiao Sun, Xin Huang, Junru Gu, Brian C Williams, and Hang Zhao. M2i: From factored marginal trajectory prediction to interactive prediction. In CVPR , 2022.

Simon Suo, Sebastian Regalado, Sergio Casas, and Raquel Urtasun. Trafficsim: Learning to simulate realistic multi-agent behaviors. In CVPR , 2021.

Charlie Tang and Russ R Salakhutdinov. Multiple futures prediction. In NeurIPS , 2019.

Balakrishnan Varadarajan, Ahmed Hefny, Avikalp Srivastava, Khaled S Refaat, Nigamaa Nayakanti, Andre Cornman, Kan Chen, Bertrand Douillard, Chi Pang Lam, Dragomir Anguelov, et al. Multipath++: Efficient information fusion and trajectory aggregation for behavior prediction. In ICRA , 2022.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS , 2017.

Benyou Wang, Lifeng Shang, Christina Lioma, Xin Jiang, Hao Yang, Qun Liu, and Jakob Grue Simonsen. On position embeddings in bert. In ICLR , 2021.

Xishun Wang, Tong Su, Fang Da, and Xiaodong Yang. Prophnet: Efficient agent-centric motion forecasting with anchor-informed proposals. In CVPR , 2023.

Yuting Wang, Hangning Zhou, Zhigang Zhang, Chen Feng, Huadong Lin, Chaofei Gao, Yizhi Tang, Zhenting Zhao, Shiyu Zhang, Jie Guo, et al. Tenet: Transformer encoding network for effective temporal flow on motion prediction. arXiv preprint arXiv:2207.00170 , 2022.

Waymo. Waymo open dataset interaction prediction challenge 2021. https://waymo.com/open/ challenges/2021/interaction-prediction/ , 2021. Accessed: 2023-05-10.

Waymo. Waymo open dataset motion prediction challenge 2022. https://waymo.com/open/ challenges/2022/motion-prediction/ , 2022. Accessed: 2023-05-10.

Waymo. Waymo open dataset sim agent challenge 2023. https://waymo.com/open/challenges/ 2023/sim-agents/ , 2023. Accessed: 2023-05-10.

Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, et al. Argoverse 2: Next generation datasets for self-driving perception and forecasting. In NeurIPS , 2021.

Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, and Hongyang Chao. Rethinking and improving relative position encoding for vision transformer. In ICCV , 2021.

Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In ICML , 2020.

Ekim Yurtsever, Jacob Lambert, Alexander Carballo, and Kazuya Takeda. A survey of autonomous driving: Common practices and emerging technologies. IEEE access , 2020.

Fangao Zeng, Bin Dong, Yuang Zhang, Tiancai Wang, Xiangyu Zhang, and Yichen Wei. Motr: End-to-end multiple-object tracking with transformer. In ECCV , 2022.

Wenyuan Zeng, Ming Liang, Renjie Liao, and Raquel Urtasun. Lanercnn: Distributed representations for graph-centric motion forecasting. In IROS , 2021.

Wei Zhan, Liting Sun, Hengbo Ma, Chenran Li, Xiaosong Jia, Masayoshi Tomizuka, et al. Interpret: Interaction-dataset-based prediction challenge. http://challenge.interaction-dataset. com/leader-board , 2021. Accessed: 2023-05-10.

Zhejun Zhang, Alexander Liniger, Dengxin Dai, Fisher Yu, and Luc Van Gool. End-to-end urban driving by imitating a reinforcement learning coach. In ICCV , 2021.

Zhejun Zhang, Alexander Liniger, Dengxin Dai, Fisher Yu, and Luc Van Gool. Trafficbots: Towards world models for autonomous driving simulation and motion prediction. In ICRA , 2023.

Hang Zhao, Jiyang Gao, Tian Lan, Chen Sun, Benjamin Sapp, Balakrishnan Varadarajan, Yue Shen, Yi Shen, Yuning Chai, and Cordelia Schmid. Tnt: Target-driven trajectory prediction. In CoRL , 2021.

Zikang Zhou, Luyao Ye, Jianping Wang, Kui Wu, and Kejie Lu. Hivt: Hierarchical vector transformer for multi-agent motion prediction. In CVPR , 2022.

A Output representation and training strategies

For each anchor token ˆ z AG i , i ∈ { 1 , . . . , N AG · N AC } , the confidence head predicts the logits p k , k ∈ { 1 , . . . , 6 } , whereas the trajectory head predicts 6 trajectories, each of which is represented as ( µ t x , µ t y , log σ t x , log σ t y , ρ t , v t x , v t y , θ t , s t ) , t ∈ { 1 , . . . , T f } , i.e. the mean of the Gaussian in x, y , the log standard deviation of the Gaussian in x, y , the correlation of the Gaussians, the velocity in x, y , the heading angle and the speed. We denote the ground truth as (ˆ x, ˆ y, ˆ v x , ˆ v y , ˆ θ, ˆ s ) . The negative log-likelihood loss for position is formulated as

The negative cosine loss for the heading angle is formulated as

The Huber loss for velocities and speed is formulated as

where L δ is the Huber loss. We use δ = 1 for all Huber losses. The final regression loss for a trajectory is the unweighted sum

which is averaged over the future time steps where the ground truth is available. We use a hard assignment strategy, i.e. among the 6 predictions of each agent we select the one that is closest to the ground truth in terms of average displacement error and optimize only for that prediction. Denoting the index of this prediction as ˆ k , we train the confidence head via the cross entropy loss by taking ˆ k as the ground truth:

The final training loss for the complete model is the unweighted sum

Our output representation and training strategies are the same as prior works [36, 47, 52], except for the auxiliary losses on velocities, speeds and heading angles.

B Implementation details

B.1 KNARPE implementation

Figure 7 shows how KNARPE is implemented with the most basic matrix operations, i.e. matrix indexing, summation and element-wise multiplication.

B.2 Network architectures

We use ReLU activation and set the hidden dimension D to 256. Our KNARPE is implemented with multi-head attention with 4 heads. We use Transformer with pre-layer normalization [62] with a dropout rate of 0.1. The feed-forward hidden dimension of Transformers is set to 1024. The base frequency ω of the sinusoidal positional encoding is set to 1000. We train a single model for all types of agents, while each type of agent has its own anchors. The polyline-line level PointNet and MLPs have 3 layers, the intra-MP Transformer encoder has 6 layers, and the inter-class as well as the AC-to-all Transformer decoders, have 2 layers. Our HPTR has 15.2M trainable parameters in total. The same setup is used for both the WOMD dataset and the AV2 dataset.

In the following, we report the configuration of ablation models. The HPTR with diagonal attention has 6 layers of intra-MP, 3 layers of intra-TL and 3 layers of intra-AG Transformer. It has 15.4M trainable parameters. The HPTR with full attention has 6 layers of all-to-all and 6 layers of AC-to-all Transformer. It has 15.2M parameters. The HPTR with diagonal followed by full attention has 6 layers of intra-MP, 2 layers of intra-TL, 2 layers of intra-AG, 2 layers of all-to-all and 2 layers of AC-to-all Transformer. It has 15.4M trainable parameters. Both the HPTR with full attention

K nearest

neighbor

Tile

Elementwise

Multiplication

Sum Over

Sum Over

Elementwise

Multiplication

Tile

Relative pose:

between

and K nearest

Sinusoidal

Positional Embedding

and the HPTR with diagonal followed by full attention have to be trained on GPUs with 24GB of VRAM (RTX 3090 in our case) because they require more GPU memory at training time. The scene-centric baseline uses the scene-centric representation and the standard Transformer. Following SceneTransformer [37], the input 2D positions and 2D directions are pre-processed using sinusoidal positional encoding. The base frequency is set to 1000 for 2D positions and 10 for 2D directions. The output dimension of the positional encoding is 256. This model has 13M trainable parameters. The agent-centric baseline closely follows Wayformer [36]. It has 6 layers of all-to-all Transformer and 8 layers of AC-to-all Transformer. The number of latent queries is 192. The learning rate starts at 2e-4 and it is multiplied by 0.5 every 20 epochs. The training of the agent-centric baseline takes 100 epochs to converge. We do not use auxiliary losses on velocities, speeds and yaw angles to train this model. This model has 15.6M trainable parameters.

B.3 Pre-processing and post-processing

Our pre-processing and post-processing closely follow MPA [31]. The post-processing manipulates only the confidences via greedy non-maximum suppression. The distance threshold is 2.5m for vehicles, 1m for pedestrians and 1.5m for cyclists. We use the average displacement error to compute the distance between predicted trajectories. For A V2 we simply use softmax with a temperature of 0.5 instead of doing non-maximum suppression.

B.4 Training details

Due to the large size of motion prediction datasets, each epoch would take a very long time if trained on the complete training split. In order to track losses more frequently, we randomly sample a fraction of all training data in each epoch. This is equivalent to using the complete training dataset if the training runs for many epochs. We observe a statistically significant correlation between the model performance and the initialization of anchors. We recommend to use a large variance for the initialization distributions. Specifically, we use Xavier initialization and multiply the initialized values by 5.

Our final models for the leaderboard submission are trained for 10 days and models for ablation and development are trained for 5 days. This long training time is because on the one hand WOMD is a very large-scale dataset, and on the other hand we only use 4 RTX 2080Ti GPUs for the training. While comparing the wall time duration of training, the computational resources should be taken into consideration. As a reference, HDGT [28] uses 8 V100 and trains for 4-5 days, GoRela [11] uses 16 GPUs (model not specified but most likely A100/V100), MTR [47] uses 8 RTX 8000, Wayformer [36] uses 16 TPU v3 cores and ProphNet [55] uses 16 V100. All of these methods use a much higher number of more powerful GPUs than we use. Given comparable computational resources, the training time of our method could be reduced to 1-2 days. In terms of sample efficiency, our method is on par

waymo_pred/mean_average_precision

waymo_pred/min_ade

scr_womd_Apr4_ctd

scr_womd_Apr4

scr_womd_Apr4_ctd

scr_womd_Apr4

0.4

0.8

0.3

0.6

0.2

0.4

0.1

0.2

0

0

0

20

40

60

80

100

0

20

40

60

80

100

with other methods. As shown in Figure 8, our HPTR converges after 15 epochs (5 days) and our final model is trained for 30 epochs (10 days) on WOMD. As a reference, HDGT is trained for 30 epochs, MTR is trained for 30 epochs and ProphNet is trained for 60 epochs on WOMD.

C Explanation of the visualization

We use white line for lane centers of freeway, aluminium line for lane centers of surface street, dark orange line for stop sign, chocolate line for lane centers of bike lane, dark blue line for road edges boundary, dark plum line for road edges median, butter line for all types of broken road lines, magenta line for all types of solid single road lines, scarlet red line for all types of double road lines, chameleon line for speed bumps and entrances to driveways, sky blue line for crosswalks.

The intersections are cluttered because we visualize traffic lights by overlaying the lanes they control with their color: red line for stop light state, yellow line for caution light state, green line for go light state, light aluminum line for unknown light state, violet line for flashing light state.

The ground truth is in orange. The target agent and the predictions are in cyan. Confidence are reflected by the transparency and thickness of the trajectory. The most confident prediction has the least transparent color, the thickest line and the biggest cross.

D Additional ablation studies

In Table 3 we ablate different ways to incorporate RPE into the dot-product attention. The differences are insignificant in terms of performance. However, our approach, i.e. adding projected RPE to projected key and value, consumes less memory at training time. We use this setup in our main paper because it can be trained on the RTX 2080 Ti GPUs (12GB VRAM), which are more accessible than the RTX 3090 GPUs (24GB VRAM) in practice.

E Additional results

In Tables 4, 5, and 6, we provide the complete results of our HPTR on the WOMD test split, the WOMD valid split, and the A V2 valid split, respectively.

In Figures 9, 10, and 11, we provide more qualitative results on WOMD valid of our HPTR predicting vehicles, pedestrians, and cyclists, respectively.

(a) Change lane.

(b) Reverse.

→

→

(c) U-Turn.

(d) Multi-modal futures.

(c) Walk alone on the road edge.

(d) Cross the street without using crosswalk.

→

→

(c) Ride on the road edge.

(d) Stop at red light.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method to do future motion prediction for autonomous driving agents with the focus of having a computational complexity that is suited for real-time deployment. To do so, they propose a new attention mechanism, called KNARPE, and a hierarchical transformer architecture, called HPTR. By leveraging these components, they achieve SOTA performance (found in agent-centric models) while being close to maintaining the efficiency of scene-centric models. They empirically show this by comparing it to relevant models on both the Waymo and Argoverse 2 datasets.\n\nSTRENGTHS:\nS1) The paper is well-written, has good notation, and is easy to understand. Especially compared to many of the prior works.\n\nS2) Agent-centric approaches may be infeasible for real-time systems because they need one forward-pass per agent. The proposed approach mitigates this issue and reaches the good performance of agent-centric approaches while having a lower computational cost.\n\nS3) The computational cost is rather thoroughly measured, as it looks at GPU memory consumption, offline inference time, and online inference time.\n\nS4) The proposed approach obtains good performance on two standard benchmarks.\n\nS5) There is a theoretical comparison to WayFormer (besides the empirical results).\n\nAs a \"side-note strength\", the authors state an intention to release the code publicly, which should be valuable to the research community.\n\nWEAKNESSES:\nW1) The proposed approach is claimed to reach SotA performance at a computational cost that scales in a nice way with the number of agents for which predictions are to be made. There is a thorough comparison to WayFormer, but not to other approaches. For instance, GoRela seems to (very slightly) outperform the proposed approach while also scaling gracefully with additional agents.\n\nW2) The performance of WayFormer in table 2 seems low. Is this some variant of it? In the WayFormer paper, no results are provided on the valid-set, which is what is used for table 2. However, the valid-set seems to be a bit easier than the test-set, so one would expect WayFormer to land a bit above 0.4335 soft mAP, instead of the provided 0.397. Why is this the case?\n\nQUESTIONS:\nQ1) What is Figure 1 contributing to? In my opinion, it is just confusing and lacks proper motivation.\n\nQ2) At l139, should not $t=0$ be included in the history?\n\nQ3) The training takes around 10 days. How is the model convergence, in terms of losses and final KPIs?\n\nQ4) Do other approaches train as long as the proposed approach?\n\nQ5) During experimentation, it is key to revise the method swiftly. If training takes 10 days, did the development of this approach use a shorter schedule for experimentation?\n\nQ6) Based on the development of this approach, do the results of a shorter training schedule seem to correlate well with the results of a longer training schedule?\n\nQ7) How computationally efficient is the proposed approach compared to GoRela? Also see W1.\n\nQ8) See W2.\n\nQ9) The attention to all (all2all) should be a superset of the proposed HPTR architecture. However, as indicated in Table 2, the all2all model is not as good as the HPTR model. Why is this the case? Is there some intuition on this?\n\nQ10) When caching the map features during online inference, these are cached for one timestep, right? I.e., the next timestep the map features are recomputed? Or, are these cached temporally in some way as well?\n\nLIMITATIONS:\nThere is a discussion on limitations that provides some additional clarity and insight.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a motion prediction framework HPTR. As agent-centric presentation usually has a high computational cost and poor scalability, the paper uses the transformer to encode pairwise-relative representation with K-nearest neighbor attention and relative pose encoding. It proposes a hierarchical transformer-based framework to efficiently encodes intra-class and inter-class information which allows asynchronous update for better online inference efficiency. Experiments on Waymo Open Motion Dataset and Argoverse-2 motion dataset show its superior performance and good efficiency.\n\nSTRENGTHS:\n1. The proposed hierarchical transformer-based framework is efficient and enables asynchronous token update which is usually ignored in other motion prediction works.\n2. The paper adopts relative pose encoding to better unleash the expressiveness of the pairwise-relative representation. \n3. Experiment results have shown that the proposed method has achieved a good balance between performance and efficiency.\n4. The overall writing is clear and easy to follow.\n\nWEAKNESSES:\n1. It is claimed in the paper that this paper uses transformer and pairwise-relative representation which is less computational than GNN. However, GNN and transformer can be viewed as equivalent if attention is used to aggregate and update information among the nodes as in HDGT. A more in-depth analysis should be provided to better clarify the difference and advantages of the proposed method over GNN based method with pairwise-relative representation like HDGT.\n2. ProphNet also achieves very good results in terms of both performance and efficiency. And in ProphNet's paper, the single model (without ensembling) has achieved 1.89 brier-min FDE6 on the AV2 dataset. The author should include this result and compare the proposed with it in the efficiency part as well. And the analysis of the comparison is also important.\n3. As previous pairwise-relative representation methods do not use pairwise-relative representation, the ablation experiment on RPE should be included to analyze how much performance gain is due to RPE.\n\nQUESTIONS:\nAs the proposed method is built upon transformer blocks and the transformer is well-recognized for its good scalability, have the authors tried to scale up the model to see its performance improvements?\n\nLIMITATIONS:\nThe authors have well discussed the limitation in Sec5.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a novel method for motion forecasting which uses an efficient attention mechanism with pairwise relative representation and asynchronous updates for the static & dynamic parts of the scene. Extensive experiments on Waymo and Argoverse datasets show competitive performance to existing methods while being more efficient than agent-centric methods.\n\nSTRENGTHS:\n- This work incorporates a pairwise relative representation in a k-nearest neighborhood for attention mechanism which is more efficient than quadratic attention.\n- The proposed method minimizes redundancies in computation by sharing context among agents and using asynchronous updates for static and dynamic tokens. This makes it as efficient as scene-centric methods.\n- The proposed method is competitive to existing approaches (Table 1) on Waymo and Argoverse datasets which do not use ensembles, while more efficient in terms of memory consumption and inference latency.\n- The ablations (Table 2) are helpful in understanding the benefits of different components in the proposed method. Also, mean & standard deviation over 3 runs are reported to account for the randomness in training.\n\nWEAKNESSES:\n- For pairwise relative poses, it'd be useful to compare with a simpler alternative of using relative distances directly (instead of sinusoidal encoding), as done in Interaction Transformer (eq 4. in [1]).\n- How does pairwise-relative representation retain the good scalability of scene-centric representation? Doesn't the scalability come from k-nearest neighbors which reduces the number of agents considered for attention?\n- L165 states that the performance deteriorates if the input is scene-centric. Why is this the case? The pairwise relative representation should help with standard self-attention as well.\n- What is the difference between the middle and right plots in Fig. 4?\n- It'd be useful to have efficiency comparison against other approaches, eg. GoRela since it also uses pairwise relative representation and achieves similar performance. \n- MTR-e2e and GoRela have similar performance to the proposed approach in Table 1. Without efficiency comparisons with these methods, the performance benefits are not clear.\n- Does the WF baseline in Table 2 use ensembles? \n\n[1] Li et al. End-to-end Contextual Perception and Prediction with Interaction Transformer. IROS 2020\n\nQUESTIONS:\nThe main concern is that the performance benefits of the proposed approach are not clear. MTR-e2e and GoRela have similar performance to the proposed approach and efficiency comparisons with these methods are not provided. Other clarifications required are mentioned in the weaknesses above.\n\nLIMITATIONS:\nLimitations are discussed.\n\n---\nI have read the rebuttal, other reviews, and discussion. I appreciate the additional ablations and clarifications provided by the authors. While I agree that efficiency is an important consideration, I am not convinced about the effectiveness of the proposed approach. Since efficiency is the central claim, I would expect to see clear gains over baselines that use some combination of vectorized inputs, pairwise relative information, and/or transformer in their architecture. Looking at the results in Wayformer & HiVT papers, they report latency in the range of 30-60ms for different variants which is in the similar range as HPTR. I think the results should be shown in the form of performance vs latency vs capacity plots (similar to Fig. 4,5,6 in Wayformer paper) while comparing to different baselines to show the benefits of HPTR. So, I am retaining my rating of `Borderline reject`.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces several ideas to boost the efficiency of marginal motion prediction: (1) represent all input entities as polylines without global pose attributes, (2) use transformer architectures but limit attention to K nearest neighbors, (3) directly use relative pose in transformer computations, (4) apply full self-attention only to map tokens, which can be cached during online inference, (5) obtain traffic light and agent features hierarchically, with cross-attention, and (6) use a final cross-attention block for all agent-anchor pairs to directly decode trajectories without any clustering or ensembling. All of these ideas are intuitive and an ablation study discusses some of their individual contributions. The final model obtains reasonable performance on WOMD and Argoverse 2 while scaling to dense traffic much more feasibly than one of the existing SoTA methods, Wayformer.\n\nSTRENGTHS:\nThe key contribution of this work lies in clearly highlighting of some problematic practices which are still commonly used in most research on motion forecasting in autonomous driving (heavy emphasis on the offline setting), and bringing efficiency for online inference to the forefront. The ideas presented to improve efficiency are not all new, but in combination interesting and well-motivated. Despite the large number of complex technical concepts covered in the draft, the presentation is clear and it is possible to follow and understand all components.\n\nWEAKNESSES:\nWhile all the proposed ideas are simple and intuitive, putting them all together yields a complex architecture with a large space of design choices and hyper-parameters. This model trains for 10 days, despite the efficient vectorized input and hierarchical architecture focused on efficiency. The runtime analysis only presents a comparison to an agent-centric baseline Wayformer, which fails to provide evidence for whether the proposed model is efficient among relative pose based forecasting methods (e.g., no evidence for the claim made in L092 that GNNs are more demanding than transformers in the online inference setting).\n\nQUESTIONS:\n1. Is the training time of HTPR similar to Wayformer for the same number of training epochs?\n2. How is the KNARPE operation implemented in practice? Do you still compute and mask a dense attention matrix, or implement custom kernels to only compute attention where needed?\n3. How important is the post-processing described in L254-257? Are these techniques commonly applied by methods on these leaderboards?\n4. Could you please elaborate on L261-262, what does sampling 25% and 50% mean in this context?\n5. Would it be possible to compare the inference time (Fig. 4) to a GNN method with relative pose encodings?\n\nMinor:\n\n1. Have you tried adding the blue dots from Fig. 1b to Fig. 1a as well? This could make it clearer to understand which agents are being used in Fig. 1b.\n2. Could Fig. 5 be simplified, in particular by removing the striking colors for the map elements? An alternative option would be to add a legend describing all colors.\n\nUpdate:\n\nThank you for the detailed responses to all questions. The rebuttal addresses all of my concerns, and I would like to maintain my positive rating.\n\nLIMITATIONS:\nLimitations are discussed in Section 5.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
--

STRENGTHS:
--

WEAKNESSES:
--

QUESTIONS:
--

LIMITATIONS:
--
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-13