What Truly Matters in Trajectory Prediction for Autonomous Driving?
Authors: Tran Phong
Haoran Wu
Cunjun Yu
Panpan Cai
Sifa Zheng
David Hsu
Venue: NeurIPS 2023 poster (2023)
Keywords: trajectory prediction; autonomous driving
TL;DR: we demonstrate that an interactive, task-driven evaluation approach for trajectory prediction is crucial in autonomous driving.

Abstract: Trajectory prediction plays a vital role in the performance of autonomous driving systems, and prediction accuracy, such as average displacement error (ADE) or final displacement error (FDE), is widely used as a performance metric. However, a significant disparity exists between the accuracy of predictors on fixed datasets and driving performance when the predictors are used downstream for vehicle control, because of a dynamics gap. In the real world, the prediction algorithm influences the behavior of the ego vehicle, which, in turn, influences the behaviors of other vehicles nearby. This interaction results in predictor-specific dynamics that directly impacts prediction results. In fixed datasets, since other vehicles' responses are predetermined, this interaction effect is lost, leading to a significant dynamics gap. This paper studies the overlooked significance of this dynamics gap. We also examine several other factors contributing to the disparity between prediction performance and driving performance. The findings highlight the trade-off between the predictor's computational efficiency and prediction accuracy in determining real-world driving performance. In summary,  an interactive, task-driven evaluation protocol for trajectory prediction is crucial to capture its effectiveness for autonomous driving. Source code along with experimental settings is available online (https://whatmatters23.github.io/).

What Truly Matters in Trajectory Prediction for Autonomous Driving?
Phong Tran
Haoran Wu
Cunjun Yu!
Panpan Cai
Sifa Zheng
David Hsu
National University of Singapore
2 Tsinghua University
Shanghai Jiao Tong University
Abstract
Trajectory prediction plays vital role the performance of autonomous driving systems, and prediction accuracy, such as average displacement error (ADE) final displacement eπror (FDE), is widely used performance metrie. However, a significant disparity exists between the accuracy of predictors on fixed datasets and driving performance when the predictors are used downstream for vehicle control, because of a dynamics gap. In the real world, the prediction algorithm influences the behavior of the ego vehicle, which, in turn, influences the behaviors of other vehicles nearby. This interaction results in predictor-specific dynamics that directly impacts prediction results. fixed datasets, since other vehicles' responses are predetermined, this interaction effect is lost, leading to significant dynamics this dynamics gap. We
gap. This paper studies the overlooked significance also examine several other factors contributing to the disparity between prediction performance and driving performance. The findings highlight the trade-off between the predictor's computational efficiency and prediction accuracy in determining real-world driving performance. In summary, an interactive, task-driven evalu- ation protocol for trajectory prediction is crucial to capture its effectiveness for autonomous driving. Source code along with experimental settings is available online!
1 Introduction
Current trajectory prediction evaluation 229 9 5 relies real worldworld datasets, operating under the assumption that dataset accuracy is equivalent to prediction capability. We refer to this as Static Evaluation. This methodology. however, falls short when the predictor serves sub-module for downstream tasks in Autonomous Driving (AD) 28 24]. As illustrated in Figure I the static evaluation metrics datasets, such Average Displacement Error (ADE) and Final Displacement Error (FDE), do not necessarily reflect the actual driving performance 36 g 10 In contrast to the conventional focus uncertainty [3 [6] and potential interaction (27. demonstrate that this
disparity stems from the overlooked dynamics gap between fixed datasets and AD systems and the computational efficiency of predictors. The trade-off between these two factors matters in the actual
prediction performance within the entire AD system.
The dynamics gap arises from the fact that the behavior of the autonomous vehicle, also known as the ego-agent, varies with different trajectory predictors, as presented in Figure 2) In real-world scenarios, the ego-agent utilizes predictors to determine its actions. Different predictors result in varied behaviors of the ego-agent, which, in turn, influence the future behaviors other road users, leading to different dynamics within the environment. This directly affects the accuracy of predictions, as other agents behave differently. Since the ego-agent' 's actions are predetermined on
"Equal contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

ADE us Devin Performance
Driing
Ditklng
LTDZ
LEDE
UIDE
WOD
OM
MADT
MGE
200 2 2INTET
Comto
Combart
Comtork
Sufety
Confect
T6E vs Driding Performance
HIVT
LaneGGN
LSTM
S-LSTM
JICE
Briiing
Drinng
LEBE
Lnving
MYDE
237
10C
Comt
Saltey
Combict
Sabry
Cantort
Cortee
Figure I: Prediction accuracy driving performance. Contrary popular belief (left, black curves), our study indicates no strong correlation between common prediction evaluation metrics and driving performance (left, red curves). Eight representative prediction models are selected: Constant Velocity (CV) [3], Constant Acceleration (CA) [30], K- K-Nearest Neighbor (KNN) [91, Social KNN (S-KNN) [9]. HiVT [4]] LaneGCN [22] LSTM, and Social LSTM (S-LSTM) 2]. The definition of driving performance metrics can be found in Section 4.4 Details can be found in the supplementary materials.
the dataset, there exists a significant disparity between the dynamics represented in the dataset and the actual driving scenario when evaluating a specific trajectory predictor. To tackle this issue, we propose the use of an interactive simulation environment evaluate the predictor for downstream decision-making. This environment enables Dynamic Evaluation the ego-agent operates with the specific predictor, thus, mitigating the dynamics gap. We demonstrate a strong correlation between the dynamic evaluation metrics and driving performance through extensive experiments. This underscores the dominant role the dynamics gap in aligning prediction accuracy with driving performance and emphasizes the importance of incorporating it into the evaluation process.
The awareness of the dynamics gap significantly enhances the correlation between prediction ac- curacy and driving performance. However, there are still factors that affect driving performance apart from prediction accuracy. The downstream modules in AD systems, with different tasks and varying levels of complexity, impose different requirements on prediction models. In our exper- iments, we vary the type of planner and the time constraint for the motion planning task. Our findings suggest that predictors' computational efficiency plays vital role in real-time tasks. Moreover, it is the trade-off between prediction accuracy and computational efficiency that deter- mines driving performance. This highlights the necessity for task-driven prediction evaluation.
fox
hea
Avi
Pren
Groent Trudt
Prodiction
Producio
ADE
(a)
(b)
Figure 2: Dynamics Gap. (a) In static evaluation, the agent's 's motion is determined and unaffected by predictors. (b) In the real-world, different predic- tors result in varied behaviors of the agent, which directly affects the ground truth of prediction.
In this paper, we aim to address two specitic aspects of trajectory prediction for AD systems. Firstly, uncover the limitation of static prediction evaluation systems in accurately reflecting driving per- formance. demonstrate the dominant role of the dynamics gap in causing the disparity. Secondly, we emphasize the necessity of interactive, task-driven evaluation protocol that incorporates the trade-off between dynamic prediction accuracy and computational efficiency. The protocol presents a promising way for applying prediction models in real-world autonomous driving, which is the truly mattered aspect in trajectory prediction for autonomous driving.
2 Related Work
2.1 Motion Prediction and Evaluation
Motion prediction methods can be classified along three dimensions 17 modeling approach, output type, and situational awareness. The modeling approach includes physics-based models [30]

---

that use physics to simulate agents" forward motion, and learning- barning-bbsed models [40] [2] that learn
and predict motion patterns from data. The oulput type can be intention, single-trajectory [26].
multi-trajectory [8. or occupancy map [1] 15 These outputs differ in the type of motion they
predict and how they handle the uncertainty of future states. The situational awareness includes unawareness, interaction [2], scene, and map awareness [1]. It refers to the predictor's ability to
incorporate environmental information, which is crucial for collision avoidance and efficient driving. Most researchers [29 255 and competitions
[9 [ evaluate the performance of prediction models on real-world datasets, which ADE/FDE and their probabilistic variants minADE/minFDD are commonly used metrics. However, these metries fail to capture the dynamics gap between datasets and real- werl-world scenarios, as the actions of the ego-agent remain unaffected by predictors. In this study, we select four booeel-based and six learning-basse based models with varying output types and situational awareness to cover a wide range of prediction models. We implement these predictors in both datasets and interactive simulation environment to illustrate the limitation of current prediction evaluation in accurately reflecting driving performance, due to the neglect of the dynamics gap.
2.2 Task-aware Motion Prediction and Evaluation
Twsk- motion prediction remains an underexplored area in research. While some studies touch upon the subject, they focus proposing task-aware metries for training climinating improper predictions on datasets. One notable example of training on task-aware metrics is the Planning KL. Divergence (PKL) metric (28) Although designed for 3D object detection, it measures the similarity between detection and ground truth by calculating the difference in ego-plan performance. In the context of motion prediction, Rowan et al. 24 propose control-aware metric (CAPO) similar to PKL. CAPO employs attention [35] to find correlations between predicted trajectories, assigning higher weights to agents inducing more significant reactions. Similarly, the Task-Informed method 19] assumes a set of candidate trajectories from the planner and adds the training loss for each candidate. Another line of work focuses on designing task-aware functions to eliminate improper predictions [3]. The proposed metric can capture unrealistic predictions and better correlate with driving performance in planner-agnostic settings. However, these works are conducted in an open-loop manner, neglecting the disparity between open-loop evaluation and real-world driving performance. To the best of our knowledge, we are the first to conduct thorough investigation into the disparity and demonstrate the dominant role of the dynamics gap on this issue.
3 Planning with Motion Prediction
3.1 Problem Formulation
In this section, present the problem formulation for motion prediction with planning in the context of autonomous driving. Different from conventional formulations that consider the planner as a policy, our formulation employs the predictor as the policy. This enables us to identify the primary challenge in developing predictors for real-world autonomous driving. We define the prediction problem as an MDP: M (S,A,T, R).
State. The state of the system represented by the tuple s = (8*90 1 _000) for s E S, where noo denotes the state of the AV including historical information, g=20 denotes that of surrounding traffic participants. Specifically, =000 { :............ st (90), }, where lots represents the observation horizon and denotes the current timestep. The state for each vehicle includes its position, velocity, heading, and other relevant attributes, e.g., s, ego = (x ego ego 1/ .0550).
Action. At each time step, the AV can take an action a E A, where A is the action space. We denote the n surrounding traffic participants as i E (1....,.7) n). all exo-agents, e.g., a = {al [a!.....a") a").
The action consists of the AV's prediction of
Transition Function. The transition function T: S x A S defines the dynamics of the system and how the system evolves as a result of the action of the ego vehicle. In the MDP, it refers to: T(s--1/8,, nx) = Ti (< |8,, a,) )2 (s ss,), T E TT We denote the planner as a mapping function Ti that takes the current state s and action a; input and outputs the AV's next state ego The motion models of exo-agents are integrated into the mapping function T2, which takes the current state 8, as input and generates the next state s442 for all exo-agents.
Predictor. The predictor is represented as a policy a = xxx) that takes the current state 8 as input, and produces an action A as output.
3

---

Objective Function. The objective of prediction is to accurately predict exo-agents future states while considering their interactions and map information. The reward function R S + R maps a state-action pair to real-valued reward. The objective function is the accumulated reward over the task horizon H, defined as ass Σ R(st,
Optimal Policy. The objective of autonomous driving identify the optimal policy that minimizes the objective function under the real-world transition function. The optimal policy x is expressed as: B 1 H R(s d,)], where I represents any realistic planners used in AVs, and t=
arg max
Ti 7i= -1211 T2 denotes the integrated motion model of exo-agents in the real world.
3.2 The Limitation of Static Evaluation
Traditional prediction methods overlook the difference between the transition function Tī in real- world autonomous driving and the represented Fi in datasets. These methods focus training policy using well-designed reward functions to perform effectively within the state distribution of datasets. The trained policy satisfies:
H E. Σ [()). TT=TTTTT
arg
(I)
In this context, 7i serves as static planner that remains unaffected by predictors. Regardless of the input, the static planner T2 generates the recorded next state. As a result, the mapping function Tý remains consistent with the real world since the input state is unchanged. We define the disparity in future states resulting from changes in the ego-planner as the dynamics gap in static evaluation, denoted as G = TT TT.
The proposed planning aware metrics, such PKL (28 and CAPO 2 aimed to address the
disparity between prediction accuracy and the ultimate driving performance by improving the cost function. However, unless the dynamics gap is mitigated, the optimization deviation plays dominant role causing the disparity and should be addressed as priority.
3.3 Measuring the Impact of Dynamics Gap
Conducting real-world tests is the most effective way to address and assess the impact of the dynamics gap. However, due to its high cost, the simulator serves as proxy. The ego-agent utilizes the realistic planner 7 and the motion model T sim of the simulator. The optimal policy क Tiim satisfies:
arg max
F.
R(s,,a)))
Ti=Ti T2=mi
To evaluate the impact of the dynamics gap, we train various predictors on the Alignment dataset obtained from the simulator, the trained policy simm satisfies:
H
Waim
arg Max
E.
R(s..
2=
The equations suggest that the dynamics gap between the Alignment dataset and simulator mirrors the dynamics gap between the real-world dataset and actual autonomous driving scenarios. Thus, is reasonable to employ the simulator for evaluating the influence of the dynamics gap. However, the simulator merely serves as a substitute to mitigate the dynamics gap, leaving a remaining dynamics gap represented by G' =T T
Experimental Setup
Our aim is to identify the key factors involved in trajectory prediction for autonomous driving and recognize their impact on driving performance by simulating real-world scenarios that vehicles might encounter. The ultimate goal is to introduce an interactive, task-driven prediction evaluation protocol. To achieve this objective, we need to determine four crucial components: /) motion prediction methods to be covered; 2) realistic planners to employ prediction models; 3) the simulator replicate interactive scenarios; 4) evaluation metrics to assess the effectiveness of the key factors involved in motion prediction with respect to real-world driving performance.

---

Table 1: Selected prediction methods.
Modeling Approach
Method
Output Type Interaction Aware
Scene Aware
Map Aware
CV
ST
CA
ST
Model-bused
KNN 9
MT
X
S-KNN
MT
LSTM
ST
S-LSTM 2
ST
Data-driven
1ivT
LaneCCN 22
MT
HOME [4
OM
DSP
FS
MT
"Abbreviations: ST: Sinjle-Traccctory. MT: Multi Multi-Trajcctory, OM: Occupancy Map.
4.1 Motion Prediction Methods
We select 10 representative prediction models to achieve comprehensive coverage of mainstream approaches, ranging from simple model- methods to complex data-driven approaches, as presented in Table Constant Velocity (CV) and Constant Acceleration (CA) [30] assume that the predicted agent maintains a constant speed or acceleration within the prediction horizon. K-Nearest Neighbor (KNN) predicts an agent's s future trajectory based on most similar trajectories, while Social-KNN (S-KNN) [9] extends it by also considering the similarity of surrounding agents. These methods are widely used as baselines given their widespread effectiveness
simple prediction cases. Social LSTM (S-LSTM) (2). HiVT [40], LaneGCN (22), and HOME [4 represent four distinct types neural networks: RNN, Transformer, GNN, and CNN. DSP utilizes a hybrid design of neural networks, representing state-of oftte-of-hheearr prediction models.
4.2 Planners
An ideal planner should: 7) be able to handle state and action uncertainty; 2) take into account multiple factors of driving performance, such safety (collision avoidance), efficiency (timely goal achievement). and comfort (smooth driving); 3) be aware of interactions with other agents; and 4) supports real-time execution. We select realistic planners based on these criteria: a simplistic planner that only meets 2) and 4), and sophisticated planner that satisfies all of /) 4). This allows us draw planner-agnostic conclusions.
a simplistic planner that solves the optimization problem in the velocity space under collision avoidance constraints. The planner uses motion predictions avoid possible collisions with the deterministic motions, and thus does not consider the state and action uncertainty. As the RVO planner does not maintain states between consecutive timesteps, it also cannot optimize its planning with respect to interactions with other agents. The objective function of the RVO planner involves safety and efficiency within a short time window, and the RVO planner executes in real time.
RVO. The RVO planner
DESPOT. The DESPOT planner 32 state-of-the-eee belief-space planning algorithm that address uncertainties near-optimally. To account for stochastic states and actions, we adopt the bicycle model, kinematic model with two degrees of freedom, and introduce Gaussian noise to the displacement. DESPOT considers the system state, context information, and the ego-agent's action to predict the future states of exo-agents. This allows consider the interaction between agents. Moreover, the objective function of DESPOT incorporates safety, efficiency, and comfort metrics, making it an ideal algorithm for planning complex and dynamic environments.
We use these planners to control the speed of the ego-agent while pure- mure-pursuit algorithm for adjusting the steering angle. Details can be found in the supplementary materials.
4.3 Simulator
In order to evaluate different prediction models, the ideal simulator should: /) provide real-world maps and agents; 2) model potential unregulated behaviors; 3) accurately mirror the interactions between agents; and 4) provide realistic perception data for effective planning. We choose the
SUMMIT simulator [6] for our experiments since it meets the criteria, as demonstrated in Table SUMMIT is sophisticated simulator based the Carla [2] framework, offering various real-world
5

---

Table
Comparison between simulators.
Simulator
Real- World Maps
Unregulated Behaviors
Dense Interactions
Realistic Visuals
SUMO
25
X
X
TrafficSim
33
x
X
TORCS
7
x
X
BARK
4
X
V
X
X
Apollo
X
X
Symphony 20
SUMMIT
maps and agents to create diverse and challenging scenarios. It uses realistic motion model to simulate interactions between agents and supports the simulation of crowded scenes, complex traffic conditions, and unregulated behaviors.
There are two distinct concepts of time in our experiments: simulation time and real time. The former corresponds to the duration of actions in the simulator, while the latter represents the wall time consumed by the planner. By default, the simulator runs in asynchronous mode. In this mode, the simulator and planner run individually, and the simulation time equal the real time, However, in this study, we employ the SUMMIT simulator in synchronous mode to accommodate slow predictors. In this mode, the simulator waits for the planner establish communication before proceeding to the next step. Since the simulation time for each execution step remains constant at 0.03 s, the ratio between simulation time and real time can be manually set by varying the communication frequency. known as the tick rate. For example, once the tick rate is set to 30 Hz, the simulation time equals the real time. But, when the tick rate is set to 3 Hz, the ratio between simulation time and real time 0.1.
4.4 Evaluation Protocols
Motion Prediction Performance Metries. Four commonly used prediction performance metrics employed in this study, as presented in Table 3 In our experiments, the consensus of K=6 is adopted. While ADE/FDE can be applied evaluate angle-trajectory prediction models, their probabilistic variants minADE and minFDE can be applied to evaluate multi-trajectory predictors.
Driving Performance Metrics. The driving performance is primarily determined by three factors: safety, comfort, and efficiency. Let represent the total timestep for each scenario.
Safety is typically evaluated in terms of the collision rate. The collision is determined whether the smallest distance between the ego-agent and surrounding agents is smaller than a threshold c, that is:
P'aatty = # Σ" [[mi ( [| - s|| e .s..... s..,.. } < c). where || is the L2 distance between the ego-agent's 's bounding box and the exo-agent's bounding box, ll the boolean indicator function. We set = 1m for our experiments since the DESPOT planner rarely causes real collisions.
Efficiency is evaluated by the average speed of the ego-agent over the whole scenario: PEfficieccy H DE 044 Comfort represented by the jerk of the ego-agent, which the rate of change of acceleration with respect to time: PComfort H Σ 84
1
Since these three metries are not comparable, normalized each metric to [0, 1] before calculating the driving performance. Furthermore, we standardized the direction of these metrics, with higher values indicating better performance:
Pm min(PPm) P max Pm.) min( P..)) 1" max P..) min( Pm
P.. min( (..))
(4)
The boolean indicator n. = T[m E (Efficiency]l is employed to adjust the direction of driving performance metrics. The ultimate driving performance is derived through the average of normalized metrics for safety, efficiency, and comfort. Pseudocode can be found in the supplementary materials.
Experimental Design. We conduct two types of experiments for both planners in the SUMMIT simulator: Fixed Prediction Ability and Fixed Planning Ability.
Fixed Prediction Ability: The planner is required to perform a fixed number of predic- tions within an interactive simulation environment, regardless of the predictor's execution speed. The objective clarify the factor that dominates the disparity between prediction performance and driving performance while ensuring the predictive ability of methods.

---

Table 3: Trajectory prediction metrics.
Metric Name
Metrie Equation
ADE
7 E Van 2:)2 + (yi 0:)² (3T 22) + (vr 0r) minkeK T Li=1 [x, 2, + (y 6,)
FDE
minADE
minFDE
minLeK V (TT 2r) + (ur 0x)
2. Fixed Planning Ability: The planner allocated varying time budgets simulate predictor running at different speeds. Three sub-expering sub-experiments are conducted with tick rates set at 30 Hz, 3 Hz, and Hz. The objective to investigate the factors, except for prediction accuracy, that represent the predictive ability of methods. This provides explanation for the remaining disparity between prediction accuracy and driving performance.
We collect 50 scenarios for each predictor in each experiment. For each scenario, we randomly select the start and end point for the ego-agent from one of the four real-world maps provided by the SUMMIT' simulator. A reference path of 50 meters maintained between the two points, and the ego-agent is instructed follow this path. A certain number of exo-agents including pedestrians, cyclists, and vehicles is randomly distributed within the environment. We implement all selected predictors in the simulator, except for HOME and DSP, due to their significantly longer running time, making the closed-loop evaluation infeasible. These two methods are solely employed in the Sim-Real Alignment, as shown in the supplementary materials. Notably, the RVO planner produces identical results for these two experimental settings since it conducts prediction only once per timestep.
To investigate the correlation between prediction accuracy and driving performance, selected motion prediction models on the Alignment dataset collected from the SUMMIT simulator. We collect 59,944 scenarios and separate them into two groups: 80% training and 20% validation. Each scenario consists of about 300 steps. Subsequently, it is filtered down to 50 steps by taking into account the number of agents and their occurrence frequency. The nearest three agents randomly selected to be the interested agent for prediction. The data collection was conducted on a server equipped with an Intel(R) Xeon(R) Gold 5220 CPU. We use four NVIDIA GeForce RTX 2080 Ti to speed up the running.
train all
Experimental Results
The experiment part is designed and organized to answer the following questions:]) Can the current prediction evaluation system accurately reflect the driving performance?; 2) What is the main factor causing the disparity between prediction accuracy and driving performance?; and 3) How do we propose evaluating predictors based on driving performance?
5.1 The Limitation of Current Prediction Evaluation
This section illustrates the limitation of current prediction evaluation systems in accurately reflecting realistic driving performance. We use ADE as an example, with FDE results provided in the supplementary materials. We refer to the ADE and minADE calculated in the Alignment dataset as Static ADE and Static minADE since they are obtained from static evaluation. Fixed prediction ability experiments are conducted for both RVO and DESPOT planners in the SUMMIT simulator.
Static ADE. As revealed Figure 3R, there is no significant correlation between Statie ADE and driving performance. Specifically, in the DESPOT planner, we observe a counterintuitive positive relationship. According to this finding. higher Static ADE would imply better driving performance. However, this hypothesis lacks realistic basis and should be rejected since it exceeds the 95% confidence interval. Static ADE can not serve as a reliable indicator of driving performance.
Static minADE. To consider the impact of multi- modal prediction, we further investigate the correlation between Static minADE and driving performance. As depicted in |Figure 3b. the multi- modal prediction accuracy better reflects the driving performance. However, in the RVO planner, the linear regression exceeds the 95% confidence interval, thereby weakening the credibility of the evaluation metric. Furthermore, we still observe the positive correlation between prediction error and driving performance in the DESPOT planner, despite the lack of a realistic basis. These two observations prevent us from asserting the efficacy of Static minADE.
7

---

100
Lmdgs
5.30
Figure 3: The correlation between static prediction metrics and driving performance for RVO (left) and DESPOT (right) planners. (a) Static ADE versus driving performance. (b) Static minADE versus driving performance. A substantial disparity between prediction accuracy and driving performance is observed, regardless of whether the accuracy is measured by Static ADE or Static minADE.
0.36
RC 0.68 pasale: 0,000
Darrag Petterune
Diovina Parfouance
CNN
0.70
Leu0ck
0.60
SHz
1122
ADE
Dyaonic ADE
20
(a)
Dytamnic
Figure 4: The correlation between Dynamic ADE and Driving Performance with fixed prediction ability. (a) In the RVO planner. (b) In the DESPOT planner. Due to the mitigation of the dynamics gap, strong correlation between Dynamic ADE and driving performance is observed for both planners.
Figure 5: The correlation between Dynamic ADE and Driving Perfor- mance with fixed planning ability. The correlation becomes weaker when the tick rate is set higher.
In summary, current evaluation metrics merely represent the vanilla accuracy of predictors and do not fully reflect the driving performance. It is urgent to identify the overlooked factor(s) on this issue.
5.2 The Dominant Factor: Dynamics Gap
The disparity between static evaluation metrics and driving performance can attributed to various factors. However, due to the lack of quantitative analysis, remains unclear which factor primarily contributes to this disparity. In this section, we utilize fixed prediction ability experiments to identify the dominant factor affecting the disparity between prediction accuracy and driving performance. We use Dynamic ADE as an example. Similar to Static ADE, Dynamic ADE is calculated as the average L2 distance between the forecasted trajectory and the ground truth. However, the dynamic prediction accuracy in the interactive simulator differs from that of the Alignment dataset, since agents behave differently. This leads to the difference between Dynamic ADE and Static ADE.
Dynamics Gap. Figure rrppotss the correlation between Dynamic ADE and driving performance for both RVO and DESPOT planners. Compared Statie ADE, Dynamic ADE displays significantly stronger correlation with driving performance for both planners. Quantitatively, using the correlation coefficient measurement, the dynamies gap accounts for 77.0% of the inconsistency between Static ADE and driving performance in the RVO planner, and 70.3% in the DESPOT planner. This emphasizes the dominant role of the dynamics gap addressing the challenge of implementing predictors in the real world.
Furthermore, we compare the dynamics gap with factors that influence prediction accuracy and potentially affect driving performance. Except for the multi-modal prediction, the asymmetry of prediction errors (21) and occlusion selected for comparison. We measure the significance of factors by assessing their impact on the correlation coefficient between prediction accuracy and driving performance. The disparity between static ADE and driving performance denoted Total.
Multi-Modaa Prediction. We examine the variation correlation coefficient from single-trajectory prediction to their multi-trajectory variants.

---

Table 4: Impacts on the correlation coefficient between prediction accuracy and driving performance.
ACorrelation Coefficient
Factor
RVO
DESPOT
Occlusion
-0.22
0.16
Prediction Asymmetry
0.17
0.20
Multi-Modal Prediction
0.70
0.45
Dynamies Gap
0.77
1.16
Total
1.00
1.65
Prediction Asymmetry. We check the change in correlation coefficient from considering all agents' predictions to considering only the interested agent that may affect the ego-plan.
Occlusion. We examine the variation in correlation coefficient from considering only agents with complete observations considering agents with missed observations.
As depicted in Table 4, considering the asymmetry of prediction errors, multi- prediction, and dynamics gap mitigate the disparity between prediction accuracy and driving performance. The consideration of occlusion does not yield satisfactory results in the RVO planner, as the planner mainly focuses short-term predictions and is minimally affected. The asymmetric of prediction erros has noticeable impact on both planners, highlighting the importance of predictions involving agents that may affect g3o-planning. The multi-modal prediction exerts a significant impact on the RVO planner. However, when a complex planner such as DESPOT is used in conjunction, the impact becomes much weaker. It is reasonable assume that the sampling-based feature of the DESPOT planner mitigates the influence of prediction uncertainty. Among various metrics, the dynamics gap exhibits superior performance in both planners and effectively resolves the majority of the disparity.
We can conclude that the dynamics gap is the main factor that causes the disparity between prediction accuracy and realistic driving performance. The Dynamic ADE, which evaluated through interactive simulation environments, is capable of incorporating the dynamics gap and displaying significant correlation with driving performance.
5.3 On the Predictive Ability of Methods
Although the dynamics gap accounts for the majority of the disparity between prediction accuracy and driving performance, the correlation between Dynamic ADE and driving performance is unsatisfactory in the DESPOT planner. To address this issue, we must answer two additional questions: /) Does prediction accuracy fully reflect the predictive ability of methods? 2) If not, what other factor(s) most important? We examine three well-known but never fully investigated factors and their correlation with driving performance: the temporal consistency of predictions [37], the distribution of prediction errors, and the computational efficiency of predictors.
Temporal Consistency. Each prediction scenario contains multiple successive frames within fixed temporal chunk. Any two overlapping chunks of input data with small time-shift should produce consistent results. The difference between the predictions of the two overlapping chunks is measured as the temporal consistency, with time-shift set to one frame.
Error Distribution. The error distribution of motion prediction contains the mean and variance. While the mean has been captured by the prediction accuracy, we incorporate the variance of prediction errors to account for the remaining impact.
Computational Efficiency. The computational efficiency of predictors is determined by their inference time when applied to downstream tasks. Feature extraction time also included.
According to Figure 6 surprisingly, the temporal consistency of predictions and the distribution of prediction errors show a weak correlation with driving performance in both planners. However, the computational efficiency of predictors exerts a significant influence on the driving performance, as shown in Table 5) three sub-experiments, the ranking of driving performance aligned with the ranking of predictors' inference time in the DESPOT planner. We can conclude that the prediction accuracy can not fully represent the predictive ability methods, and the computational efficiency of predictors is also significant metric that affects driving performance.

---

Figure 6: Contrary to common belief, the correlation between temporal consistency/error distribution and driving performance is not not strong for both RVO (left) and DESPOT (right) planners. (a) Temporal Consistency versus Driving Performance. (b) Error Distribution versus Driving Performance.
Table 5: The correlation between predictors' computation efficiency and driving performance.
Driving Performance (1)
Method
Inference Time (L)
30Hz
3Hz
1Hz
CV
0.001s
0.753
0.771
0.761
CA
0.001s
0.739
0.774
0.779
LSTM
0.010s
0.644
0.753
0.766
S-LSTM
0.014s
0.649
0.715
0.733
HiVT
0.024s
0.616
0.730
0.753
LaneGCN
0.024s
0.637
0.711
0.742
KNN
0.224s
0.526
0.692
S-KNN
0.248s
0.677
5.4 The Trade-Off between Accuracy and Speed
It should be noted that there is trade-off between computational efficiency and dynamic prediction accuracy for predictors to derive driving performance. As shown in Figure 5] the correlation between Dynamic ADE and driving performance becomes less strong when the tick rate is higher. This is indicated by the data points deviating further from the best-fit-line in higher tick rates. At this time, is the computational efficiency rather than dynamic prediction accuracy that decides the driving performance, as shown in Table 5] When the tick rate is set to 30Hz, the planner cannot generate an optimal solution, whereby the ranking of driving performance is determined by computational efficiency. When the tick rate is set to 3Hz, the CA outperforms CV since they have near-optimal solutions. When the Tick Rate set 1Hz, the L.STM also outperforms CV. The driving performance is determined by both dynamic prediction accuracy and computational efficiency of predictors in trade-off manner, highlighting the significance of task-driven prediction evaluation.
Conclusion
Our study demonstrates the limitation of current prediction evaluation systems in accurately reflecting realistic driving performance. We identify the dynamics gap as the dominant factor contributing to this disparity. Furthermore, our findings reveal that the ultimate driving performance is determined by the trade-off between prediction accuracy and computational efficiency, rather than solely relying on prediction accuracy. We recommend further research incorporating interactive and task-driven evaluation protocols to assess prediction models. Despite these insights, there is still work to be done in the future. Our study focuses sampling-basee and reactive planners, and incorporating optimization- based or geometric planners could further substantiate our conclusions. Oracle percep- tion data from the simulator not fully represent the complexities of real-works situations. It would be beneficial to use raw sensor data to understand the comprehensive interplay within the entire AD system.
Acknowledgments and Disclosure of Funding
This research is supported part by National Key R&D Program of China under Grant 2020YFB 1600202, National Research Foundation of Singapore under its Al Singapore Programme (AISG Award No: AISG2-PhD SSSG2-PDD-2022-011 2022-01-036/T), and China Scholarship Council.
10

---

References
[1] Baidu Apollo team (2017), Apollo: Open Source Autonomous Driving. https://githu: comm/Appo1loAttooapplloo
[2] Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social Istm: Human trajectory prediction crowded spaces. Proceedings of the IEEE conference computer vision and pattern recognition, pages 961-971, 2016.
[3] Ivo Batkovic, Mario Zanon, Nils Lubbe, and Paolo Falcone. A computationally efficient model for pedestrian motion prediction. In 2018 European control conference (ECC), pages 374-379.
IEEE, 2018.
[4] Julian Bernhard, Klemens Esterle, Patrick Hart, and Tobias Kessler. Bark: Open behavior benchmarking in multi-agent environments. 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6201-6200 IEEE, 2020.
[5] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijhom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621-11631 2020.
[6] Panpan Cai, Yiyuan Lee, Yuanfu Luo, and David Hsu. Summit: simulator for urban driving in massive mixed traffic. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 4023-4023 IEEE, 2020.
[7] Luigi Cardamone, Daniele Loiacono, and Pier Luca Lanzi. On-line neuroevolution applied to the open racing car simulator. In 2009 IEEE Congress on Evolutionary Computationn pages 2622 2629. IEEE, 2009.
[8] Vinton G Cerf. A comprehensive self-driving car test. Communications of the ACM, 61(2). 2018.
[9] Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, al. Argoverse: 3d tracking and forecasting with rich maps. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8748-8757, 2019.
[10] Felipe Codevilla, Antonio M Lopez, Vladlen Koltun, and Alexey Dosovitskiy. On offline evaluation of vision-based driving models. In Proceedings of the European Conference on Computer Vision (ECCV), pages 236-251, 2018.
[11] Craig Coulter. Implementation of the pure pursuit path tracking algorithm. Technical report, Carnegie-Mellon UNIV Pittsburgh PA Robotics INST, 1992.
[12] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator, 2017.
[13] Alec Farid, Sushant Veer, B. Ivanovic, Karen Leung, and M. Pavone. Task- Resk-Relevant Failure Detection for Trajectory Predictors in Autonomous Vehicles. 2022.
[14] Thomas Gilles, Stefamo Sabatini, Dzmitry Tsishkou, Bogdan Stanciulescu, and Fabien Moutarde. Home: Heatmap output for future motion estimation. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pages 500-507. IEEE, 2021.
[15] Thomas Gilles, Stefano Sabatini, Dzmitry Tsishkou, Bogdan Stanciulescu, and Fabien Moutarde. Gohome: Graph-oriented heatmap output for future motion estimation. In 2022 International Conference on Robotics and Automation (ICRA), pages 9107-9114, IEEE, 2022.
[16] Junru Gu, Chen Sun, and Hang Zhao. Densetnt: End-to-end trajectory prediction from dense goal sets. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 15303-15312, 2021.
[17] Mahir Gulzar, Yar Muhammad, and Naveed Muhammad. survey on motion prediction of pedestrians and vehicles for autonomous driving. IEEE Access, 9: 9:177977-337969, 2021.
11

---

[18] Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2255-2264, 2018.
[19] Xin Huang, Guy Rosman, Ashkan Jasour, Stephen G McGill, John J Leonard. and Brian C Williams. Tip: Fask-informed motion prediction for intelligent vehicles. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1 11432-11439 IEEE, 2022.
[20] Maximilian Igl, Daewoo Kim, Alex Kuefler, Paul Mougin, Punit Shah, Kyriacos Shiarlis, Dragomir Anguelov, Mark Palatucci, Brandyn White, and Shimon Whiteson. Symphony: Learning realistic and diverse agents for autonomous driving simulation. In 2022 International Conference on Robotics and Automation (ICRA), pages 2445-24424222 2451. IEFE, 2022.
[21] Boris Ivanovic and Marco Pavone. Injecting Blanning-awarent into prediction and detection evaluation. In 2022 IEEE Intelligent Vehicles Symposium (IV), pages 821 IEEE, 2022.
[22] Ming Liang, Bin Yang. Rui Hu, Yun Chen, Renjie Liao, Song Feng. and Raquel Urtasun. Learning lane graph representations for motion forecasting, In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 16, pages 541-556. Springer, 2020.
[23] Pablo Alvarez Lopez, Michael Behrisch, Laura Bicker-Wald Jakob Erdmann, Yun-Pang Flöt- teröd, Robert Hilbrich, Leonhard Lücken, Johannes Rummel, Peter Wagner, and Evamaric WieBner. Microscopic traffic simulation using sumo. In 2018 21st international conference on intelligent transportation systems (ITSC), pages 2575 25822. IEEE, 2018.
[24] Rowan McAlliste, Blake Wulfe, Jean Mercat, Logan Ellis, Sergey Levine, and Adrien Gaidon. Control-aware prediction objectives for autonomous driving. In 2022 International Conference on Robotics and Automation (ICRA), pages 01-08. IEEE, 2022.
[25] Sajjad Mozaffari, Omar Y Al-Jarrah, Mehrdad Dianati, Paul Jennings, and Alexandros Mouza- kitis. Deep learning-based vehicle behavior prediction for autonomous driving applications: A review. JEEE Transactions on Intelligent Transportation Systems, 23(1):33-47, 2020.
[26] Andreas Møgelmose, Mohan M. Trivedi, and Thomas B. Moeslund. Trajectory analysis and prediction for improved pedestrian safety: Integrated framework and evaluations. In 2015 IEEE Intelligent Vehicles Symposium (IV), pages 330-335, June 2015.
[27] Jiquan Ngiam, Vijay Vasudevan, Benjamin Caine, Zhengdong Zhang, Hao-Tien Lewis Chiang, Jeffrey Ling, Rebecca Roelofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, et al. Scene transformer: A unified architecture for predicting future trajectories of multiple agents. In International Conference on Learning Representations, 2021.
[28] Jonah Philion, Amlan Kar, and Sanja Fidler. Learning to evaluate perception models using planner-centric metrics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 14055-14064, 2020.
[29] Andrey Rudenko, Luigi Palmieri, Michael Herman, Kris M Kitani, Dariu M Gavrila, and Kai O Arras. Human motion trajectory prediction: A survey. The International Journal of Robotics Research, 39(8):895-995 9355, 2020.
[30] Christoph Schöller, Vincent Aravantinos, Florian Lay, and Alois Knoll. What the constant velocity model can teach us about pedestrian motion prediction. IEEE Robotics and Automation Letters, 5(): 2020.
[31] Marcel Schreiber, Vasileios Belagiannis, Claudius Gläser, and Klaus Dietmayer. Dynamic Occupancy Grid Mapping with Recurrent Neural Networks. In 2021 IEEE International Conference Robotics and Automation (ICRA), pages 6717-6724, May 2021.
[32] Adhiraj Somani, Nan Ye, David Hsu, and Wee Sun Lee. Despot: Online pomdp planning with regularization. Advances in neural information processing systems, 26, 2013.
12

---

[33] Simon Suo, Sebastian Regalado, Sergio Casas, and Raquel Urtasun. Trafficsim: Learning simulate realistic multi-agen, behaviors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10400-10409, 2021.
[34] Jur Van den Berg, Ming Lin, and Dinesh Manocha. Reciprocal velocity obstacles for real-time multi-agent navigation. In 2008 IEEE international conference on robotics and automation, pages
Icee, 2008.
[35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[36] Nick Webb, Dan Smith, Christopher Ludwick, Trent Victor, Qi Hommes, Francesca Favaro, George Ivanov, and Tom Daniel. Waymo's s safety methodologies and safety readiness determi- nations. arXiv preprint arXiv2001..00054 2020.
[37] Maosheng Ye, Jiamiao Xu, Xunnong Xu, Tengfei Wang, Tongyi Cao, and Qifeng Chen. Boot- strap motion forecasting with self-consistent constraints, 2023.
[38] Lu Zhang, Peiliang Li, Jing Chen, and Shaojic Shen. Trajectory prediction with graph- based dual-scale context fusion. 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1 11374-1153 11381. IEEE, 2022.
[39] Hang Zhao, Jiyang Gao, Tian Lan, Chen Sun, Ben Sapp, Balakrishnan Varadarajan, Yue Shen, Yi Shen, Yuning Chai, Cordelia Schmid, et al. Tnt: Target-driven trajectory prediction. In Conference on Robot Learning, pages 895-904. PMLR, 2021.
[40] Zikang Zhou, Luyao Ye, Jianping Wang. Kui Wu, and Kejie Lu. HiVT: Hierarchical Vector Transformer for Multi-Agent Motion Prediction. In 2022 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 8813-8823, June 2022.
13


Reviews:
[{"review_id": "NAWZ5wh53i", "reviewer": "Reviewer_Aygk", "summary": "In the article, the author discusses the limitations of the current open-loop trajectory evaluation method: 1. It lacks consideration of closed-loop interactions with the ego-planner 2. It overlooks computational timeliness. These two factors result in a misalignment between open-loop trajectory evaluations and closed-loop driving scores. The author also proposes a new metric, Dynamic ADE/FED, which enables a positive correlation to be established between open-loop trajectory evaluations and closed-loop driving scores.", "questions": "Please refer to the 'weaknesses' section.", "limitations": "Please refer to the 'weaknesses' section.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* Originality: This paper is original, providing a new evaluation method for the field of trajectory prediction.\n* Significance: The foundation of this paper is sound. It focuses on analyzing the flaws and issues in the standards of trajectory prediction evaluation, and conducts a substantial amount of experiments across different models.\n* Quality & Clarity: I believe this section is the weakness of the paper. Please refer to the 'Weakness' part.", "weaknesses": "* Clarity & Quality: \n    - 1. The core Dynamic ADE/FDE metrics in this paper are not clearly explained; there is hardly any section dedicated specifically to elaborating on this aspect. It would be beneficial to include an illustration for better understanding. \n\n    - 2. Despite the numerous experiments conducted, some of them fail to support the arguments made. For instance, on lines 243-245, you stated that the positive correlation demonstrated by different prediction models on the Argoverse and Alignment datasets indicates that there isn't a significant domain gap between the SUMMIT simulator and real-world data. I find this reasoning flawed. A positive correlation merely indicates consistency in the predictive ability of the models across the two datasets; it does not necessarily imply that the behaviors in the two datasets are consistent. Moreover, a closer look at the specific numerical values on the x and y-axes in Figure 2 reveals significant discrepancies between the Argoverse and Alignment datasets. The ADE distribution for Argoverse lies between 1.5-3.5, while that for Alignment falls between 0.5-1.5. This suggests that the Argoverse dataset is substantially more challenging than the Alignment dataset, which contradicts your claim. \n    - 3. Besides this, there are many other similar issues. The overall experimental conclusions lack substantial empirical support and are simply based on statistical analyses of a small number of data points, yielding correlations that aren't particularly strong (the paper only employs eight different prediction models and two different planning models). I remain unconvinced by statistical conclusions drawn from such a limited set of data points. \n* Novelty: \n    - 4. This paper seems more akin to an experimental report. Despite the multitude of experiments conducted, they remain rather superficial. For a NeurIPS paper, a deeper level of analysis is required, such as quantitative analysis, comparing what specific interactions in which scenarios cause differences between static and dynamic ADE in open-loop and closed-loop evaluations. Additionally, as I previously mentioned, many of the experimental results do not effectively support the conclusions drawn. \n    - 5. The work done in the 'Computational Efficiency' section is quite trivial. The conclusions drawn here are rather obvious, and in the field of autonomous driving engineering, this aspect has already been thoroughly considered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful comments! We kindly ask the reviewer to let us know if further clarification is needed.\n\n>The core Dynamic ADE/FDE metrics are not clear… It would be beneficial to include an illustration.\n\nThis is a great point for us to clarify more! We add a figure in the attached PDF. Similar to static ADE, dynamic ADE is also calculated by the average L2 distance between the forecasted trajectory and the ground truth. However, in dynamic evaluation, different predictors result in varied behaviors of the ego-agent, which, in turn, influence the future behaviors of other road users, leading to different dynamics within the environment. This directly affects the ground truth of prediction as other agents behave differently.\n\n>Some experiments fail to support the arguments… A positive correlation does not necessarily imply that behaviors in the two datasets are consistent.\n\n**The argument we support is the consistency of prediction performance across Argoverse and Alignment datasets**. By guaranteeing this, real-world prediction performance is provided for experiments. Real-world behavior is a means to gather actual driving performance, as conducting real-world tests is often unaffordable, we employ SUMMIT to conduct our research. The SUMMIT simulator, built upon Carla, a widely-used simulator in recent competitions [4] and research [5,6], offers behavior approximating reality.\n\n>The ADE for Argoverse lies … while that for Alignment lies… Argoverse is more challenging than Alignment.\n\nThe prediction results for a given predictor can vary, even when applied to two real-world datasets. However, such variations do not necessarily imply that one dataset is more challenging than the other. (SOTA of nuScenes: minADE5=1.092m in 6s prediction horizon, SOTA of Waymo: minADE6=0.535m in 8s prediction horizon) Nevertheless, the underlying behavior of drivers should be similar across real-world datasets. The observed differences can be influenced by many factors, e.g. average speed, number of agents, or other unidentified variables. **We argue that if the accuracy ranking of predictors remains consistent (as indicated by ADE/FDE alignment), the prediction ability of models are aligned, enabling us to evaluate predictors with SUMMIT**.\n\n>Conclusions lack solid empirical backing, relying on limited data points for statistical analysis, resulting in weak correlations.\n\n**We want to clarify that we have conducted extensive experiments to support our claim**. Specifically, we collected 50 scenarios for each predictor in each setting, resulting in a total of 1600 simulation scenarios. To ensure clarity of results, we only utilize the average performance for each predictor, and the correlation to driving performance is strong. **Additionally, the incorporation of a predictor or planner is expensive**. Assuming a predictor requires 0.02s to execute once, the DESPOT planner calls the predictor 1000 times at each step to explore adequate tree nodes, leading to 20s per planning step. Given the planner's requirement to operate at 3Hz, we set tick rate to 1Hz, allowing for 10s real time for each planning. Each scenario comprises around 200 steps, the total runtime for one predictor sums up to 200\\*50\\*10/3600=28 days. As many predictors are even much slower than 0.02s (e.g., KNN: 0.224s), it is infeasible to exhaust all possible predictors given the computational cost, not to mention the implementation time required.\n\n>Despite the multitude of experiments conducted, they remain rather superficial.\n\n**This paper shed light on the equally essential need for evaluating seemingly static modules, such as prediction, through closed-loop evaluation**, which has been largely overlooked in most studies [1,2] and SOTA of Argoverse Competition [3]. In addition, we reveal the **trade-off between computational efficiency and accuracy even when the predictor is fast enough for the planner's execution. We are the first to provide a comprehensive investigation on these problems**. We further analyze the correlation between other two possible factors and driving performance to support our core research in the attached PDF.\n\n>The work done in Computational Efficiency is trivial. The conclusions are obvious and have been thoroughly considered.\n\nThe important idea we want to point out is **the trade-off between computational efficiency and accuracy even when the predictor is fast enough for the planner's execution**, which has been neglected by most research. In the context of autonomous driving, a consensus exists that the whole perception system must meet a specified threshold for proper planning. Once this threshold is satisfied, the focus shifts to ensuring accuracy. (e.g., 100ms [7,8]) However, our experiments demonstrate a notable trade-off that persists much below the specified threshold. When the predictor runs much faster than the threshold (2ms), the driving performance still remains dominated by the fastest prediction method (CV). Only with an ample planning time budget (Tick Rate = 1Hz), methods with better accuracy dominate. This trade-off holds crucial importance, as various planners exhibit unique trade-offs in addition to their fundamental requirement.\n\n[1]Hu et al. Planning-oriented autonomous driving. CVPR 2023\n\n[2]Liang et al. Learning lane graph representations for motion forecasting. ECCV 2020\n\n[3]Zhou et al. Query-Centric Trajectory Prediction. CVPR 2023\n\n[4]NeurIPS CARLA Autonomous driving challenge, 2022\n\n[5]Danesh et al. LEADER: Learning Attention over Driving Behaviors for Planning under Uncertainty. CoRL 2023\n\n[6]Ulfsjöö et al. On integrating POMDP and scenario MPC for planning under uncertainty–with applications to highway driving. IV 2022\n\n[7]Lin et al. The architectural implications of autonomous driving: Constraints and acceleration. ASPLOS 2018\n\n[8]Yamaguchi et al. In-vehicle distributed time-critical data stream management system for advanced driver assistance. JIP 2017"}, {"review_id": "cK3wOwNGZo", "reviewer": "Reviewer_Bibu", "summary": "This work attempts to look for evaluation metrics for trajectory prediction beyond the widely used ADE/FDE metrics. In particular, the authors focus on two overlooked aspects in prediction evaluation, which are: 1) the dynamics gap between the dataset and closed-loop driving scenarios; 2) the trade-off between computational efficiency and performance. The authors argue that we should turn to closed-loop prediction evaluation to address these two problems.", "questions": "1. I do not think the authors appropriately answer the question raised in the title (i.e., what truly matters in trajectory prediction for autonomous driving?). The paper focuses on two perspectives on this question, which are very limited and far from compiling a comprehensive and insightful answer. The experimental setting is also insufficient as an attempt to answer such a bold question (e.g., lacking analysis on the impact of simulation on closed-loop evaluation). \n2. It would be beneficial to have some qualitative analysis and visualization beyond statistics. For instance, visualizing some representative examples could be helpful in understanding why static ADE and dynamic ADE are not correlated. \n3. The citation for SUMMIT is missing in the main text.", "limitations": "The authors discussed some limitations and future directions in Sec. 6. However, I think the limitations are not sufficiently addressed. For instance, I would consider the limited evaluation on a single simulator and the missing discussion on the impact of simulation realism as crucial drawbacks of this work.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This work focuses on a very important and open question for trajectory prediction, which is looking for evaluation metrics for trajectory prediction beyond the widely used ADE/FDE metrics. It is good that the authors emphasize and validate the limitation of open-looped prediction accuracy and the importance of closed-loop evaluation for trajectory prediction.", "weaknesses": "1. It is good to see that the authors showed the limitation of open-looped evaluation and the importance of closed-loop evaluation in their experiments. However, I did not think the authors provided any new perspective or insightful solution to the problem of closed-loop evaluation. The value of closed-loop evaluation has been widely acknowledged in the autonomous driving community. It is one of the main motivations behind the emerging research on data-driven traffic simulation. The bottleneck is not the lack of prediction evaluation metrics under the closed-loop evaluation scheme, which is targeted by one of the two claimed contributions of the paper (i.e., dynamic ADE vs. static ADE), but the closed-loop simulation itself. How to synthesize a reliable simulator with realistic reactive agent behavior is an open question. We even do not have a commonly acknowledged set of metrics for evaluating simulation at this stage. Unfortunately, the authors did not discuss the impact of the simulation environment on the closed-loop evaluation result in this paper (e.g., whether the simulated driving performance and the simulated dynamic ADE metric are correlated with real-world driving performance). While the authors adopted different prediction models and planners in their evaluation, they only used a single simulator (i.e., SUMMIT). Moreover, the authors did not include a literature review on traffic simulation and evaluation metrics for traffic simulation. \n\n2. The inherent logic behind the experiments is fundamentally flawed. In Sec. 5.2, the authors showed that static ADE is not correlated with dynamic ADE and driving performance. However, the authors used the correlation of static ADEs on the Argoverse and the Alignment dataset collected from SUMMIT as evidence to validate the realism of SUMMIT. Just like the authors' observation in Sec. 5.2, the correlation in static ADE does not necessarily indicate that the SUMMIT simulation is reliable and realistic under closed-loop simulation and evaluation. To justify the realism of SUMMIT, the authors should adopt those evaluation metrics from the latest literature and compare SUMMIT against those state-of-the-art simulation models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments! We kindly ask the reviewer to let us know if further clarification is needed.\n>I did not think the authors provide any new perspective or insightful solution. The value of closed-loop evaluation has been widely acknowledged in autonomous driving.\n\nWe disagree with the question’s premise. While the value of closed-loop evaluation has been widely recognized, **its emphasis mainly centered on planning and control [7,8,9]**. It would be great if the reviewer can provide a pointer for prediction. Our work aims to shed light on the equally essential need for evaluating seemingly static modules, such as prediction, through closed-loop evaluation, which has been largely overlooked in most studies [1,2] and SOTA of competitions [3].\n\n>The bottleneck is not the lack of prediction evaluation metrics under the closed-loop evaluation, but synthesizing a reliable simulator with realistic reactive agent behavior.\n\nNot quite. As we have claimed in A1, the primary motivation behind this work is to draw attention to the importance of evaluating seemingly static prediction models with closed-loop simulation. We challenge the current evaluation system and propose a better evaluation protocol. The dynamic ADE is presented as an additional contribution. Synthesizing a real simulator falls beyond the scope of this research and suffices as a standalone research.\n\n>Unfortunately, the authors do not discuss the impact of the simulation environment… They only use a single simulator.\n\n**The impact of the simulation environment is not related to our core contribution**. To cope with realistic scenarios, the only way is to drive in the real-world, which is often unaffordable. Thus, simulators are usually taken as the proxy to carry out research. We would like to reiterate that our contribution is to point out that static prediction evaluation is not correlated with driving performance, which is stated in some papers [4,5], but is never adequately discussed and validated. We conduct extensive experiments with representative predictors and planners to support this claim. In addition, SUMMIT is built upon Carla, the widely-used simulator in recent competitions [6] and research [10,11].\n\n>Moreover, the authors did not include a literature review on traffic simulation and evaluation metrics for it.\n\n**Although traffic simulation is not relevant to our core contribution**, we agree that incorporating a comparison of simulators and explaining our choice of SUMMIT will strengthen our paper. We provide a literature review in the general response.\n\n>The inherent logic behind the experiments is fundamentally flawed… To justify the realism of SUMMIT, the authors should adopt those evaluation metrics against those SOTA simulation models.\n\n**The objective of this experiment is not to validate the realism of SUMMIT, but to ensure the consistency of prediction performance across different environments**. Even in different real-world datasets, the performance of the same predictor will differ. Thus, what we care about is predictors’ ranking of accuracy. We conduct the alignment experiment to demonstrate that the ranking of prediction metrics keeps stable across real-world dataset Argoverse and simulation dataset Alignment.\n\n>I do not think the authors appropriately answer the question raised in the title (i.e., what truly matters in trajectory prediction for autonomous driving?).\n\nWe partially agree that our paper does not fully answer what truly matters in motion prediction. Our work focuses on proving that static evaluation does not correlate with driving performance, and advocates the superiority of dynamic evaluation. **What truly matters in motion prediction is a bold question to answer**. Expecting one paper to completely resolve this issue is unrealistic. Nevertheless, our work takes a stride forward, and our experiments fully support the current lack of prediction evaluation, leaving this question open for further exploration. With that being said, we concur that a more precise title would enhance the understanding of our main idea, and we will modify our title after further discussions.\n\n>It would be beneficial to have some qualitative analysis and visualization beyond statistics. \n\nWe add the visualization for showing why Static ADE and Dynamic ADE are not correlated in the attached PDF. In static evaluation, the ground truth is determined based on dataset records and remains unaffected by predictors. However, in dynamic evaluation, different predictors result in varied behaviors of the ego-agent, which, in turn, influence the future behaviors of other road users, leading to different dynamics within the environment. This directly affects the ground truth of prediction as other agents behave differently, thus causing the disparity between Static ADE and Dynamic ADE.\n\n>The citation for SUMMIT is missing.\n\nThank you for pointing it out. We will add the citation [12] in final version.\n\n[1]Hu et al. Planning-oriented autonomous driving CVPR 2023\n\n[2]Liang et al. Learning lane graph representations for motion forecasting ECCV 2020\n\n[3]Zhou et al. Query-Centric Trajectory Prediction. CVPR 2023\n\n[4]Ivanovic et al. Injecting planning-awareness into prediction and detection evaluation IV 2022\n\n[5]Ivanovic et al. Rethinking trajectory forecasting evaluation. arXiv 2021\n\n[6]NeurIPS CARLA Autonomous driving challenge, 2022\n\n[7]nuPlan Planning Challenging: Closed-loop reactive agents, 2023\n\n[8]Phan-Minh et al. Driving in Real Life with Inverse Reinforcement Learning. ArXiv 2023\n\n‌[9]Cheng et al. MPNP: Multi-Policy Neural Planner for Urban Driving. IROS 2022\n\n[10]Danesh et al. LEADER: Learning Attention over Driving Behaviors for Planning under Uncertainty CoRL 2023\n\n[11]Ulfsjöö et al. On integrating POMDP and scenario MPC for planning under uncertainty–with applications to highway driving IV 2022\n\n[12]Cai et al. Summit: A simulator for urban driving in massive mixed traffic ICRA 2020"}, {"review_id": "6o9YsuD6qp", "reviewer": "Reviewer_tS95", "summary": "The paper provides an extensive study on the discrepancy between model prediction accuracy and driving performance. It explores two major factors: 1. The dynamics difference; and 2. The computational efficiency of predictors. Various methods are tested in two settings: 1. Fixed Number of Predictions; and 2. Fixed Planning Time. The paper evaluates the model performance on both open-loop prediction and closed-loop diving.", "questions": "It would be great if the author could provide add some discussions on possible other factors that cause the discrepancy and why the two factors mentioned in the paper are dominant.", "limitations": "See questions.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The experiments are well-designed and extensive, which provides strong evidence for the statements in the paper.", "weaknesses": "While the two factors are well supported by the experiment results given, it is insufficient to say whether other factors are less important, such as the data quality of the open loop eval, the unknowability (future events influencing the ground truth future trajectory) issue for open loop prediction, as well as multimodality in the diving scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thoughtful analysis and constructive feedback on our paper. Your insights have been valuable, and we are grateful for the time you spent evaluating our work. Below, we address your main concerns:\n\n> While the two factors are well supported by the experiment results given, it is insufficient to say whether other factors are less important, such as the data quality of the open loop eval, the unknowability (future events influencing the ground truth future trajectory) issue for open loop prediction, as well as multimodality in the diving scenarios.\n\nThank you for raising this insightful point. We acknowledge the importance of considering factors beyond the two emphasized in our study. Your comment has prompted us to further investigate the correlation between temporal consistency [1] and driving performance, focusing on how sudden changes in prediction might impact driving outcomes.\n\nAdditionally, we have examined the relationship between prediction variance and driving performance to deepen our understanding of the influence of multimodality. The detailed findings from these analyses are provided in the PDF file, which will be available on August 9th \"global response\" . We believe that these additional investigations will contribute to a more comprehensive understanding of the complex interplay between these various factors.\n\n> It would be great if the author could provide some discussions on possible other factors that cause the discrepancy and why the two factors mentioned in the paper are dominant.\n\nWe appreciate your suggestion to delve into other possible factors contributing to the observed discrepancies. One common and critical factor is the asymmetry of prediction error [2]. Even when two prediction models exhibit the same ADE, their driving performances will not be identical. This discrepancy arises from the different influence of prediction errors to planning. For instance, errors associated with the closest exo-agent can have a higher impact on the ego-agent's plan. The conflict between the predicted state of the exo-agent with the ego-agent's plan also matters. \n\n**However, it is important to note that the asymmetry of prediction error is not universal, which means that it occurs only in specific scenarios or for specific agents**. As a result, the influence of this factor is less significant than our proposed dynamic evaluation and computational efficiency, both of which impact prediction evaluation in every scenario and at each timestep. To provide quantitative evidence, we have effectively accounted for the most critical aspect of asymmetry prediction error by comparing \"Closest Dynamic ADE\" with \"Dynamic ADE.\" Consequently, the influence of asymmetry on the correlation between ADE and Driving Performance is considerably less than the impact of dynamic evaluation. (Influence For RVO Planner: 0.16 vs 0.61; Influence For DESPOT Planner: 0.20 vs 0.96)\n\n\n[1] Ye, Maosheng, et al. “DCMS: Motion Forecasting with Dual Consistency and Multi-Pseudo-Target Supervision.” ArXiv.org, 27 Feb. 2023, arxiv.org/abs/2204.05859.\n\n[2] Ivanovic, Boris, and Marco Pavone. “Injecting Planning-Awareness into Prediction and Detection Evaluation.” IEEE Xplore, 1 June 2022, ieeexplore.ieee.org/document/9827101/."}, {"review_id": "KTHWPH2JVQ", "reviewer": "Reviewer_RCyQ", "summary": "The authors examine the discrepancy between trajectory prediction accuracy and driving performance in the task of autonomous driving when a fixed dataset is used and in the presence of multiple surrounding traffic participants (such as other vehicles). Specifically, they assert that this discrepancy arises from:\n-  A dynamics gap between the static dataset that the trajectory predictor is trained on and its subsequent usage for planning in real-world driving scenario. This is because the ego-vehicle’s planner may act differently to the static dataset, which results in novel reactions from the other entities in the environment (vehicles, pedestrians, etc.) and thus out-of-distribution scenarios occur.\n- Computational efficiency issues caused by the planner and trajectory predictor being too slow to accommodate real-time operation.\n\nAlthough the work does not propose a novel method, the authors conduct of study of existing methods to illustrate the importance of a dynamic approach where the ego-vehicle collects data with the specific predictor (instead of relying on a single fixed static dataset) and efficient trajectory prediction. \n\nA wide variety of trajectory prediction models are used (Table 1) to ensure high test coverage along with two planners (RVO and DESPOT) (section 4.2)\n\nFor trajectory prediction evaluation they use the Average/Final Distance Error (ADE/FDE) in Table 2. To evaluate driving performance a combination of collision avoidance, speed to goal and comfort (minimize jerk) are considered (section 4.4).\n\nExperiments first attempt to illustrate how their chosen simulator (SUMMIT, extended from Carla) is suitable to evaluate real-world performance based on simulation (section 5.1, Figure 2) as trajectory prediction simulation results (SUMMIT) and real-world dataset results (Argoverse dataset) are positively correlated.\n\nThey then illustrate the performance gap that occurs when evaluating against a static dataset versus one collected interactively (dynamically) from simulation where the ego vehicle operates with the predictor (section 5.2, Figure 3. versus Figure 4). Most notably, they find that efficiency concerns are most prevalent under a high planner-simulation tick rate. In this case the predictor efficiency has the greatest effect on driving performance instead of the dynamics gap since it cannot keep up with the high tick rate (Table 3). Table 4 further shows that the discrepancy between driving performance and trajectory prediction accuracy (ADE) is reduced when only nearby agents with full observations is preferred in addition to dynamic data collection.", "questions": "- Should the minADE and minFDE equations in Table 2 be a function of k? I assume that each predicted  $\\hat{x}_i$ is a sample taken K times?\n\n- In Table 3, is the “Inference Time” for one network inference loop and not the overall time for planning? If it is the overall time for planning I would be confused as to why performance changes for some models below the 30hz refresh rate when going from 30 to 3 to 1 hz.\n\n- In section 5.1, lines 237-238 it says that the “nearest three agents are randomly selected to be the interested agent for prediction”. Is there a reason why only 3 agents were considered? Was this the best trade-off between computation time and performance?", "limitations": "The authors did not explicitly list any limitations of their method. However, their work is primarily a study of existing methods and so a limitations section is perhaps less relevant. Nonetheless, one possible limitation is the focus solely on simulation for illustrating static versus dynamic driving performance discrepancy. Please refer to the weaknesses section for more possible limitations.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Experiments are well done and compare a variety of approaches (ten trajectory prediction models, two planners).\n\n- Claims are backed clearly by data. The results show the value of using an interactive simulation environment for data gathering and favoring efficient predictors when doing real-time planning for autonomous driving.\n\n- In general, well written and clear.", "weaknesses": "- The work does not propose a novel approach but is instead of survey of existing methods. \n\n- It could also be argued that the survey results are somewhat obvious (which compounds with the first criticism): 1) relying solely on a static dataset for model-based training may result in a distributional shift during real-world evaluation once planning is done, causing failure and 2) efficient predictors are important for real-time planning.\n\nMinor Criticism: \n\n- The combination of performance metrics $P_{\\mathrm{Safety}}, P_{\\mathrm{Efficiency}}, P_{\\mathrm{Comfort}}$ could be clarified in the text (section 4.4). Sensibly, $P_{\\mathrm{Safety}}$ and $P_{\\mathrm{Comfort}}$ are to be minimized while $P_{\\mathrm{Efficiency}}$ is maximized. But all metrics are labeled “performance” so I was originally expecting them to be handled uniformly. The text does mention that “we normalized the direction of these three metrics” but this is still somewhat ambiguous. Things are fully clarified once looking at the supplemental material where the sign on $P_{\\mathrm{Safety}}$ and $P_{\\mathrm{Comfort}}$ are reversed during normalization. However, I would find it helpful if this was more clearly mentioned to in the text.\n\n- I feel that the first experimental section (section 5.1) is perhaps not relevant to the main purpose of the paper besides illustrating that trajectory prediction accuracy in simulation and the real-world are positively correlated across different models. I imagine most semi-photorealistic simulation environments would possess this property, especially given its extension from the mature Carla simulator. They also do not explicitly show that driving performance is ultimately related (only trajectory prediction performance). There does appear to be a notable ADE/FDE error increase when moving to the real-world dataset. Thus, once applied to planning, I could see a case where a model might perform “well” in simulation but “poorly” in the real world (despite performing better than its peers).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments! We kindly ask the reviewer to let us know if further clarification or information is needed. \n\n>The work does not propose a novel approach but is instead a survey of existing methods.\n\nWe do not provide a new approach, but instead a prediction evaluation protocol, which is also important in the field of autonomous driving. Proper evaluation is an important yet often overlooked aspect of machine learning research. Prior works focusing on evaluation from NeurIPS [1,2] play an important role in their respective field.  \n\n>It could also be argued that the survey results are somewhat obvious.\n\nThis is a great point for us to elaborate more! This paper shed light on **the equally essential need for evaluating seemingly static modules, such as prediction, through closed-loop evaluation**, which has been overlooked in recent studies [3,4] and SOTA of Argoverse Competition [5]. Another important idea is **the trade-off between computational efficiency and accuracy even when the predictor is fast enough for the planner's execution**. In the context of autonomous driving, a consensus exists that the perception system must meet a specified threshold for proper planning. Once this threshold is satisfied, the focus shifts to ensuring accuracy. (e.g., 100ms [6,7]) However, our experiments demonstrate the trade-off persists even below the threshold. When the predictor runs much faster than the threshold (2ms), the driving performance remains dominated by the fastest prediction method (CV). This trade-off holds importance, as various planners exhibit their trade-offs in addition to their fundamental requirement.\n\n>The combination of performance metrics could be clarified in the main text.\n\nSure! We will put the explanations in our main text to make it clearer, detalis in the general response due to word limitation.\n\n>The first experimental section is not relevant to the main purpose of the paper.\n\nAs pointed out, our first experiment is to establish the positive correlation between trajectory prediction accuracy across various environments. This serves as the foundation of our experiments, but not supporting the main idea. **To avoid misunderstanding of our core concept, we will include the Sim-Real Alignment experiment in the Appendix.**\n\n>A model might perform “well” in simulation but “poorly” in the real world.\n\nThe gap you mentioned arises from the disparities between simulation and real-world, which we can refer to as interaction gap. This mismatch arises due to unreal exo-agent behavior, as the simulator controls their actions. Unfortunately, this can only be addressed through real-world tests, which is often unaffordable. Our paper focuses on addressing the dynamics gap that exists between static evaluation and dynamic evaluation. In static evaluation, both interaction gap and dynamics gap remain unresolved. In dynamic evaluation, only the interaction gap remains. This is our main contribution, and how to solve the interaction gap lies beyond the scope of this paper.\n\n>Should the minADE and minFDE equations in Table 2 be a function of k? I assume that each predicted  xi Is a sample taken K times?\n\nThank you for bringing this to our attention. We apologize for the error in formulating minADE/FDE. We take our experiment setting based on the consensus of Argoverse/Waymo Dataset, setting K=6. We will explicitly state it in the final version.\n\n>Is the “Inference Time” for one network inference loop and not the overall time for planning?\n\nYes. The inference time is for one network inference loop and not the overall time for planning. It is essential to distinguish the tick rate as the time of simulator, not the inference time. In a simulator with Tick Rate=30Hz, planners are constrained to act within 0.3s (real time). Conversely, Tick Rate=3Hz allows 3s for planning. Adjusting the tick rate grants planners more time to act, enabling them to explore more potential future states, thereby influencing driving performance.\n\n>Is there a reason why only 3 agents were considered? Was this the best trade-off between computation time and performance?\n\nThis setting is designed to replicate the data generation process of Argoverse. In the Argoverse dataset, each scenario selects one agent as the interested agent, which must possess complete observation and future states. As the nearest agents are more likely to have full observation, we choose it as the interested agent in our data generation. To preserve multi-modality, we randomly choose one from the three nearest agents. However, during our experiments in Section 5.2~5.3, predictions are made for all agents.\n\n>The authors did not explicitly list any limitations. One possible limitation is the focus solely on illustrating static versus dynamic driving performance discrepancy.\n\n**Though we have discussed our limitation on planner selection and simulation scope in Section 6, we totally agree it is crucial to emphasize the limitation posed**. One common and critical factor is the asymmetry of prediction error. Even when two prediction models exhibit the same ADE, their driving performances will not be identical. This discrepancy arises from the different influence of prediction errors to planning. We will add this limitation in our final version.\n\n[1]Agarwal et al. Deep reinforcement learning at the edge of the statistical precipice. NeurIPS 2021.\n\n[2]Pillutla et al. Mauve: Measuring the gap between neural text and human text using divergence frontiers. NeurIPS 2021.\n\n[3]Hu et al. Planning-oriented autonomous driving. CVPR 2023.\n\n[4]Liang et al. Learning lane graph representations for motion forecasting. ECCV 2020.\n\n[5]Zhou et al. Query-Centric Trajectory Prediction. CVPR 2023.\n\n[6]Lin et al. The architectural implications of autonomous driving: Constraints and acceleration. ASPLOS 2018.\n\n[7]Yamaguchi et al. In-vehicle distributed time-critical data stream management system for advanced driver assistance. JIP 2017.\n"}, {"review_id": "H70vw49rxH", "reviewer": "Reviewer_1Qcn", "summary": "This paper presents a timely and well-executed study on motion prediction, focusing on the relationship between the dominant evaluation paradigm (static, offline metrics) and the true objective of research in this domain (safer planning). The study uncovers several important and surprising findings: (1) static offline metrics are not correlated to planning performance, (2) evaluating on the frames observed by the planner (dynamic offline metrics) are significantly better correlated, (3) computational efficiency in prediction is increasingly important for sophisticated planners, and (4) simple baselines for prediction (constant velocity/constant acceleration) perform best when considering downstream planning performance.", "questions": "1. A very interesting finding that I suggest should be highlighted more is that in all closed-loop evaluation settings, the best among the prediction methods used in this study is always either CV or CA - the naive baselines.\n2. Do the CV and CA baselines forecast the positions along a straight line (heading direction), or do they take into account angular speed/acceleration as well?\n3. In L316, could you please elaborate on what \"complete/incomplete observations\" refers to?\n4. L328 recommends Dynamic FDE with Closest, when closed loop simulation is possible. However, why would this be preferable to simply ranking prediction methods by the “Driving Performance” of the planner?\n5. Please cite https://arxiv.org/abs/1809.04843 while wrapping up Section 2.2 - this is a well-known study with a similar setup, aimed at planning instead of motion prediction. It would also be interesting to contrast your findings to theirs, given the change in task.\n6. Could part of the discrepancy arise from the fact that prediction models penalize errors on all vehicle instances equally, while some are less relevant for planning than others? The result in Table 4 showing the improved correlation when considering only the nearest agents seems to indicate this.\n    1. Could you add a row with the “Closest” checkmark for the static, multimodal ADE?\n    2. If it is possible to include a task-aware offline metric such as PKL [15], CAPO [18] or some equivalent in the analysis, this would add significant value to the paper, and I would be happy to increase my rating.\n\nMinor:\n\n1. L078: Lastname et al. instead of Firstname et al.\n\nUpdate:\n\nThank you for taking the time to carefully answer my questions, I appreciate the effort. The rebuttal clarifies most of my questions. I understand the choice of Dynamic ADE/FDE for the experiments in this paper (as the rebuttal points out, \"a prediction metric that employs the same calculation methodology but behaves distinctively in these two evaluation frameworks\" helps provide evidence for their claims).\n\nI am still leaning towards a positive rating as I see the main contribution of this paper as highly relevant to the motion prediction community for driving. This is a large research area with hundreds of published papers, and little incentive to move away from static metrics (even though the lack of alignment between open-loop and closed-loop metrics for planning is reasonably well-known). This paper opens up a discussion on the inadequacy of the prevalent motion forecasting metrics, and could be a valuable first step towards more comprehensive evaluation of prediction models in the future.", "limitations": "Limitations are discussed in Section 6.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The study setup is excellent, making well-motivated choices for the task, simulator, predictors, planners, metrics, etc. The writing is clear. The presentation of the key findings both visually and numerically also makes this paper an interesting and easy-to-understand read. Most importantly, it highlights a flaw in the widely used metrics that the community optimizes motion prediction models for, and shows that naive baselines are sufficient to outperform SoTA learned forecasting when using the right metrics.", "weaknesses": "Overall, the paper has very few weaknesses in my opinion. While the selected ML prediction models do not include the current leaderboard winners, I believe they are representative. The experiments could have been conducted on more realistic scenarios (e.g. the nuPlan simulator), but Section 5.1 shows that the simulator used is sufficiently aligned to real datasets for the purpose of this study. However, some technical details regarding the study are unclear (please see the “Questions” section #2, #3, #4 for specifics).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comprehensive comments! We kindly ask the reviewer to let us know if further clarification or information is needed. \n\n>A very interesting finding that I suggest should be highlighted more is that in all closed-loop evaluation settings, the best among the prediction methods used in this study is always either CV or CA - the naive baselines.\n\nYes, that’s an important observation! When the time budget is limited, the simple rule-based model gives very good driving performance, which is in line with previous research: a simple Constant Velocity Model can outperform even state-of-the-art neural models [1]. We will include this paper as a reference in our final version.\n\n>Do the CV and CA baselines forecast the positions along a straight line, or do they take into account angular speed/acceleration as well?\n\nThat’s a good question. We solely utilize position for our predictions, extracting speed and acceleration information. As a result, angular speed or acceleration is not taken into consideration. This approach aligns with the baseline of the Argoverse Competition.\n\n>In L316, could you please elaborate on what \"complete/incomplete observations\" refers to?\n\nThe term \"complete\" signifies the presence of all 20 observations of an agent, while \"incomplete\" denotes instances where certain observations are absent due to factors like occlusion, being outside the receptive field, etc.\n\n>L328 recommends Dynamic FDE with Closest, when closed loop simulation is possible. However, why would this be preferable to simply ranking prediction methods by the “Driving Performance” of the planner?\n\nThis is a great point for us to clarify more! The correlation between Dynamic ADE/FDE and driving performance is strong for both planners, rendering them suitable for ranking prediction models. However, the core contribution of this paper lies in revealing the absence of correlation between static evaluation and driving performance. To illustrate this, we adopt a prediction metric that employs the same calculation methodology but behaves distinctively in these two evaluation frameworks. By showing this, we emphasize the significance of dynamic evaluation. \n\n>Please cite https://arxiv.org/abs/1809.04843 while wrapping up Section 2.2 - this is a well-known study with a similar setup, aimed at planning instead of motion prediction. It would also be interesting to contrast your findings to theirs, given the change in task.\n\nThank you for bringing this paper to our attention. We concur with the significant factors highlighted within it, like the asymmetry of prediction error. We will include this paper in related works and contract our findings with it in Q6.\n\n>Could part of the discrepancy arise from the fact that prediction models penalize errors on all vehicle instances equally, while some are less relevant for planning than others? The result in Table 4 showing the improved correlation when considering only the nearest agents seems to indicate this.\n\nYes. Even when two prediction models present the same ADE, their driving performances are not identical. This discrepancy arises from the different influence of prediction errors to planning. For instance, errors associated with the closest exo-agent can have a higher impact on the ego-agent's plan. The conflict between the predicted state of the exo-agent with the ego-agent's plan also matters. \n\n**However, it is important to note that the asymmetry of prediction error is not universal, which means it may occur only in specific scenarios or for specific agents**. As a result, the influence of this factor is less significant than dynamic evaluation and computational efficiency, both of which impact prediction evaluation are ubiquitous. To provide evidence, we have accounted for the most critical aspect of asymmetry prediction error by comparing \"Closest Dynamic ADE\" with \"Dynamic ADE.\" Consequently, the influence of asymmetry on the correlation between ADE and Driving Performance is less than dynamic evaluation. (Influence For RVO Planner: 0.16 vs 0.61; Influence For DESPOT Planner: 0.20 vs 0.96) \n\nHence, in this paper, our primary focus is on discussing the significance of dynamic evaluation and computational efficiency while giving less emphasis to this particular factor, even though it remains crucial in prediction evaluation.\n\n>Could you add a row with the “Closest” checkmark for the static, multimodal ADE?\n\nThe static and multimodal ADEs are already “Closest”. Within the Alignment dataset, we randomly select the three nearest agents as the “interested agent” for prediction, with predictions solely made for this \"interested agent\" in each scenario. This approach is implemented to mirror the configuration of an interested agent as in the Argoverse dataset.\n\n>If it is possible to include a task-aware offline metric such as PKL [15], CAPO [18] or some equivalent in the analysis, this would add significant value to the paper, and I would be happy to increase my rating.\n\nThese task-aware offline metrics offer valuable insights in open-loop evaluation where static datasets are available. However, applying these metrics poses challenges within closed-loop, where the ego planner functions in real-time, causing the ground truth to dynamically shift based on the planner's choices. This dynamic nature eliminates the existence of a fixed ground truth, against which to benchmark the planner's performance. Given this absence of a consistent ground truth within closed-loop, conducting PKL or CAPO measurements becomes complicated. While we recognize the value of task-aware offline metrics, we will explore means of adapting such measures for closed-loop evaluation in future.\n\n>L078: Lastname et al. instead of Firstname et al.\n\nThank you for bringing this error to our attention. We will correct this to \"McAllister et al.\" in our final version.\n\n[1] Schöller et al. What the constant velocity model can teach us about pedestrian motion prediction. RAL 2020.\n"}]

Author rebuttal:
We sincerely thank all the reviewers for your constructive feedback and recognition of this work. The reviewers found:

* The problem is “original” (R-Aygk) and “interesting” (R-1Qcn).
* The experiments are “well-done” (R-tS95, R-RCyQ, R-1Qcn) and  “extensive” (R-tS95, R-RCyQ). 
* The work provides “an excellent study setup” (R-1Qcn) and “highlights the flaws in the widely used metrics of trajectory prediction evaluation,” (R-1Qcn, R-Aygk).

We would like to re-emphasize this work's technical contributions:
* We shed light on the equally essential need for evaluating seemingly static modules, such as prediction, through closed-loop evaluation.
* We reveal the trade-off between computational efficiency and accuracy even when the predictor is fast enough for the planner's execution.
* We propose Dynamic ADE as a new evaluation metric in the closed-loop evaluation.

We have made the following claims more clear according to all the reviewers’ insightful comments. 
* **The purpose of Sim-Real Alignment (Section 5.1)**. Even in different real-world datasets, the performance of predictors will differ. (SOTA of nuScenes: minADE5 = 1.092m in 6s, SOTA of Waymo: minADE6 = 0.5345m in 8s) Thus, what we care about is predictors’ ranking of accuracy. As a result, we conduct the alignment experiment to demonstrate that the ranking of prediction performance keeps stable across real-world dataset Argoverse and simulation dataset Alignment.
* **The detailed definition of Dynamic ADE and its difference to Static ADE**. Similar to Static ADE, Dynamic ADE is also calculated by the average L2 distance between the forecasted trajectory and the ground truth. However, in closed-loop evaluation, different predictors result in varied behaviors of the ego-agent, which, in turn, influence the future behaviors of other road users, leading to different dynamics within the environment. This directly affects the ground truth of prediction as other agents behave differently, as explained in the attached PDF.
* **The trade-off between computational efficiency and accuracy**. A consensus exists that the perception system must meet a specified threshold (e.g. 100ms) for proper planning. Once it is satisfied, the focus shifts to ensuring accuracy. However, our experiments demonstrate a notable trade-off even when the predictor runs considerably faster than the threshold (2ms). Only in experiments with ample planning time budget (Tick Rate = 1Hz), the method with better accuracy dominates.

We also include more analysis to support our ideas:
* Other possible factors that may cause the discrepancy and why the two factors mentioned in the paper are dominant.
* Exploring the correlation between temporal consistency and driving performance to investigate whether sudden changes in prediction (mainly occur when encountering unknown future issues) impact driving performance.
* Examining the correlation between prediction variance and driving performance to gain a more profound understanding of the influence of multimodality.

We add the related works to make it clearer why we choose SUMMIT:
* Several rule-based simulators are designed for autonomous driving systems. CARLA [1] offers a range of sensors and agent types but relies on predefined maps and exhibits relatively low density with simple rule-based behaviors. Another simulator [2] supports real-world maps, yet its rule-based planner complicates prediction performance evaluation. 
* Recently, learning-based simulators [3,4] have emerged, generating diverse and realistic behaviors from actual data using open-source datasets. However, these simulators are restricted to predetermined maps. Generating realistic behaviors for new maps may prove challenging. 
* SUMMIT [5] stands out by offering rich-context urban maps, realistic visuals, and intricate traffic behavior. Agent behaviors are generated using GAMMA [6], an advanced multi-agent motion prediction model. Leveraging SUMMIT's advanced capabilities, we can provide a comprehensive assessment with various planners and predictions.

We also add the equation for the combination of performance metrics to the main text, following the valuable comment of reviewer RCyQ:

$$
\bar{P}_{\textrm{metrics}}=\begin{cases}
\frac{P\_{\textrm{metrics}} - P^{\textrm{min}}\_{\textrm{metrics}}}{P^{\textrm{max}}\_{\textrm{metrics}} - P^{\textrm{min}}\_{\textrm{metrics}}}, & \textrm{metrics} = \\{\textrm{efficiency}\\} \\\\
1-\frac{P\_{\textrm{metrics}} - P^{\textrm{min}}\_{\textrm{metrics}}}{P^{\textrm{max}}\_{\textrm{metrics}} - P^{\textrm{min}}\_{\textrm{metrics}}}, & \textrm{metrics} = \\{\textrm{safety}, \textrm{comfort}\\}
\end{cases}
$$
where $P^{\textrm{min}}\_{\textrm{metrics}}$ and $P^{\textrm{max}}\_{\textrm{metrics}}$ represent the minimum and maximum pairs of each performance metric among all scenarios.

We kindly ask the reviewers to let us know if further clarification or information is needed. 

[1] Dosovitskiy et al. CARLA: An open urban driving simulator CoRL 2017

[2] Lopez et al. Microscopic traffic simulation using sumo ITSC 2018

‌[3] Caesaret et al. NuScenes: A Multimodal Dataset for Autonomous Driving CVPR 2020

‌[4] Igl et al. Symphony: Learning Realistic and Diverse Agents for Autonomous Driving Simulation ICRA 2022

[5] Cai et al. Summit: A simulator for urban driving in massive mixed traffic ICRA 2020

[6] Luo et al. Gamma: A general agent motion model for autonomous driving RAL 2022


Meta-review: The paper investigates for trajectory prediction the discrepancy between performance on fixed datasets and the driving performance when applied to downstream applications. It proposes two overlooked factors that might affect the broader impact and yet less well-studied in the community. 

All reviewers reach concensus that the paper is well written; the formulation is easy to follow and the problem setup worth investigating. The experiments have been conducted effectively to verify the key claims. The final rating is 4,4,4,5,7. The main concerns by the negative comments are:

- "the effect of simulation on the closed-loop evaluation". The proposed metric (e.g. dynamic ADE) does not imply a good indicator for a proposed method. 

- the paper looks more like a survey / detailed report of existing methods.

Authors did a detailed rebuttal. There are quite a few back and forth dicussions among reviewers and authors. Since this is a borderline paper, AC steps in, read the paper, rebuttal, author feedback and discussions carefully. The final recommendation is Accept based on:

- The paper did actually observe key factors that impede the motion prediction community: discrepancy between fixed dataset and real scenarios (e.g. interactive with other agents in the enviroment). The observations found in the paper are **highly relevant as great reference for the general audience**.

- As mentioned by Reviewer 1Qcn, the paper opens a discussion on how to align the discrepancy between open-loop (fixed dataset) and closed-loop (realistic scenario) metric for planning. This could serve as the preliminary step to trigger more investigation on this direction.

As such, despite there are some flaws (e.g. experiments on simulation rather than real-world) in the paper, AC believes that these are another story or fixable, and trust authors (based on rebuttal) could polish the manuscript to great extent. It is highly recommended to consolidate all comments, open-source the project and make results reproducible in a timely manner.
Decision: Accept (poster)