Title: What Truly Matters in Trajectory Prediction for Autonomous Driving?
Venue: NeurIPS 2023
Accepted: yes
Keywords: trajectory prediction; autonomous driving
Abstract: Trajectory prediction plays a vital role in the performance of autonomous driving systems, and prediction accuracy, such as average displacement error (ADE) or final displacement error (FDE), is widely used as a performance metric. However, a significant disparity exists between the accuracy of predictors on fixed datasets and driving performance when the predictors are used downstream for vehicle control, because of a dynamics gap. In the real world, the prediction algorithm influences the behavior of the ego vehicle, which, in turn, influences the behaviors of other vehicles nearby. This interaction results in predictor-specific dynamics that directly impacts prediction results. In fixed datasets, since other vehicles' responses are predetermined, this interaction effect is lost, leading to a significant dynamics gap. This paper studies the overlooked significance of this dynamics gap. We also examine several other factors contributing to the disparity between prediction performance and driving performance. The findings highlight the trade-off between the predictor's computational efficiency and prediction accuracy in determining real-world driving performance. In summary,  an interactive, task-driven evaluation protocol for trajectory prediction is crucial to capture its effectiveness for autonomous driving. Source code along with experimental settings is available online (https://whatmatters23.github.io/).

What Truly Matters in Trajectory Prediction for Autonomous Driving?

Abstract

Trajectory prediction plays a vital role in the performance of autonomous driving systems, and prediction accuracy, such as average displacement error (ADE) or final displacement error (FDE), is widely used as a performance metric. However, a significant disparity exists between the accuracy of predictors on fixed datasets and driving performance when the predictors are used downstream for vehicle control, because of a dynamics gap . In the real world, the prediction algorithm influences the behavior of the ego vehicle, which, in turn, influences the behaviors of other vehicles nearby. This interaction results in predictor-specific dynamics that directly impacts prediction results. In fixed datasets, since other vehicles' responses are predetermined, this interaction effect is lost, leading to a significant dynamics gap. This paper studies the overlooked significance of this dynamics gap. We also examine several other factors contributing to the disparity between prediction performance and driving performance. The findings highlight the trade-off between the predictor's computational efficiency and prediction accuracy in determining real-world driving performance. In summary, an interactive , task-driven evaluation protocol for trajectory prediction is crucial to capture its effectiveness for autonomous driving. Source code along with experimental settings is available online.

1 Introduction

Current trajectory prediction evaluation [29, 9, 5] relies on real-world datasets, operating under the assumption that dataset accuracy is equivalent to prediction capability. We refer to this as Static Evaluation . This methodology, however, falls short when the predictor serves as a sub-module for downstream tasks in Autonomous Driving (AD) [28, 24]. As illustrated in Figure 1, the static evaluation metrics on datasets, such as Average Displacement Error (ADE) and Final Displacement Error (FDE), do not necessarily reflect the actual driving performance [36, 8, 10]. In contrast to the conventional focus on uncertainty [39, 16] and potential interaction [27], we demonstrate that this disparity stems from the overlooked dynamics gap between fixed datasets and AD systems and the computational efficiency of predictors. The trade-off between these two factors matters in the actual prediction performance within the entire AD system.

The dynamics gap arises from the fact that the behavior of the autonomous vehicle, also known as the ego-agent, varies with different trajectory predictors, as presented in Figure 2. In real-world scenarios, the ego-agent utilizes predictors to determine its actions. Different predictors result in varied behaviors of the ego-agent, which, in turn, influence the future behaviors of other road users, leading to different dynamics within the environment. This directly affects the accuracy of predictions, as other agents behave differently. Since the ego-agent's actions are predetermined on

1.75 2.00 2.25 2.50 2.75 3.00 3.25

ADE

0.70

0.72

0.74

0.76

0.78

CA

KNN

S-KNN

HiVT LaneGCN

LSTM

S-LSTM

ADE vs Driving Performance

Reality

Assumption

1/ADE

1/FDE

Driving

Comfort

Safety

1/ADE

1/FDE

Driving

Comfort

Safety

CA

1/ADE

1/FDE

Driving

Comfort

Safety

KNN

1/ADE

1/FDE

Driving

Comfort

Safety

S-KNN

3.5

4.0

4.5

5.0

5.5

6.0

6.5

FDE

0.70

0.71

0.72

0.73

0.74

0.75

0.76

0.77

Driving Performance

CA

KNN

S-KNN

HiVT

LaneGCN

LSTM

S-LSTM

FDE vs Driving Performance

Reality

Assumption

1/ADE

1/FDE

Driving

Comfort

Safety

HiVT

1/ADE

1/FDE

Driving

Comfort

Safety

LaneGCN

1/ADE

1/FDE

Driving

Comfort

Safety

LSTM

1/ADE

1/FDE

Driving

Comfort

Safety

S-LSTM

the dataset, there exists a significant disparity between the dynamics represented in the dataset and the actual driving scenario when evaluating a specific trajectory predictor. To tackle this issue, we propose the use of an interactive simulation environment to evaluate the predictor for downstream decision-making. This environment enables Dynamic Evaluation as the ego-agent operates with the specific predictor, thus, mitigating the dynamics gap. We demonstrate a strong correlation between the dynamic evaluation metrics and driving performance through extensive experiments. This underscores the dominant role of the dynamics gap in aligning prediction accuracy with driving performance and emphasizes the importance of incorporating it into the evaluation process.

The awareness of the dynamics gap significantly enhances the correlation between prediction accuracy and driving performance. However, there are still factors that affect driving performance apart from prediction accuracy. The downstream modules in AD systems, with different tasks and varying levels of complexity, impose different requirements on prediction models. In our experiments, we vary the type of planner and the time constraint for the motion planning task. Our findings suggest that predictors' computational efficiency plays a vital role in real-time tasks. Moreover, it is the trade-off between prediction accuracy and computational efficiency that determines driving performance. This highlights the necessity for task-driven prediction evaluation.

(

a

)

(

b

)

In this paper, we aim to address two specific aspects of trajectory prediction for AD systems. Firstly, we uncover the limitation of static prediction evaluation systems in accurately reflecting driving performance. We demonstrate the dominant role of the dynamics gap in causing the disparity. Secondly, we emphasize the necessity of an interactive, task-driven evaluation protocol that incorporates the trade-off between dynamic prediction accuracy and computational efficiency. The protocol presents a promising way for applying prediction models in real-world autonomous driving, which is the truly mattered aspect in trajectory prediction for autonomous driving.

2 Related Work

2.1 Motion Prediction and Evaluation

Motion prediction methods can be classified along three dimensions [17]: modeling approach, output type, and situational awareness. The modeling approach includes physics-based models [30, 3]

that use physics to simulate agents' forward motion, and learning-based models [40, 22] that learn and predict motion patterns from data. The output type can be intention, single-trajectory [26], multi-trajectory [18], or occupancy map [14, 15]. These outputs differ in the type of motion they predict and how they handle the uncertainty of future states. The situational awareness includes unawareness, interaction [2], scene, and map awareness [31]. It refers to the predictor's ability to incorporate environmental information, which is crucial for collision avoidance and efficient driving. Most researchers [29, 25] and competitions [9, 5] evaluate the performance of prediction models on real-world datasets, in which ADE/FDE and their probabilistic variants minADE/minFDE are commonly used metrics. However, these metrics fail to capture the dynamics gap between datasets and real-world scenarios, as the actions of the ego-agent remain unaffected by predictors. In this study, we select four model-based and six learning-based models with varying output types and situational awareness to cover a wide range of prediction models. We implement these predictors in both datasets and an interactive simulation environment to illustrate the limitation of current prediction evaluation in accurately reflecting driving performance, due to the neglect of the dynamics gap.

2.2 Task-aware Motion Prediction and Evaluation

Task-aware motion prediction remains an underexplored area in research. While some studies touch upon the subject, they focus on proposing task-aware metrics for training or eliminating improper predictions on datasets. One notable example of training on task-aware metrics is the Planning KL Divergence (PKL) metric [28]. Although designed for 3D object detection, it measures the similarity between detection and ground truth by calculating the difference in ego-plan performance. In the context of motion prediction, Rowan et al. [24] propose a control-aware metric (CAPO) similar to PKL. CAPO employs attention [35] to find correlations between predicted trajectories, assigning higher weights to agents inducing more significant reactions. Similarly, the Task-Informed method [19] assumes a set of candidate trajectories from the planner and adds the training loss for each candidate. Another line of work focuses on designing task-aware functions to eliminate improper predictions [21, 13]. The proposed metric can capture unrealistic predictions and better correlate with driving performance in planner-agnostic settings. However, these works are conducted in an open-loop manner, neglecting the disparity between open-loop evaluation and real-world driving performance. To the best of our knowledge, we are the first to conduct a thorough investigation into the disparity and demonstrate the dominant role of the dynamics gap on this issue.

3 Planning with Motion Prediction

3.1 Problem Formulation

In this section, we present the problem formulation for motion prediction with planning in the context of autonomous driving. Different from conventional formulations that consider the planner as a policy, our formulation employs the predictor as the policy. This enables us to identify the primary challenge in developing predictors for real-world autonomous driving. We define the prediction problem as an MDP: M = ( S , A , T , R ) .

State. The state of the system is represented by the tuple s = ( s ego , s exo ) for s ∈ S , where s ego denotes the state of the A V including historical information, s exo denotes that of surrounding traffic participants. Specifically, s ego = { s ego t -t obs , . . . , s ego t } , where t obs represents the observation horizon and t denotes the current timestep. The state for each vehicle includes its position, velocity, heading, and other relevant attributes, e.g., s ego t = ( x ego t , y ego t , v ego t , θ ego t ) .

Action. At each time step, the A V can take an action a ∈ A , where A is the action space. We denote the n surrounding traffic participants as i ∈ { 1 , . . . , n } . The action consists of the A V's prediction of all exo-agents, e.g., a = { a 1 , . . . , a n } .

Transition Function. The transition function T : S × A → S defines the dynamics of the system and how the system evolves as a result of the action of the ego vehicle. In the MDP, it refers to: T ( s t +1 | s t , a t ) = T 1 ( s ego t +1 | s t , a t ) T 2 ( s exo t +1 | s t ) , T ∈ T . We denote the planner as a mapping function T 1 that takes the current state s t and action a t as input and outputs the AV's next state s ego t +1 . The motion models of exo-agents are integrated into the mapping function T 2 , which takes the current state s t as input and generates the next state s exo t +1 for all exo-agents.

Predictor. The predictor is represented as a policy a = π ( s ) that takes the current state s as input, and produces an action a ∈ A as output.

Objective Function. The objective of prediction is to accurately predict exo-agents future states while considering their interactions and map information. The reward function R : S ×A → R maps a state-action pair to a real-valued reward. The objective function is the accumulated reward over the task horizon H , defined as J ( s, a ) = ∑ H t =0 R ( s t , a t ) .

Optimal Policy. The objective of autonomous driving is to identify the optimal policy that minimizes the objective function under the real-world transition function. The optimal policy π ∗ is expressed as: π ∗ = arg max T 1 = T ∗ 1 ,T 2 = T ∗ 2 E π [ ∑ H t =0 R ( s t , a t )] , where T ∗ 1 represents any realistic planners used in A Vs, and

T ∗ 2 denotes the integrated motion model of exo-agents in the real world.

3.2 The Limitation of Static Evaluation

Traditional prediction methods overlook the difference between the transition function T ∗ 1 in realworld autonomous driving and the represented ˆ T 1 in datasets. These methods focus on training a policy using well-designed reward functions to perform effectively within the state distribution of datasets. The trained policy π satisfies:

In this context, ˆ T 1 serves as a static planner that remains unaffected by predictors. Regardless of the input, the static planner ˆ T 1 generates the recorded next state. As a result, the mapping function T ∗ 2 remains consistent with the real world since the input state is unchanged. We define the disparity in future states resulting from changes in the ego-planner as the dynamics gap in static evaluation, denoted as G = T ∗ 1 -ˆ T 1 .

The proposed planning-aware metrics, such as PKL [28] and CAPO [24], aimed to address the disparity between prediction accuracy and the ultimate driving performance by improving the cost function. However, unless the dynamics gap is mitigated, the optimization deviation plays a dominant role in causing the disparity and should be addressed as a priority.

3.3 Measuring the Impact of Dynamics Gap

Conducting real-world tests is the most effective way to address and assess the impact of the dynamics gap. However, due to its high cost, the simulator serves as a proxy. The ego-agent utilizes the realistic planner T ∗ 1 and the motion model T ∗ sim of the simulator. The optimal policy π ∗ sim satisfies:

To evaluate the impact of the dynamics gap, we train various predictors on the Alignment dataset obtained from the simulator, the trained policy π sim satisfies:

The equations suggest that the dynamics gap between the Alignment dataset and simulator mirrors the dynamics gap between the real-world dataset and actual autonomous driving scenarios. Thus, it is reasonable to employ the simulator for evaluating the influence of the dynamics gap. However, the simulator merely serves as a substitute to mitigate the dynamics gap, leaving a remaining dynamics gap represented by G ′ = T ∗ 2 -T sim 2 .

4 Experimental Setup

Our aim is to identify the key factors involved in trajectory prediction for autonomous driving and recognize their impact on driving performance by simulating real-world scenarios that vehicles might encounter. The ultimate goal is to introduce an interactive, task-driven prediction evaluation protocol. To achieve this objective, we need to determine four crucial components: 1 ) motion prediction methods to be covered; 2 ) realistic planners to employ prediction models; 3 ) the simulator to replicate interactive scenarios; 4 ) evaluation metrics to assess the effectiveness of the key factors involved in motion prediction with respect to real-world driving performance.

*Abbreviations: ST : Single-Trajectory, MT : Multi-Trajectory, OM : Occupancy Map.

4.1 Motion Prediction Methods

We select 10 representative prediction models to achieve comprehensive coverage of mainstream approaches, ranging from simple model-based methods to complex data-driven approaches, as presented in Table 1. Constant Velocity (CV) and Constant Acceleration (CA) [30] assume that the predicted agent maintains a constant speed or acceleration within the prediction horizon. K-Nearest Neighbor (KNN) predicts an agent's future trajectory based on most similar trajectories, while Social-KNN (S-KNN) [9] extends it by also considering the similarity of surrounding agents. These methods are widely used as baselines given their widespread effectiveness in simple prediction cases. Social LSTM (S-LSTM) [2], HiVT [40], LaneGCN [22], and HOME [14] represent four distinct types of neural networks: RNN, Transformer, GNN, and CNN. DSP [38] utilizes a hybrid design of neural networks, representing state-of-the-art prediction models.

4.2 Planners

An ideal planner should: 1 ) be able to handle state and action uncertainty; 2 ) take into account multiple factors of driving performance, such as safety (collision avoidance), efficiency (timely goal achievement), and comfort (smooth driving); 3 ) be aware of interactions with other agents; and 4 ) supports real-time execution. We select two realistic planners based on these criteria: a simplistic planner that only meets 2 ) and 4 ), and a sophisticated planner that satisfies all of 1 ) -4 ). This allows us to draw planner-agnostic conclusions.

RVO. The RVO planner [34] is a simplistic planner that solves the optimization problem in the velocity space under collision avoidance constraints. The planner uses motion predictions to avoid possible collisions with the deterministic motions, and thus does not consider the state and action uncertainty. As the RVO planner does not maintain states between consecutive timesteps, it also cannot optimize its planning with respect to interactions with other agents. The objective function of the RVO planner involves safety and efficiency within a short time window, and the RVO planner executes in real time.

DESPOT. The DESPOT planner [32] is a state-of-the-art belief-space planning algorithm that address uncertainties near-optimally. To account for stochastic states and actions, we adopt the bicycle model, a kinematic model with two degrees of freedom, and introduce Gaussian noise to the displacement. DESPOT considers the system state, context information, and the ego-agent's action to predict the future states of exo-agents. This allows it to consider the interaction between agents. Moreover, the objective function of DESPOT incorporates safety, efficiency, and comfort metrics, making it an ideal algorithm for planning in complex and dynamic environments.

We use these planners to control the speed of the ego-agent while pure-pursuit algorithm [11] for adjusting the steering angle. Details can be found in the supplementary materials.

4.3 Simulator

In order to evaluate different prediction models, the ideal simulator should: 1 ) provide real-world maps and agents; 2 ) model potential unregulated behaviors; 3 ) accurately mirror the interactions between agents; and 4 ) provide realistic perception data for effective planning. We choose the SUMMIT simulator [6] for our experiments since it meets all the criteria, as demonstrated in Table 2. SUMMIT is a sophisticated simulator based on the Carla [12] framework, offering various real-world

maps and agents to create diverse and challenging scenarios. It uses a realistic motion model to simulate interactions between agents and supports the simulation of crowded scenes, complex traffic conditions, and unregulated behaviors.

There are two distinct concepts of time in our experiments: simulation time and real time . The former corresponds to the duration of actions in the simulator, while the latter represents the wall time consumed by the planner. By default, the simulator runs in asynchronous mode. In this mode, the simulator and planner run individually, and the simulation time is equal to the real time. However, in this study, we employ the SUMMIT simulator in synchronous mode to accommodate slow predictors. In this mode, the simulator waits for the planner to establish communication before proceeding to the next step. Since the simulation time for each execution step remains constant at 0 . 03 s , the ratio between simulation time and real time can be manually set by varying the communication frequency, known as the tick rate. For example, once the tick rate is set to 30 Hz , the simulation time equals the real time. But, when the tick rate is set to 3 Hz , the ratio between simulation time and real time is 0 . 1 .

4.4 Evaluation Protocols

Motion Prediction Performance Metrics. Four commonly used prediction performance metrics are employed in this study, as presented in Table 3. In our experiments, the consensus of K=6 is adopted. While ADE/FDE can be applied to evaluate single-trajectory prediction models, their probabilistic variants minADE and minFDE can be applied to evaluate multi-trajectory predictors.

Driving Performance Metrics. The driving performance is primarily determined by three factors: safety, comfort, and efficiency. Let H represent the total timestep for each scenario.

Safety is typically evaluated in terms of the collision rate. The collision is determined whether the smallest distance between the ego-agent and surrounding agents is smaller than a threshold ϵ , that is: P Safety = 1 H ∑ H t =1 I [ min {|| s t A -s || : s ∈ { s t 1 , ..., s t n }} < ϵ ] , where || · || is the L2 distance between the ego-agent's bounding box and the exo-agent's bounding box, I is the boolean indicator function. We set ϵ = 1 mfor our experiments since the DESPOT planner rarely causes real collisions.

Efficiency is evaluated by the average speed of the ego-agent over the whole scenario: P Efficiency = 1 H ∑ H t =1 v t A . Comfort is represented by the jerk of the ego-agent, which is the rate of change of acceleration with respect to time: P Comfort = 1 H ∑ H t =1 ¨ v t A .

Since these three metrics are not comparable, we normalized each metric to [0 , 1] before calculating the driving performance. Furthermore, we standardized the direction of these metrics, with higher values indicating better performance:

The boolean indicator n = I [ m ∈ { Efficiency } ] is employed to adjust the direction of driving performance metrics. The ultimate driving performance is derived through the average of normalized metrics for safety, efficiency, and comfort. Pseudocode can be found in the supplementary materials.

Experimental Design. We conduct two types of experiments for both planners in the SUMMIT simulator: Fixed Prediction Ability and Fixed Planning Ability .

Fixed Prediction Ability: The planner is required to perform a fixed number of predictions within an interactive simulation environment, regardless of the predictor's execution speed. The objective is to clarify the factor that dominates the disparity between prediction performance and driving performance while ensuring the predictive ability of methods.

Fixed Planning Ability: The planner is allocated varying time budgets to simulate a predictor running at different speeds. Three sub-experiments are conducted with tick rates set at 30 Hz, 3 Hz, and 1 Hz. The objective is to investigate the factors, except for prediction accuracy, that represent the predictive ability of methods. This provides an explanation for the remaining disparity between prediction accuracy and driving performance.

We collect 50 scenarios for each predictor in each experiment. For each scenario, we randomly select the start and end point for the ego-agent from one of the four real-world maps provided by the SUMMIT simulator. A reference path of 50 meters is maintained between the two points, and the ego-agent is instructed to follow this path. A certain number of exo-agents including pedestrians, cyclists, and vehicles is randomly distributed within the environment. We implement all selected predictors in the simulator, except for HOME and DSP, due to their significantly longer running time, making the closed-loop evaluation infeasible. These two methods are solely employed in the Sim-Real Alignment, as shown in the supplementary materials. Notably, the RVO planner produces identical results for these two experimental settings since it conducts prediction only once per timestep.

To investigate the correlation between prediction accuracy and driving performance, we train all selected motion prediction models on the Alignment dataset collected from the SUMMIT simulator. We collect 59,944 scenarios and separate them into two groups: 80% training and 20% validation. Each scenario consists of about 300 steps. Subsequently, it is filtered down to 50 steps by taking into account the number of agents and their occurrence frequency. The nearest three agents are randomly selected to be the interested agent for prediction. The data collection was conducted on a server equipped with an Intel(R) Xeon(R) Gold 5220 CPU. We use four NVIDIA GeForce RTX 2080 Ti to speed up the running.

5 Experimental Results

The experiment part is designed and organized to answer the following questions: 1 ) Can the current prediction evaluation system accurately reflect the driving performance?; 2 ) What is the main factor causing the disparity between prediction accuracy and driving performance?; and 3 ) How do we propose evaluating predictors based on driving performance?

5.1 The Limitation of Current Prediction Evaluation

This section illustrates the limitation of current prediction evaluation systems in accurately reflecting realistic driving performance. We use ADE as an example, with FDE results provided in the supplementary materials. We refer to the ADE and minADE calculated in the Alignment dataset as Static ADE and Static minADE since they are obtained from static evaluation. Fixed prediction ability experiments are conducted for both RVO and DESPOT planners in the SUMMIT simulator.

Static ADE. As revealed in Figure 3a, there is no significant correlation between Static ADE and driving performance. Specifically, in the DESPOT planner, we observe a counterintuitive positive relationship. According to this finding, higher Static ADE would imply better driving performance. However, this hypothesis lacks a realistic basis and should be rejected since it exceeds the 95% confidence interval. Static ADE can not serve as a reliable indicator of driving performance.

Static minADE. To consider the impact of multi-modal prediction, we further investigate the correlation between Static minADE and driving performance. As depicted in Figure 3b, the multimodal prediction accuracy better reflects the driving performance. However, in the RVO planner, the linear regression exceeds the 95% confidence interval, thereby weakening the credibility of the evaluation metric. Furthermore, we still observe the positive correlation between prediction error and driving performance in the DESPOT planner, despite the lack of a realistic basis. These two observations prevent us from asserting the efficacy of Static minADE.

1

.

75

2

.

00

2

.

25

2

.

50

2

.

75

3

.

00

3

.

25

Static ADE

0

.

70

0

.

72

0

.

74

0

.

76

0

.

78

Driving Performance

CA

KNN

S-KNN

HiVT LaneGCN

LSTM

S-LSTM

R 2

: 0.00

/u1D45D

-value: 0.932

1

.

75

2

.

00

2

.

25

2

.

50

2

.

75

3

.

00

3

.

25

Static ADE

0

.

62

0

.

64

0

.

66

0

.

68

0

.

70

0

.

72

Driving Performance

CA

KNN

S-KNN

HiVT

LaneGCN

LSTM

S-LSTM

R 2

: 0.42

/u1D45D

-value: 0.080

1

.

00

1

.

25

1

.

50

1

.

75

2

.

00

2

.

25

2

.

50

Static minADE

0

.

70

0

.

72

0

.

74

0

.

76

0

.

78

Driving Performance

CA

KNN

S-KNN

HiVT

LaneGCN

LSTM

S-LSTM

R 2

: 0.49

/u1D45D

-value: 0.055

1

.

0

1

.

5

2

.

0

2

.

5

Static minADE

0

.

56

0

.

58

0

.

60

0

.

62

0

.

64

0

.

66

0

.

68

Driving Performance

CA

KNN

S-KNN

HiVT

LaneGCN

LSTM

S-LSTM

R 2

: 0.04

/u1D45D

-value: 0.628

(

a

)

(

b

)

1

.

8

2

.

0

2

.

2

2

.

4

2

.

6

2

.

8

3

.

0

Dynamic ADE

0

.

70

0

.

72

0

.

74

0

.

76

0

.

78

Driving Performance

CA

KNN

S-KNN

HiVT

LaneGCN

LSTM

S-LSTM

R 2

: 0.59

/u1D45D

-value: 0.026

2

.

0

2

.

5

3

.

0

3

.

5

4

.

0

4

.

5

5

.

0

Dynamic ADE

0

.

62

0

.

64

0

.

66

0

.

68

0

.

70

0

.

72

Driving Performance

CA

KNN

S-KNN

HiVT

LaneGCN

LSTM

S-LSTM

R 2

: 0.26

/u1D45D

-value: 0.202

(

a

)

(

b

)

10

20

30

Dynamic ADE

0

.

60

0

.

65

0

.

70

0

.

75

Driving Performance

R 2

: 0.68

/u1D45D

-value: 0.000

30Hz

3Hz

1Hz

In summary, current evaluation metrics merely represent the vanilla accuracy of predictors and do not fully reflect the driving performance. It is urgent to identify the overlooked factor(s) on this issue.

5.2 The Dominant Factor: Dynamics Gap

The disparity between static evaluation metrics and driving performance can be attributed to various factors. However, due to the lack of quantitative analysis, it remains unclear which factor primarily contributes to this disparity. In this section, we utilize fixed prediction ability experiments to identify the dominant factor affecting the disparity between prediction accuracy and driving performance. We use Dynamic ADE as an example. Similar to Static ADE, Dynamic ADE is calculated as the average L2 distance between the forecasted trajectory and the ground truth. However, the dynamic prediction accuracy in the interactive simulator differs from that of the Alignment dataset, since agents behave differently. This leads to the difference between Dynamic ADE and Static ADE.

Dynamics Gap. Figure 4 reports the correlation between Dynamic ADE and driving performance for both RVO and DESPOT planners. Compared to Static ADE, Dynamic ADE displays a significantly stronger correlation with driving performance for both planners. Quantitatively, using the correlation coefficient as a measurement, the dynamics gap accounts for 77.0% of the inconsistency between Static ADE and driving performance in the RVO planner, and 70.3% in the DESPOT planner. This emphasizes the dominant role of the dynamics gap in addressing the challenge of implementing predictors in the real world.

Furthermore, we compare the dynamics gap with factors that influence prediction accuracy and potentially affect driving performance. Except for the multi-modal prediction, the asymmetry of prediction errors [21] and occlusion are selected for comparison. We measure the significance of factors by assessing their impact on the correlation coefficient between prediction accuracy and driving performance. The disparity between static ADE and driving performance is denoted as Total .

Multi-Modal Prediction. We examine the variation in correlation coefficient from single-trajectory prediction to their multi-trajectory variants.

Table 4: Impacts on the correlation coefficient between prediction accuracy and driving performance.

Prediction Asymmetry. We check the change in correlation coefficient from considering all agents' predictions to considering only the interested agent that may affect the ego-plan.

Occlusion. We examine the variation in correlation coefficient from considering only agents with complete observations to considering agents with missed observations.

As depicted in Table 4, considering the asymmetry of prediction errors, multi-modal prediction, and dynamics gap can mitigate the disparity between prediction accuracy and driving performance. The consideration of occlusion does not yield satisfactory results in the RVO planner, as the planner mainly focuses on short-term predictions and is minimally affected. The asymmetric of prediction errors has a noticeable impact on both planners, highlighting the importance of predictions involving agents that may affect ego-planning. The multi-modal prediction exerts a significant impact on the RVO planner. However, when a complex planner such as DESPOT is used in conjunction, the impact becomes much weaker. It is reasonable to assume that the sampling-based feature of the DESPOT planner mitigates the influence of prediction uncertainty. Among various metrics, the dynamics gap exhibits superior performance in both planners and effectively resolves the majority of the disparity.

We can conclude that the dynamics gap is the main factor that causes the disparity between prediction accuracy and realistic driving performance. The Dynamic ADE, which is evaluated through interactive simulation environments, is capable of incorporating the dynamics gap and displaying a significant correlation with driving performance.

5.3 On the Predictive Ability of Methods

Although the dynamics gap accounts for the majority of the disparity between prediction accuracy and driving performance, the correlation between Dynamic ADE and driving performance is unsatisfactory in the DESPOT planner. To address this issue, we must answer two additional questions: 1 ) Does prediction accuracy fully reflect the predictive ability of methods? 2 ) If not, what other factor(s) is most important? We examine three well-known but never fully investigated factors and their correlation with driving performance: the temporal consistency of predictions [37], the distribution of prediction errors, and the computational efficiency of predictors.

Temporal Consistency. Each prediction scenario contains multiple successive frames within a fixed temporal chunk. Any two overlapping chunks of input data with a small time-shift should produce consistent results. The difference between the predictions of the two overlapping chunks is measured as the temporal consistency, with time-shift set to one frame.

Error Distribution. The error distribution of motion prediction contains the mean and variance. While the mean has been captured by the prediction accuracy, we incorporate the variance of prediction errors to account for the remaining impact.

Computational Efficiency. The computational efficiency of predictors is determined by their inference time when applied to downstream tasks. Feature extraction time is also included.

According to Figure 6, surprisingly, the temporal consistency of predictions and the distribution of prediction errors show a weak correlation with driving performance in both planners. However, the computational efficiency of predictors exerts a significant influence on the driving performance, as shown in Table 5. In three sub-experiments, the ranking of driving performance is aligned with the ranking of predictors' inference time in the DESPOT planner. We can conclude that the prediction accuracy can not fully represent the predictive ability of methods, and the computational efficiency of predictors is also a significant metric that affects driving performance.

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

1

.

2

1

.

4

Temporal Consistency

0

.

70

0

.

72

0

.

74

0

.

76

0

.

78

Driving Performance

CA

KNN

S-KNN

HiVT

LaneGCN

LSTM

S-LSTM

R 2

: 0.07

/u1D45D

-value: 0.519

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

1

.

75

Temporal Consistency

0

.

56

0

.

58

0

.

60

0

.

62

0

.

64

0

.

66

0

.

68

Driving Performance

CA

KNN

S-KNN

HiVT

LaneGCN

LSTM

S-LSTM

R 2

: 0.13

/u1D45D

-value: 0.374

2

.

2

2

.

4

2

.

6

2

.

8

3

.

0

Variance of Dynamic ADE

0

.

70

0

.

72

0

.

74

0

.

76

0

.

78

Driving Performance

CA

KNN

S-KNN

HiVT

LaneGCN

LSTM

S-LSTM

R 2

: 0.15

/u1D45D

-value: 0.338

2

.

50

2

.

75

3

.

00

3

.

25

3

.

50

3

.

75

4

.

00

Variance of Dynamic ADE

0

.

56

0

.

58

0

.

60

0

.

62

0

.

64

0

.

66

0

.

68

Driving Performance

CA

KNN

S-KNN

HiVT

LaneGCN

LSTM

S-LSTM

R 2

: 0.03

/u1D45D

-value: 0.669

(

a

)

(

b

)

5.4 The Trade-Off between Accuracy and Speed

It should be noted that there is a trade-off between computational efficiency and dynamic prediction accuracy for predictors to derive driving performance. As shown in Figure 5, the correlation between Dynamic ADE and driving performance becomes less strong when the tick rate is set higher. This is indicated by the data points deviating further from the best-fit-line in higher tick rates. At this time, it is the computational efficiency rather than dynamic prediction accuracy that decides the driving performance, as shown in Table 5. When the tick rate is set to 30Hz, the planner cannot generate an optimal solution, whereby the ranking of driving performance is determined by computational efficiency. When the tick rate is set to 3Hz, the CA outperforms CV since they have near-optimal solutions. When the Tick Rate is set to 1Hz, the LSTM also outperforms CV. The driving performance is determined by both dynamic prediction accuracy and computational efficiency of predictors in a trade-off manner, highlighting the significance of task-driven prediction evaluation.

6 Conclusion

Our study demonstrates the limitation of current prediction evaluation systems in accurately reflecting realistic driving performance. We identify the dynamics gap as the dominant factor contributing to this disparity. Furthermore, our findings reveal that the ultimate driving performance is determined by the trade-off between prediction accuracy and computational efficiency, rather than solely relying on prediction accuracy. We recommend further research incorporating interactive and task-driven evaluation protocols to assess prediction models. Despite these insights, there is still work to be done in the future. Our study focuses on sampling-based and reactive planners, and incorporating optimization-based or geometric planners could further substantiate our conclusions. Oracle perception data from the simulator can not fully represent the complexities of real-world situations. It would be beneficial to use raw sensor data to understand the comprehensive interplay within the entire AD system.

Acknowledgments and Disclosure of Funding

This research is supported in part by National Key R&D Program of China under Grant 2020YFB1600202, National Research Foundation of Singapore under its AI Singapore Programme (AISG Award No: AISG2-PhD-2022-01-036[T]), and China Scholarship Council.

References

Baidu Apollo team (2017), Apollo: Open Source Autonomous Driving. https://github. com/ApolloAuto/apollo .

Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social lstm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 961-971, 2016.

Ivo Batkovic, Mario Zanon, Nils Lubbe, and Paolo Falcone. A computationally efficient model for pedestrian motion prediction. In 2018 European control conference (ECC) , pages 374-379. IEEE, 2018.

Julian Bernhard, Klemens Esterle, Patrick Hart, and Tobias Kessler. Bark: Open behavior benchmarking in multi-agent environments. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 6201-6208. IEEE, 2020.

Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11621-11631, 2020.

Panpan Cai, Yiyuan Lee, Yuanfu Luo, and David Hsu. Summit: A simulator for urban driving in massive mixed traffic. In 2020 IEEE International Conference on Robotics and Automation (ICRA) , pages 4023-4029. IEEE, 2020.

Luigi Cardamone, Daniele Loiacono, and Pier Luca Lanzi. On-line neuroevolution applied to the open racing car simulator. In 2009 IEEE Congress on Evolutionary Computation , pages 2622-2629. IEEE, 2009.

Vinton G Cerf. A comprehensive self-driving car test. Communications of the ACM , 61(2), 2018.

Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, et al. Argoverse: 3d tracking and forecasting with rich maps. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8748-8757, 2019.

Felipe Codevilla, Antonio M Lopez, Vladlen Koltun, and Alexey Dosovitskiy. On offline evaluation of vision-based driving models. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 236-251, 2018.

R Craig Coulter. Implementation of the pure pursuit path tracking algorithm. Technical report, Carnegie-Mellon UNIV Pittsburgh PA Robotics INST, 1992.

Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator, 2017.

Alec Farid, Sushant Veer, B. Ivanovic, Karen Leung, and M. Pavone. Task-Relevant Failure Detection for Trajectory Predictors in Autonomous Vehicles. 2022.

Thomas Gilles, Stefano Sabatini, Dzmitry Tsishkou, Bogdan Stanciulescu, and Fabien Moutarde. Home: Heatmap output for future motion estimation. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC) , pages 500-507. IEEE, 2021.

Thomas Gilles, Stefano Sabatini, Dzmitry Tsishkou, Bogdan Stanciulescu, and Fabien Moutarde. Gohome: Graph-oriented heatmap output for future motion estimation. In 2022 International Conference on Robotics and Automation (ICRA) , pages 9107-9114. IEEE, 2022.

Junru Gu, Chen Sun, and Hang Zhao. Densetnt: End-to-end trajectory prediction from dense goal sets. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15303-15312, 2021.

Mahir Gulzar, Yar Muhammad, and Naveed Muhammad. A survey on motion prediction of pedestrians and vehicles for autonomous driving. IEEE Access , 9:137957-137969, 2021.

Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2255-2264, 2018.

Xin Huang, Guy Rosman, Ashkan Jasour, Stephen G McGill, John J Leonard, and Brian C Williams. Tip: Task-informed motion prediction for intelligent vehicles. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 11432-11439. IEEE, 2022.

Maximilian Igl, Daewoo Kim, Alex Kuefler, Paul Mougin, Punit Shah, Kyriacos Shiarlis, Dragomir Anguelov, Mark Palatucci, Brandyn White, and Shimon Whiteson. Symphony: Learning realistic and diverse agents for autonomous driving simulation. In 2022 International Conference on Robotics and Automation (ICRA) , pages 2445-2451. IEEE, 2022.

Boris Ivanovic and Marco Pavone. Injecting planning-awareness into prediction and detection evaluation. In 2022 IEEE Intelligent Vehicles Symposium (IV) , pages 821-828. IEEE, 2022.

Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, and Raquel Urtasun. Learning lane graph representations for motion forecasting. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16 , pages 541-556. Springer, 2020.

Pablo Alvarez Lopez, Michael Behrisch, Laura Bieker-Walz, Jakob Erdmann, Yun-Pang Flötteröd, Robert Hilbrich, Leonhard Lücken, Johannes Rummel, Peter Wagner, and Evamarie Wießner. Microscopic traffic simulation using sumo. In 2018 21st international conference on intelligent transportation systems (ITSC) , pages 2575-2582. IEEE, 2018.

Rowan McAllister, Blake Wulfe, Jean Mercat, Logan Ellis, Sergey Levine, and Adrien Gaidon. Control-aware prediction objectives for autonomous driving. In 2022 International Conference on Robotics and Automation (ICRA) , pages 01-08. IEEE, 2022.

Sajjad Mozaffari, Omar Y Al-Jarrah, Mehrdad Dianati, Paul Jennings, and Alexandros Mouzakitis. Deep learning-based vehicle behavior prediction for autonomous driving applications: A review. IEEE Transactions on Intelligent Transportation Systems , 23(1):33-47, 2020.

Andreas Møgelmose, Mohan M. Trivedi, and Thomas B. Moeslund. Trajectory analysis and prediction for improved pedestrian safety: Integrated framework and evaluations. In 2015 IEEE Intelligent Vehicles Symposium (IV) , pages 330-335, June 2015.

Jiquan Ngiam, Vijay Vasudevan, Benjamin Caine, Zhengdong Zhang, Hao-Tien Lewis Chiang, Jeffrey Ling, Rebecca Roelofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, et al. Scene transformer: A unified architecture for predicting future trajectories of multiple agents. In International Conference on Learning Representations , 2021.

Jonah Philion, Amlan Kar, and Sanja Fidler. Learning to evaluate perception models using planner-centric metrics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14055-14064, 2020.

Andrey Rudenko, Luigi Palmieri, Michael Herman, Kris M Kitani, Dariu M Gavrila, and Kai O Arras. Human motion trajectory prediction: A survey. The International Journal of Robotics Research , 39(8):895-935, 2020.

Christoph Schöller, Vincent Aravantinos, Florian Lay, and Alois Knoll. What the constant velocity model can teach us about pedestrian motion prediction. IEEE Robotics and Automation Letters , 5(2):1696-1703, 2020.

Marcel Schreiber, Vasileios Belagiannis, Claudius Gläser, and Klaus Dietmayer. Dynamic Occupancy Grid Mapping with Recurrent Neural Networks. In 2021 IEEE International Conference on Robotics and Automation (ICRA) , pages 6717-6724, May 2021.

Adhiraj Somani, Nan Ye, David Hsu, and Wee Sun Lee. Despot: Online pomdp planning with regularization. Advances in neural information processing systems , 26, 2013.

Simon Suo, Sebastian Regalado, Sergio Casas, and Raquel Urtasun. Trafficsim: Learning to simulate realistic multi-agent behaviors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10400-10409, 2021.

Jur Van den Berg, Ming Lin, and Dinesh Manocha. Reciprocal velocity obstacles for real-time multi-agent navigation. In 2008 IEEE international conference on robotics and automation , pages 1928-1935. Ieee, 2008.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Nick Webb, Dan Smith, Christopher Ludwick, Trent Victor, Qi Hommes, Francesca Favaro, George Ivanov, and Tom Daniel. Waymo's safety methodologies and safety readiness determinations. arXiv preprint arXiv:2011.00054 , 2020.

Maosheng Ye, Jiamiao Xu, Xunnong Xu, Tengfei Wang, Tongyi Cao, and Qifeng Chen. Bootstrap motion forecasting with self-consistent constraints, 2023.

Lu Zhang, Peiliang Li, Jing Chen, and Shaojie Shen. Trajectory prediction with graph-based dual-scale context fusion. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 11374-11381. IEEE, 2022.

Hang Zhao, Jiyang Gao, Tian Lan, Chen Sun, Ben Sapp, Balakrishnan Varadarajan, Yue Shen, Yi Shen, Yuning Chai, Cordelia Schmid, et al. Tnt: Target-driven trajectory prediction. In Conference on Robot Learning , pages 895-904. PMLR, 2021.

Zikang Zhou, Luyao Ye, Jianping Wang, Kui Wu, and Kejie Lu. HiVT: Hierarchical Vector Transformer for Multi-Agent Motion Prediction. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8813-8823, June 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a timely and well-executed study on motion prediction, focusing on the relationship between the dominant evaluation paradigm (static, offline metrics) and the true objective of research in this domain (safer planning). The study uncovers several important and surprising findings: (1) static offline metrics are not correlated to planning performance, (2) evaluating on the frames observed by the planner (dynamic offline metrics) are significantly better correlated, (3) computational efficiency in prediction is increasingly important for sophisticated planners, and (4) simple baselines for prediction (constant velocity/constant acceleration) perform best when considering downstream planning performance.\n\nSTRENGTHS:\nThe study setup is excellent, making well-motivated choices for the task, simulator, predictors, planners, metrics, etc. The writing is clear. The presentation of the key findings both visually and numerically also makes this paper an interesting and easy-to-understand read. Most importantly, it highlights a flaw in the widely used metrics that the community optimizes motion prediction models for, and shows that naive baselines are sufficient to outperform SoTA learned forecasting when using the right metrics.\n\nWEAKNESSES:\nOverall, the paper has very few weaknesses in my opinion. While the selected ML prediction models do not include the current leaderboard winners, I believe they are representative. The experiments could have been conducted on more realistic scenarios (e.g. the nuPlan simulator), but Section 5.1 shows that the simulator used is sufficiently aligned to real datasets for the purpose of this study. However, some technical details regarding the study are unclear (please see the “Questions” section #2, #3, #4 for specifics).\n\nQUESTIONS:\n1. A very interesting finding that I suggest should be highlighted more is that in all closed-loop evaluation settings, the best among the prediction methods used in this study is always either CV or CA - the naive baselines.\n2. Do the CV and CA baselines forecast the positions along a straight line (heading direction), or do they take into account angular speed/acceleration as well?\n3. In L316, could you please elaborate on what \"complete/incomplete observations\" refers to?\n4. L328 recommends Dynamic FDE with Closest, when closed loop simulation is possible. However, why would this be preferable to simply ranking prediction methods by the “Driving Performance” of the planner?\n5. Please cite https://arxiv.org/abs/1809.04843 while wrapping up Section 2.2 - this is a well-known study with a similar setup, aimed at planning instead of motion prediction. It would also be interesting to contrast your findings to theirs, given the change in task.\n6. Could part of the discrepancy arise from the fact that prediction models penalize errors on all vehicle instances equally, while some are less relevant for planning than others? The result in Table 4 showing the improved correlation when considering only the nearest agents seems to indicate this.\n    1. Could you add a row with the “Closest” checkmark for the static, multimodal ADE?\n    2. If it is possible to include a task-aware offline metric such as PKL [15], CAPO [18] or some equivalent in the analysis, this would add significant value to the paper, and I would be happy to increase my rating.\n\nMinor:\n\n1. L078: Lastname et al. instead of Firstname et al.\n\nUpdate:\n\nThank you for taking the time to carefully answer my questions, I appreciate the effort. The rebuttal clarifies most of my questions. I understand the choice of Dynamic ADE/FDE for the experiments in this paper (as the rebuttal points out, \"a prediction metric that employs the same calculation methodology but behaves distinctively in these two evaluation frameworks\" helps provide evidence for their claims).\n\nI am still leaning towards a positive rating as I see the main contribution of this paper as highly relevant to the motion prediction community for driving. This is a large research area with hundreds of published papers, and little incentive to move away from static metrics (even though the lack of alignment between open-loop and closed-loop metrics for planning is reasonably well-known). This paper opens up a discussion on the inadequacy of the prevalent motion forecasting metrics, and could be a valuable first step towards more comprehensive evaluation of prediction models in the future.\n\nLIMITATIONS:\nLimitations are discussed in Section 6.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors examine the discrepancy between trajectory prediction accuracy and driving performance in the task of autonomous driving when a fixed dataset is used and in the presence of multiple surrounding traffic participants (such as other vehicles). Specifically, they assert that this discrepancy arises from:\n-  A dynamics gap between the static dataset that the trajectory predictor is trained on and its subsequent usage for planning in real-world driving scenario. This is because the ego-vehicle’s planner may act differently to the static dataset, which results in novel reactions from the other entities in the environment (vehicles, pedestrians, etc.) and thus out-of-distribution scenarios occur.\n- Computational efficiency issues caused by the planner and trajectory predictor being too slow to accommodate real-time operation.\n\nAlthough the work does not propose a novel method, the authors conduct of study of existing methods to illustrate the importance of a dynamic approach where the ego-vehicle collects data with the specific predictor (instead of relying on a single fixed static dataset) and efficient trajectory prediction. \n\nA wide variety of trajectory prediction models are used (Table 1) to ensure high test coverage along with two planners (RVO and DESPOT) (section 4.2)\n\nFor trajectory prediction evaluation they use the Average/Final Distance Error (ADE/FDE) in Table 2. To evaluate driving performance a combination of collision avoidance, speed to goal and comfort (minimize jerk) are considered (section 4.4).\n\nExperiments first attempt to illustrate how their chosen simulator (SUMMIT, extended from Carla) is suitable to evaluate real-world performance based on simulation (section 5.1, Figure 2) as trajectory prediction simulation results (SUMMIT) and real-world dataset results (Argoverse dataset) are positively correlated.\n\nThey then illustrate the performance gap that occurs when evaluating against a static dataset versus one collected interactively (dynamically) from simulation where the ego vehicle operates with the predictor (section 5.2, Figure 3. versus Figure 4). Most notably, they find that efficiency concerns are most prevalent under a high planner-simulation tick rate. In this case the predictor efficiency has the greatest effect on driving performance instead of the dynamics gap since it cannot keep up with the high tick rate (Table 3). Table 4 further shows that the discrepancy between driving performance and trajectory prediction accuracy (ADE) is reduced when only nearby agents with full observations is preferred in addition to dynamic data collection.\n\nSTRENGTHS:\n- Experiments are well done and compare a variety of approaches (ten trajectory prediction models, two planners).\n\n- Claims are backed clearly by data. The results show the value of using an interactive simulation environment for data gathering and favoring efficient predictors when doing real-time planning for autonomous driving.\n\n- In general, well written and clear.\n\nWEAKNESSES:\n- The work does not propose a novel approach but is instead of survey of existing methods. \n\n- It could also be argued that the survey results are somewhat obvious (which compounds with the first criticism): 1) relying solely on a static dataset for model-based training may result in a distributional shift during real-world evaluation once planning is done, causing failure and 2) efficient predictors are important for real-time planning.\n\nMinor Criticism: \n\n- The combination of performance metrics $P_{\\mathrm{Safety}}, P_{\\mathrm{Efficiency}}, P_{\\mathrm{Comfort}}$ could be clarified in the text (section 4.4). Sensibly, $P_{\\mathrm{Safety}}$ and $P_{\\mathrm{Comfort}}$ are to be minimized while $P_{\\mathrm{Efficiency}}$ is maximized. But all metrics are labeled “performance” so I was originally expecting them to be handled uniformly. The text does mention that “we normalized the direction of these three metrics” but this is still somewhat ambiguous. Things are fully clarified once looking at the supplemental material where the sign on $P_{\\mathrm{Safety}}$ and $P_{\\mathrm{Comfort}}$ are reversed during normalization. However, I would find it helpful if this was more clearly mentioned to in the text.\n\n- I feel that the first experimental section (section 5.1) is perhaps not relevant to the main purpose of the paper besides illustrating that trajectory prediction accuracy in simulation and the real-world are positively correlated across different models. I imagine most semi-photorealistic simulation environments would possess this property, especially given its extension from the mature Carla simulator. They also do not explicitly show that driving performance is ultimately related (only trajectory prediction performance). There does appear to be a notable ADE/FDE error increase when moving to the real-world dataset. Thus, once applied to planning, I could see a case where a model might perform “well” in simulation but “poorly” in the real world (despite performing better than its peers).\n\nQUESTIONS:\n- Should the minADE and minFDE equations in Table 2 be a function of k? I assume that each predicted  $\\hat{x}_i$ is a sample taken K times?\n\n- In Table 3, is the “Inference Time” for one network inference loop and not the overall time for planning? If it is the overall time for planning I would be confused as to why performance changes for some models below the 30hz refresh rate when going from 30 to 3 to 1 hz.\n\n- In section 5.1, lines 237-238 it says that the “nearest three agents are randomly selected to be the interested agent for prediction”. Is there a reason why only 3 agents were considered? Was this the best trade-off between computation time and performance?\n\nLIMITATIONS:\nThe authors did not explicitly list any limitations of their method. However, their work is primarily a study of existing methods and so a limitations section is perhaps less relevant. Nonetheless, one possible limitation is the focus solely on simulation for illustrating static versus dynamic driving performance discrepancy. Please refer to the weaknesses section for more possible limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
The paper provides an extensive study on the discrepancy between model prediction accuracy and driving performance. It explores two major factors: 1. The dynamics difference; and 2. The computational efficiency of predictors. Various methods are tested in two settings: 1. Fixed Number of Predictions; and 2. Fixed Planning Time. The paper evaluates the model performance on both open-loop prediction and closed-loop diving.

STRENGTHS:
The experiments are well-designed and extensive, which provides strong evidence for the statements in the paper.

WEAKNESSES:
While the two factors are well supported by the experiment results given, it is insufficient to say whether other factors are less important, such as the data quality of the open loop eval, the unknowability (future events influencing the ground truth future trajectory) issue for open loop prediction, as well as multimodality in the diving scenarios.

QUESTIONS:
It would be great if the author could provide add some discussions on possible other factors that cause the discrepancy and why the two factors mentioned in the paper are dominant.

LIMITATIONS:
See questions.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work attempts to look for evaluation metrics for trajectory prediction beyond the widely used ADE/FDE metrics. In particular, the authors focus on two overlooked aspects in prediction evaluation, which are: 1) the dynamics gap between the dataset and closed-loop driving scenarios; 2) the trade-off between computational efficiency and performance. The authors argue that we should turn to closed-loop prediction evaluation to address these two problems.\n\nSTRENGTHS:\nThis work focuses on a very important and open question for trajectory prediction, which is looking for evaluation metrics for trajectory prediction beyond the widely used ADE/FDE metrics. It is good that the authors emphasize and validate the limitation of open-looped prediction accuracy and the importance of closed-loop evaluation for trajectory prediction.\n\nWEAKNESSES:\n1. It is good to see that the authors showed the limitation of open-looped evaluation and the importance of closed-loop evaluation in their experiments. However, I did not think the authors provided any new perspective or insightful solution to the problem of closed-loop evaluation. The value of closed-loop evaluation has been widely acknowledged in the autonomous driving community. It is one of the main motivations behind the emerging research on data-driven traffic simulation. The bottleneck is not the lack of prediction evaluation metrics under the closed-loop evaluation scheme, which is targeted by one of the two claimed contributions of the paper (i.e., dynamic ADE vs. static ADE), but the closed-loop simulation itself. How to synthesize a reliable simulator with realistic reactive agent behavior is an open question. We even do not have a commonly acknowledged set of metrics for evaluating simulation at this stage. Unfortunately, the authors did not discuss the impact of the simulation environment on the closed-loop evaluation result in this paper (e.g., whether the simulated driving performance and the simulated dynamic ADE metric are correlated with real-world driving performance). While the authors adopted different prediction models and planners in their evaluation, they only used a single simulator (i.e., SUMMIT). Moreover, the authors did not include a literature review on traffic simulation and evaluation metrics for traffic simulation. \n\n2. The inherent logic behind the experiments is fundamentally flawed. In Sec. 5.2, the authors showed that static ADE is not correlated with dynamic ADE and driving performance. However, the authors used the correlation of static ADEs on the Argoverse and the Alignment dataset collected from SUMMIT as evidence to validate the realism of SUMMIT. Just like the authors' observation in Sec. 5.2, the correlation in static ADE does not necessarily indicate that the SUMMIT simulation is reliable and realistic under closed-loop simulation and evaluation. To justify the realism of SUMMIT, the authors should adopt those evaluation metrics from the latest literature and compare SUMMIT against those state-of-the-art simulation models.\n\nQUESTIONS:\n1. I do not think the authors appropriately answer the question raised in the title (i.e., what truly matters in trajectory prediction for autonomous driving?). The paper focuses on two perspectives on this question, which are very limited and far from compiling a comprehensive and insightful answer. The experimental setting is also insufficient as an attempt to answer such a bold question (e.g., lacking analysis on the impact of simulation on closed-loop evaluation). \n2. It would be beneficial to have some qualitative analysis and visualization beyond statistics. For instance, visualizing some representative examples could be helpful in understanding why static ADE and dynamic ADE are not correlated. \n3. The citation for SUMMIT is missing in the main text.\n\nLIMITATIONS:\nThe authors discussed some limitations and future directions in Sec. 6. However, I think the limitations are not sufficiently addressed. For instance, I would consider the limited evaluation on a single simulator and the missing discussion on the impact of simulation realism as crucial drawbacks of this work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn the article, the author discusses the limitations of the current open-loop trajectory evaluation method: 1. It lacks consideration of closed-loop interactions with the ego-planner 2. It overlooks computational timeliness. These two factors result in a misalignment between open-loop trajectory evaluations and closed-loop driving scores. The author also proposes a new metric, Dynamic ADE/FED, which enables a positive correlation to be established between open-loop trajectory evaluations and closed-loop driving scores.\n\nSTRENGTHS:\n* Originality: This paper is original, providing a new evaluation method for the field of trajectory prediction.\n* Significance: The foundation of this paper is sound. It focuses on analyzing the flaws and issues in the standards of trajectory prediction evaluation, and conducts a substantial amount of experiments across different models.\n* Quality & Clarity: I believe this section is the weakness of the paper. Please refer to the 'Weakness' part.\n\nWEAKNESSES:\n* Clarity & Quality: \n    - 1. The core Dynamic ADE/FDE metrics in this paper are not clearly explained; there is hardly any section dedicated specifically to elaborating on this aspect. It would be beneficial to include an illustration for better understanding. \n\n    - 2. Despite the numerous experiments conducted, some of them fail to support the arguments made. For instance, on lines 243-245, you stated that the positive correlation demonstrated by different prediction models on the Argoverse and Alignment datasets indicates that there isn't a significant domain gap between the SUMMIT simulator and real-world data. I find this reasoning flawed. A positive correlation merely indicates consistency in the predictive ability of the models across the two datasets; it does not necessarily imply that the behaviors in the two datasets are consistent. Moreover, a closer look at the specific numerical values on the x and y-axes in Figure 2 reveals significant discrepancies between the Argoverse and Alignment datasets. The ADE distribution for Argoverse lies between 1.5-3.5, while that for Alignment falls between 0.5-1.5. This suggests that the Argoverse dataset is substantially more challenging than the Alignment dataset, which contradicts your claim. \n    - 3. Besides this, there are many other similar issues. The overall experimental conclusions lack substantial empirical support and are simply based on statistical analyses of a small number of data points, yielding correlations that aren't particularly strong (the paper only employs eight different prediction models and two different planning models). I remain unconvinced by statistical conclusions drawn from such a limited set of data points. \n* Novelty: \n    - 4. This paper seems more akin to an experimental report. Despite the multitude of experiments conducted, they remain rather superficial. For a NeurIPS paper, a deeper level of analysis is required, such as quantitative analysis, comparing what specific interactions in which scenarios cause differences between static and dynamic ADE in open-loop and closed-loop evaluations. Additionally, as I previously mentioned, many of the experimental results do not effectively support the conclusions drawn. \n    - 5. The work done in the 'Computational Efficiency' section is quite trivial. The conclusions drawn here are rather obvious, and in the field of autonomous driving engineering, this aspect has already been thoroughly considered.\n\nQUESTIONS:\nPlease refer to the 'weaknesses' section.\n\nLIMITATIONS:\nPlease refer to the 'weaknesses' section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-20"}