SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Phong Tran $^{1*}$  Haoran Wu $^{1,2*}$  Cunjun Yu $^{1*}$  Panpan Cai $^{3}$  Sifa Zheng $^{2}$  David Hsu $^{1}$
$^{1}$  National University of Singapore
3 Shanghai Jiao Tong University
Trajectory prediction plays a vital role in the performance of autonomous driving systems, and prediction accuracy, such as average displacement error (ADE) or final displacement error (FDE), is widely used as a performance metric. However, a significant disparity exists between the accuracy of predictors on fixed datasets and driving performance when the predictors are used downstream for vehicle control, because of a dynamics gap. In the real world, the prediction algorithm influences the behavior of the ego vehicle, which, in turn, influences the behaviors of other vehicles nearby. This interaction results in predictor-specific dynamics that directly impacts prediction results. In fixed datasets, since other vehicles' responses are predetermined, this interaction effect is lost, leading to a significant dynamics gap. This paper studies the overlooked significance of this dynamics gap. We also examine several other factors contributing to the disparity between prediction performance and driving performance. The findings highlight the trade-off between the predictor's computational efficiency and prediction accuracy in determining real-world driving performance. In summary, an interactive, task-driven evaluation protocol for trajectory prediction is crucial to capture its effectiveness for autonomous driving. Source code along with experimental settings is available online.
Current trajectory prediction evaluation [29, 9, 5] relies on real-world datasets, operating under the assumption that dataset accuracy is equivalent to prediction capability. We refer to this as Static Evaluation. This methodology, however, falls short when the predictor serves as a sub-module for downstream tasks in Autonomous Driving (AD) [28, 24]. As illustrated in Figure 1, the static evaluation metrics on datasets, such as Average Displacement Error (ADE) and Final Displacement Error (FDE), do not necessarily reflect the actual driving performance [36, 8, 10]. In contrast to the conventional focus on uncertainty [39, 16] and potential interaction [27], we demonstrate that this disparity stems from the overlooked dynamics gap between fixed datasets and AD systems and the computational efficiency of predictors. The trade-off between these two factors matters in the actual prediction performance within the entire AD system.
The dynamics gap arises from the fact that the behavior of the autonomous vehicle, also known as the ego-agent, varies with different trajectory predictors, as presented in Figure 2. In real-world scenarios, the ego-agent utilizes predictors to determine its actions. Different predictors result in varied behaviors of the ego-agent, which, in turn, influence the future behaviors of other road users, leading to different dynamics within the environment. This directly affects the accuracy of predictions, as other agents behave differently. Since the ego-agent's actions are predetermined on
Figure 1: Prediction accuracy vs driving performance. Contrary to popular belief (left, black curves), our study indicates no strong correlation between common prediction evaluation metrics and driving performance (left, red curves). Eight representative prediction models are selected: Constant Velocity (CV) [30], Constant Acceleration (CA) [30], K-Nearest Neighbor (KNN) [9], Social KNN (S-KNN) [9], HiVT [40], LaneGCN [22], LSTM, and Social LSTM (S-LSTM) [2]. The definition of driving performance metrics can be found in Section 4.4. Details can be found in the supplementary materials.
the dataset, there exists a significant disparity between the dynamics represented in the dataset and the actual driving scenario when evaluating a specific trajectory predictor. To tackle this issue, we propose the use of an interactive simulation environment to evaluate the predictor for downstream decision-making. This environment enables Dynamic Evaluation as the ego-agent operates with the specific predictor, thus, mitigating the dynamics gap. We demonstrate a strong correlation between the dynamic evaluation metrics and driving performance through extensive experiments. This underscores the dominant role of the dynamics gap in aligning prediction accuracy with driving performance and emphasizes the importance of incorporating it into the evaluation process.
The awareness of the dynamics gap significantly enhances the correlation between prediction accuracy and driving performance. However, there are still factors that affect driving performance apart from prediction accuracy. The downstream modules in AD systems, with different tasks and varying levels of complexity, impose different requirements on prediction models. In our experiments, we vary the type of planner and the time constraint for the motion planning task. Our findings suggest that predictors' computational efficiency plays a vital role in real-time tasks. Moreover, it is the trade-off between prediction accuracy and computational efficiency that determines driving performance. This highlights the necessity for task-driven prediction evaluation.
(b)
Figure 2: Dynamics Gap. (a) In static evaluation, the agent's motion is determined and unaffected by predictors. (b) In the real-world, different predictors result in varied behaviors of the agent, which directly affects the ground truth of prediction.
In this paper, we aim to address two specific aspects of trajectory prediction for AD systems. Firstly, we uncover the limitation of static prediction evaluation systems in accurately reflecting driving performance. We demonstrate the dominant role of the dynamics gap in causing the disparity. Secondly, we emphasize the necessity of an interactive, task-driven evaluation protocol that incorporates the trade-off between dynamic prediction accuracy and computational efficiency. The protocol presents a promising way for applying prediction models in real-world autonomous driving, which is the truly mattered aspect in trajectory prediction for autonomous driving.
2.1 Motion Prediction and Evaluation
Motion prediction methods can be classified along three dimensions [17]: modeling approach, output type, and situational awareness. The modeling approach includes physics-based models [30, 3]
that use physics to simulate agents' forward motion, and learning-based models [40, 22] that learn and predict motion patterns from data. The output type can be intention, single-trajectory [26], multi-trajectory [18], or occupancy map [14, 15]. These outputs differ in the type of motion they predict and how they handle the uncertainty of future states. The situational awareness includes unawareness, interaction [2], scene, and map awareness [31]. It refers to the predictor's ability to incorporate environmental information, which is crucial for collision avoidance and efficient driving. Most researchers [29, 25] and competitions [9, 5] evaluate the performance of prediction models on real-world datasets, in which ADE/FDE and their probabilistic variants minADE/minFDE are commonly used metrics. However, these metrics fail to capture the dynamics gap between datasets and real-world scenarios, as the actions of the ego-agent remain unaffected by predictors. In this study, we select four model-based and six learning-based models with varying output types and situational awareness to cover a wide range of prediction models. We implement these predictors in both datasets and an interactive simulation environment to illustrate the limitation of current prediction evaluation in accurately reflecting driving performance, due to the neglect of the dynamics gap.
2.2 Task-aware Motion Prediction and Evaluation
Task-aware motion prediction remains an underexplored area in research. While some studies touch upon the subject, they focus on proposing task-aware metrics for training or eliminating improper predictions on datasets. One notable example of training on task-aware metrics is the Planning KL Divergence (PKL) metric [28]. Although designed for 3D object detection, it measures the similarity between detection and ground truth by calculating the difference in ego-plan performance. In the context of motion prediction, Rowan et al. [24] propose a control-aware metric (CAPO) similar to PKL. CAPO employs attention [35] to find correlations between predicted trajectories, assigning higher weights to agents inducing more significant reactions. Similarly, the Task-Informed method [19] assumes a set of candidate trajectories from the planner and adds the training loss for each candidate. Another line of work focuses on designing task-aware functions to eliminate improper predictions [21, 13]. The proposed metric can capture unrealistic predictions and better correlate with driving performance in planner-agnostic settings. However, these works are conducted in an open-loop manner, neglecting the disparity between open-loop evaluation and real-world driving performance. To the best of our knowledge, we are the first to conduct a thorough investigation into the disparity and demonstrate the dominant role of the dynamics gap on this issue.
3 Planning with Motion Prediction
In this section, we present the problem formulation for motion prediction with planning in the context of autonomous driving. Different from conventional formulations that consider the planner as a policy, our formulation employs the predictor as the policy. This enables us to identify the primary challenge in developing predictors for real-world autonomous driving. We define the prediction problem as an MDP:  $M = (\mathcal{S},\mathcal{A},\mathcal{T},\mathcal{R})$
State. The state of the system is represented by the tuple  $s = (s^{ego}, s^{exo})$  for  $s \in S$ , where  $s^{ego}$  denotes the state of the AV including historical information,  $s^{exo}$  denotes that of surrounding traffic participants. Specifically,  $s^{ego} = \{s_{t - t_{obs}}^{ego}, \ldots, s_t^{ego}\}$ , where  $t_{obs}$  represents the observation horizon and  $t$  denotes the current timestep. The state for each vehicle includes its position, velocity, heading, and other relevant attributes, e.g.,  $s_t^{ego} = (x_t^{ego}, y_t^{ego}, v_t^{ego}, \theta_t^{ego})$ .
Action. At each time step, the AV can take an action  $a \in \mathcal{A}$ , where  $\mathcal{A}$  is the action space. We denote the  $n$  surrounding traffic participants as  $i \in \{1, \dots, n\}$ . The action consists of the AV's prediction of all exo-agents, e.g.,  $a = \{a^1, \dots, a^n\}$ .
Transition Function. The transition function  $\mathcal{T}:S\times \mathcal{A}\to S$  defines the dynamics of the system and how the system evolves as a result of the action of the ego vehicle. In the MDP, it refers to:  $T(s_{t + 1}|s_t,a_t) = T_1(s_{t + 1}^{ego}|s_t,a_t)T_2(s_{t + 1}^{exo}|s_t),T\in \mathcal{T}$ . We denote the planner as a mapping function  $T_{1}$  that takes the current state  $s_t$  and action  $a_{t}$  as input and outputs the AV's next state  $s_{t + 1}^{ego}$ . The motion models of exo-agents are integrated into the mapping function  $T_{2}$ , which takes the current state  $s_t$  as input and generates the next state  $s_{t + 1}^{exo}$  for all exo-agents.
Predictor. The predictor is represented as a policy  $a = \pi(s)$  that takes the current state  $s$  as input, and produces an action  $a \in \mathcal{A}$  as output.
Objective Function. The objective of prediction is to accurately predict exo-agents future states while considering their interactions and map information. The reward function  $R: S \times \mathcal{A} \to \mathbb{R}$  maps a state-action pair to a real-valued reward. The objective function is the accumulated reward over the task horizon  $H$ , defined as  $J(s, a) = \sum_{t=0}^{H} R(s_t, a_t)$ .
Optimal Policy. The objective of autonomous driving is to identify the optimal policy that minimizes the objective function under the real-world transition function. The optimal policy  $\pi^{*}$  is expressed as:  $\pi^{*} = \underset{T_{1} = T_{1}^{*},T_{2} = T_{2}^{*}}{\arg\max}\mathbb{E}_{\pi}[\sum_{t=0}^{H}R(s_{t},a_{t})]$ , where  $T_{1}^{*}$  represents any realistic planners used in AVs, and  $T_{2}^{*}$  denotes the integrated motion model of exo-agents in the real world.
3.2 The Limitation of Static Evaluation
Traditional prediction methods overlook the difference between the transition function  $T_{1}^{*}$  in real-world autonomous driving and the represented  $\hat{T}_{1}$  in datasets. These methods focus on training a policy using well-designed reward functions to perform effectively within the state distribution of datasets. The trained policy  $\pi$  satisfies:
$$
\pi = \underset {T _ {1} = \hat {T} _ {1}, T _ {2} = T _ {2} ^ {*}} {\arg \max } \mathbb {E} _ {\pi} \left[ \sum_ {t = 0} ^ {H} R \left(s _ {t}, a _ {t}\right) \right]. \tag {1}
$$
In this context,  $\hat{T}_1$  serves as a static planner that remains unaffected by predictors. Regardless of the input, the static planner  $\hat{T}_1$  generates the recorded next state. As a result, the mapping function  $T_2^*$  remains consistent with the real world since the input state is unchanged. We define the disparity in future states resulting from changes in the ego-planner as the dynamics gap in static evaluation, denoted as  $G = T_1^* -\hat{T}_1$ .
The proposed planning-aware metrics, such as PKL [28] and CAPO [24], aimed to address the disparity between prediction accuracy and the ultimate driving performance by improving the cost function. However, unless the dynamics gap is mitigated, the optimization deviation plays a dominant role in causing the disparity and should be addressed as a priority.
3.3 Measuring the Impact of Dynamics Gap
Conducting real-world tests is the most effective way to address and assess the impact of the dynamics gap. However, due to its high cost, the simulator serves as a proxy. The ego-agent utilizes the realistic planner  $T_{1}^{*}$  and the motion model  $T_{sim}^{*}$  of the simulator. The optimal policy  $\pi_{sim}^{*}$  satisfies:
$$
\pi_ {s i m} ^ {*} = \underset {T _ {1} = T _ {1} ^ {*}, T _ {2} = T _ {2} ^ {s i m}} {\arg \max } \mathbb {E} _ {\pi} [ \sum_ {t = 0} ^ {H} R (s _ {t}, a _ {t}) ]. \tag {2}
$$
To evaluate the impact of the dynamics gap, we train various predictors on the Alignment dataset obtained from the simulator, the trained policy  $\pi_{sim}$  satisfies:
$$
\pi_ {s i m} = \underset {T _ {1} = \hat {T} _ {1}, T _ {2} = T _ {2} ^ {s i m}} {\arg \max } \mathbb {E} _ {\pi} [ \sum_ {t = 0} ^ {H} R \left(s _ {t}, a _ {t}\right) ]. \tag {3}
$$
The equations suggest that the dynamics gap between the Alignment dataset and simulator mirrors the dynamics gap between the real-world dataset and actual autonomous driving scenarios. Thus, it is reasonable to employ the simulator for evaluating the influence of the dynamics gap. However, the simulator merely serves as a substitute to mitigate the dynamics gap, leaving a remaining dynamics gap represented by  $G' = T_2^* - T_2^{sim}$ .
Our aim is to identify the key factors involved in trajectory prediction for autonomous driving and recognize their impact on driving performance by simulating real-world scenarios that vehicles might encounter. The ultimate goal is to introduce an interactive, task-driven prediction evaluation protocol. To achieve this objective, we need to determine four crucial components: 1) motion prediction methods to be covered; 2) realistic planners to employ prediction models; 3) the simulator to replicate interactive scenarios; 4) evaluation metrics to assess the effectiveness of the key factors involved in motion prediction with respect to real-world driving performance.
Table 1: Selected prediction methods.
Modeling Approach	Method	Output Type	Interaction Aware	Scene Aware	Map Aware
Model-based	CV [30]	ST	X	X	X
CA [30]	ST	X	X	X
KNN [9]	MT	X	X	X
S-KNN [9]	MT	✓	X	X
Data-driven	LSTM	ST	X	X	X
S-LSTM [2]	ST	✓	X	X
HiVT [40]	MT	✓	✓	✓
LaneGCN [22]	MT	✓	✓	✓
HOME [14]	OM	✓	✓	✓
DSP [38]	MT	✓	✓	✓
*Abbreviations: ST: Single-Trajectory, MT: Multi-Trajectory, OM: Occupancy Map.
4.1 Motion Prediction Methods
We select 10 representative prediction models to achieve comprehensive coverage of mainstream approaches, ranging from simple model-based methods to complex data-driven approaches, as presented in Table 1. Constant Velocity (CV) and Constant Acceleration (CA) [30] assume that the predicted agent maintains a constant speed or acceleration within the prediction horizon. K-Nearest Neighbor (KNN) predicts an agent's future trajectory based on most similar trajectories, while Social-KNN (S-KNN) [9] extends it by also considering the similarity of surrounding agents. These methods are widely used as baselines given their widespread effectiveness in simple prediction cases. Social LSTM (S-LSTM) [2], HiVT [40], LaneGCN [22], and HOME [14] represent four distinct types of neural networks: RNN, Transformer, GNN, and CNN. DSP [38] utilizes a hybrid design of neural networks, representing state-of-the-art prediction models.
4.2 Planners
An ideal planner should: 1) be able to handle state and action uncertainty; 2) take into account multiple factors of driving performance, such as safety (collision avoidance), efficiency (timely goal achievement), and comfort (smooth driving); 3) be aware of interactions with other agents; and 4) supports real-time execution. We select two realistic planners based on these criteria: a simplistic planner that only meets 2) and 4), and a sophisticated planner that satisfies all of  $I$  - 4). This allows us to draw planner-agnostic conclusions.
RVO. The RVO planner [34] is a simplistic planner that solves the optimization problem in the velocity space under collision avoidance constraints. The planner uses motion predictions to avoid possible collisions with the deterministic motions, and thus does not consider the state and action uncertainty. As the RVO planner does not maintain states between consecutive timesteps, it also cannot optimize its planning with respect to interactions with other agents. The objective function of the RVO planner involves safety and efficiency within a short time window, and the RVO planner executes in real time.
DESPOT. The DESPOT planner [32] is a state-of-the-art belief-space planning algorithm that address uncertainties near-optimally. To account for stochastic states and actions, we adopt the bicycle model, a kinematic model with two degrees of freedom, and introduce Gaussian noise to the displacement. DESPOT considers the system state, context information, and the ego-agent's action to predict the future states of exo-agents. This allows it to consider the interaction between agents. Moreover, the objective function of DESPOT incorporates safety, efficiency, and comfort metrics, making it an ideal algorithm for planning in complex and dynamic environments.
We use these planners to control the speed of the ego-agent while pure-pursuit algorithm [11] for adjusting the steering angle. Details can be found in the supplementary materials.
4.3 Simulator
In order to evaluate different prediction models, the ideal simulator should: 1) provide real-world maps and agents; 2) model potential unregulated behaviors; 3) accurately mirror the interactions between agents; and 4) provide realistic perception data for effective planning. We choose the SUMMIT simulator [6] for our experiments since it meets all the criteria, as demonstrated in Table 2. SUMMIT is a sophisticated simulator based on the Carla [12] framework, offering various real-world
Table 2: Comparison between simulators.
Simulator	Real-World Maps	Unregulated Behaviors	Dense Interactions	Realistic Visuals
SUMO [23]	✓	✗	✓	✗
TrafficSim [33]	✗	✓	✓	✗
TORCS [7]	✗	✓	✗	✓
BARK [4]	✗	✓	✗	✗
Apollo [1]	✗	✗	✗	✗
Symphony [20]	✓	✓	✗	✓
SUMMIT [6]	✓	✓	✓	✓
maps and agents to create diverse and challenging scenarios. It uses a realistic motion model to simulate interactions between agents and supports the simulation of crowded scenes, complex traffic conditions, and unregulated behaviors.
There are two distinct concepts of time in our experiments: simulation time and real time. The former corresponds to the duration of actions in the simulator, while the latter represents the wall time consumed by the planner. By default, the simulator runs in asynchronous mode. In this mode, the simulator and planner run individually, and the simulation time is equal to the real time. However, in this study, we employ the SUMMIT simulator in synchronous mode to accommodate slow predictors. In this mode, the simulator waits for the planner to establish communication before proceeding to the next step. Since the simulation time for each execution step remains constant at  $0.03\mathrm{s}$ , the ratio between simulation time and real time can be manually set by varying the communication frequency, known as the tick rate. For example, once the tick rate is set to  $30\mathrm{Hz}$ , the simulation time equals the real time. But, when the tick rate is set to  $3\mathrm{Hz}$ , the ratio between simulation time and real time is 0.1.
4.4 Evaluation Protocols
Motion Prediction Performance Metrics. Four commonly used prediction performance metrics are employed in this study, as presented in Table 3. In our experiments, the consensus of  $\mathrm{K} = 6$  is adopted. While ADE/FDE can be applied to evaluate single-trajectory prediction models, their probabilistic variants minADE and minFDE can be applied to evaluate multi-trajectory predictors.
Driving Performance Metrics. The driving performance is primarily determined by three factors: safety, comfort, and efficiency. Let  $H$  represent the total timestep for each scenario.
Safety is typically evaluated in terms of the collision rate. The collision is determined whether the smallest distance between the ego-agent and surrounding agents is smaller than a threshold  $\epsilon$ , that is:  $P_{\text{Safety}} = \frac{1}{H} \sum_{t=1}^{H} \mathbb{I}[\min \{|s_A^t - s| : s \in \{s_1^t, \dots, s_n^t\} \} < \epsilon]$ , where  $||\cdot||$  is the L2 distance between the ego-agent's bounding box and the exo-agent's bounding box,  $\mathbb{I}$  is the boolean indicator function. We set  $\epsilon = 1\mathrm{m}$  for our experiments since the DESPOT planner rarely causes real collisions.
Efficiency is evaluated by the average speed of the ego-agent over the whole scenario:  $P_{\text{Efficiency}} = \frac{1}{H} \sum_{t=1}^{H} v_A^t$ . Comfort is represented by the jerk of the ego-agent, which is the rate of change of acceleration with respect to time:  $P_{\text{Comfort}} = \frac{1}{H} \sum_{t=1}^{H} \ddot{v}_A^t$ .
Since these three metrics are not comparable, we normalized each metric to  $[0,1]$  before calculating the driving performance. Furthermore, we standardized the direction of these metrics, with higher values indicating better performance:
$$
\bar {P} _ {m} = \left[ \frac {P _ {m} - \min \left(P _ {m}\right)}{\max \left(P _ {m}\right) - \min \left(P _ {m}\right)} \right] ^ {n} + \left[ 1 - \frac {P _ {m} - \min \left(P _ {m}\right)}{\max \left(P _ {m}\right) - \min \left(P _ {m}\right)} \right] ^ {1 - n}. \tag {4}
$$
The boolean indicator  $n = \mathbb{I}[m \in \{\text{Efficiency}\}]$  is employed to adjust the direction of driving performance metrics. The ultimate driving performance is derived through the average of normalized metrics for safety, efficiency, and comfort. Pseudocode can be found in the supplementary materials.
Experimental Design. We conduct two types of experiments for both planners in the SUMMIT simulator: Fixed Prediction Ability and Fixed Planning Ability.
1. Fixed Prediction Ability: The planner is required to perform a fixed number of predictions within an interactive simulation environment, regardless of the predictor's execution speed. The objective is to clarify the factor that dominates the disparity between prediction performance and driving performance while ensuring the predictive ability of methods.
Table 3: Trajectory prediction metrics.
Metric Name	Metric Equation
ADE	1/T ∑i=1T√(xi-x̂i)2+(yi-ŷi)2
FDE	√(xT-ŷT)2+(yT-ŷT)2
minADE	mink∈K 1/T ∑i=1T√(xi-ŷi)2+(yi-ŷi)2
minFDE	mink∈K √(xT-ŷT)2+(yT-ŷT)2