Parting with Misconceptions about Learning-based Vehicle Motion Planning
Authors: Daniel Dauner
Marcel Hallgarten
Andreas Geiger
Kashyap Chitta
Venue: CoRL 2023 Poster (2023)
Keywords: Motion Planning
Autonomous Driving
Data-driven Simulation

Abstract: The release of nuPlan marks a new era in vehicle motion planning research, offering the first large-scale real-world dataset and evaluation schemes requiring both precise short-term planning and long-horizon ego-forecasting. Existing systems struggle to simultaneously meet both requirements. Indeed, we find that these tasks are fundamentally misaligned and should be addressed independently. We further assess the current state of closed-loop planning in the field, revealing the limitations of learning-based methods in complex real-world scenarios and the value of simple rule-based priors such as centerline selection through lane graph search algorithms. More surprisingly, for the open-loop sub-task, we observe that the best results are achieved when using only this centerline as scene context (i.e., ignoring all information regarding the map and other agents). Combining these insights, we propose an extremely simple and efficient planner which outperforms an extensive set of competitors, winning the nuPlan planning challenge 2023.

Parting with Misconceptions about Learning-based Vehicle Motion Planning
Daniel Dauner Marcel Hallgarten
Andreas Geiger Kashyap Chitta 1.2
University Tübingen Tübingen Al Center Robert Bosch GmbH
httpa:// com///uttooooomuuiiiiiaooaaaaaap
gagaggg garage
The release of nuPlan marks a new era in vehicle motion planning research, offering the first large-scale real-world dataset and evaluation schemes requiring both precise short-term planning and Long-horizon ego-forecasting. Exist- ing systems struggle simultaneously meet both requirements. Indeed, we find that these tasks are fundamentally misaligned and should be addressed independently. We further assess the current state of closed-loop planning in the field, revealing the limitations of learning-based methods in complex real-world scenarios and the value of simple rule- base- priors such as centerline selection through lane graph search algorithms. More surprisingly, for the open-loop sub-task, we observe that the best results are achieved when using only this centerline as scene context (i.e., ignoring all information regarding the map and other agents). Combining these insights, propose extremely simple and efficient planner which outperforms extensive set of competitors, winning the nuPlan planning challenge 2023.
Abstract:
Keywords: Motion Planning, Autonomous Driving, Data-driven Simulation
Introduction
Despite learning-bast -baedd systems' success in vehicle motion planning research [1, 2, 4, 5], lack of standardized large-scale datasets for benchmarking holds back their transfer from research to applications [6, 7, 8]. The recent release of the nuPlan dataset and simulator [9], a collection of 1300 hours of real-world vehicle motion data, has changed this, enabling the development of new generation of learned motion planners, which promise reduced manual design effort and improved scalability. Equipped with this new benchmark, we perform the first rigorous empirical analysis on a large-scale, open-source, and data- bata-driven simulator for vehicle motion planning, including a comprehensive set of state-of-the-art (SoTA) planners [10, 11, 12] using the official metrics. Our analysis yields several surprising findings:
Open- and closed-loop evaluation are misaligned. Most learned planners are trained through the supervised learning task of forecasting the ego vehicle's future motion conditioned on desired goal location. We refer to this setting as ego-forecasting [2, 3, 13, 14]. In nuPlan, planners can be evaluated in two ways: (1) in open-loop evaluation, which measures ego-forecasting accuracy using listance-based metrics or (2) in closed-loop evaluation, which assesses the actual driving performance in simulation with metrics such progress collision rates. Open-loop evaluation lacks dynamic feedback and can have little correlation with closed- loop driving, as previously shown on the simplistic CARLA simulator [15, 16). Our primary contribution lies in uncovering negative correlation between both evaluation schemes. Learned planners excel at ego-forecasting but struggle to make safe closed-loop plans, whereas rule-based planners exhibit the opposite trend.
Rule-based planning generalizes. We surprisingly find that an established rule-based planning baseline from over twenty years ago [17] surpasses all SoTA learning based methods in terms of closed-loop evaluation metrics on our benchmark. This contradicts the prevalent motivating claim used in most research on learned planners that rule-based planning faces difficulties in generalization.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

This was previously only verified on simpler benchmarks [4, 10, 11]. As result, most current work on learned planning only compares to other learned methods, ignoring rule-based baselines [3, 5, 18].
centerline is all you need for ego-forecasting. We implement naive learned planning baseline which does not incorporate any input about other agents in the scene and merely extrapolates the ego state given centerline representation of the desired route. This bascline sets the new SoTA for open-loop evaluation on our benchmark. It does not require intricate scene representations (e.g. lane graphs, vectorized maps, rasterized maps, tokenized objects), which have been the central subject of inquiry in previous work [10, 11, 12]. None of these prior studies considered a simple centerline-only representation as a baseline, perhaps due to its extraordinary simplicity.
Our contributions are follows: (1) We demonstrate and analyze the misalignment between open- and closed-loop evaluation schemes in planning. (2) We propose a lightweight extension of IDM [17] with real-time capability that achieves state-of-the-art closed-loop performance. (3) We conduct experiments with an open-loop planner, which is only conditioned on the current dynamic state and a centerline, showing that it outperforms sophisticated models with complex input representations. (4) By combining both models into a hybrid planner, we establish a simple baseline that outperformed 24 other, often tearning-based competing approaches and claimed victory the nuPlan challenge 2023.
2 Related Work
Rule-based planning. Rule-based planners offer structured, interpretable decision-making framework [17, 19, 20, 21, 22, 23, 24, 25, 26). They employ explicit rules to determine an autonomous vehicle's behavior (c.g., brake when an object is straight ahead). A seminal approach in rulc- blsed planning the Intelligent Driver Model (IDM [17]), which is designed to follow a leading vehicle in traffic while maintaining a safe distance. There exist extensions of IDM [27] which focus on enabling lane changes on highways. However, this is not the goal of our work. Instead, we extend IDM by executing multiple policies with different hyperparameters, and scoring them to select the best option.
Prior work also combines rule- based decision makinn with learned components, e.g., with learned agent forecasts [28], affordance indicators [23, 24), cost-based imitation learning [4, 29, 30, 31, 32], or learning- based plamning with rule-based safety filtering [33]. These hybrid planners often forecast future environmental states, enabling informed and contingent driving decisions. This forecasting can either be agent-centric [34, 35, 36], where trajectories determined for each actor, environment- centric [4, 31, 30, 29, 37, 38), involving occupancy or cost maps. Additionally, forecasting can be conditioned on the ego-plan, modeling the ego vehicle's influence on the scene's 's future [39, 40, 41, 42]. We employ an agent-centric forecasting module that is considerably simpler than existing methods, allowing for its use as a starting point in the newly released nuPlan framework.
for--oore....ng. Unlike predictive planning, ago-forecatting methods use observational data to directly determine the future trajectory. Ego-forecasting approaches include both end-to-end methods [43] that utilize LiDAR scans [44, 45], RGB images [46, 47, 48, 49, 14, 50] both [13, 51, 5, 52], as well as modular methods involving lower-dimensional inputs like bird's eye view (BEV) grids or state vectors [24, 53, 11, 54, 55, 56]. A concurrent study introduces a naive MLP inputting the current dynamic state, yielding competitive ego-forecasting results on the nuScenes dataset [57] with no scene context input [58]. Our findings complement these results, differing by evaluating long-term (8s) ago-forecasting in the challenging 2023 nuPlan challenge scenario test distribution [9].
We show that in this setting, completely removing scene context (as in [58]) is harmful, whereas simple centerline representation of the context is sufficient for strong open-loop performance.
3 Ego-forecasting and Planning are Misaligned
In this section, we provide the relevant background regarding the data-driven simulator nuPlan [9]. We describe two baselines for preliminary experiment to demonstrate that although ego-forecasting and planning are often considered related tasks, they not well-aligned given their definitions nuPlan. Improvements in one task can often lead to degradation in the other.

---

= 08
= 0.5s
38
IDM
PDM PDM-Oper Open
Figure 1: Planning vs. ego-forecasting. We present nuPlan scenario, highlighting the driveable area in grey and the original human trajectory as dashed black line. In each snapshot, we display the ego agent with its prediction. (Left) observe the significant displacement between the IDM prediction (constrained to a rule-based centerline) and the human trajectory. resulting low open-loop scores. (Mid right) after 0.5 seconds of simulation, the Iearned PDM-Open planner extrapolates its own errors and eventually veers off-road, leading to suboptimal closed-loop scores.
3.1 Background
nuPlan. The nuPlan simulator is the first publicly available real-world planning benchmark and enables rapid prototyping and testing motion planners. nuPlan constructs a simulated environment as closely as possible
real-world driving setting through data-driven simulation [59, 60, 61, 62, 63, 64, 65]. This method extracts road maps, traffic patterns, and object properties (positions, orientations, and speeds) from pre- pre-recorded dataset consisting of 1,300 hours of real-worke driving. These elements then used initialize scenarios, which 15-second simulations employed to assess open-loop and closed-loop driving performance. Hence, in simulation, our methods rely on access to detailed HD map information and ground-truth perception. i.e., no localization errors, map imperfections, or misdetections are considered. In open-loop simulation, the entire log is merely replayed (for both the ego vehicle and other actors). Conversely, closed-loop simulation, the ego vehicle operates under the control of the planner being tested. There are two versions closed-loop simulation: non-reactive, where all other actors are replayed along their original trajectory, and reactive, where other vehicles employ IDM planner [17]. which detail in the following.
Metries. nuPlan offers three official evaluation metrics: open-loop score (OLS), closed-loop score non-reactive (CLS-NR), and closed-loop score reactive (CLS-R). Although CLS-NR and CLS-R are computed identically, they differ in background traffic behavior. Each score a weighted average of sub-scores that are multiplied and heading errors, both average and final, prediction error is above threshold, the penalty results in OLS Similarly, sub-scores in CLS comprise time-to-collisoon progress along the experts' route, speed- limit compliance, and comfort. Multiplicative CLS penalties are at-fault collisions, driveable area or driving direction infringements and not making progress. These penalties result substantial CLS reductions, mostly to a zero scenario score, e.g., when colliding with vehicle. Notably, the CL.S primarily relies on short-term actions rather than on consistent long-term planning. All scores (inel. OLS/CLS) lie range from 0 to 100, where higher scores are better. Given the elaborate composition of nuPlan's 's metrics, refer to the supplementary material for detailed description.
set of penalties. OLS, the sub-scores account for displacement
extended period (8 seconds). Moreover, if the
of zero for that scenario.
Intelligent Driver Model. The simple planning baseline IDM [17] not only simulates the non-ego vehicles the CLSS- evaluation of nuPlan, but also serves as a baseline for the ego-vehicle's planning. The nuPlan map provided as graph, with centerline segments functioning as nodes. After choosing of such nodes to follow via graph search algorithm, IDM infers longitudinal trajectory along the selected centerline. Given the current longitudinal position velocity and distance to the leading vehicle along the centerline, it iteratively applies the following policy to

---

calculate longitudinal acceleration:
de
df a (1) (=))
)
The acceleration limit a, target speed Uo, safety margin s", and exponent 6 are manually selected. Intuitively, the policy uses an acceleration a unless the velocity is already close to vo or the leading vehicle is distance of only s" Additional details and our exact syper-parameter choices can be found in the supplementary material.
3.2 Misalignment
Centerline-onniitione ego-forecasting. We now propose the Predictive Driver Model (Open), ie., PDM-Open, which is straightforward multi-layer perceptron (MLP) designed to predict future waypoints. The inputs this MLP are the centerline (c) extracted by IDM and the ego history (h). To accommodate the high speeds (reaching up to 15 m/s) and ego-forecasting horizons (extending to seconds) observed in nuPlan, the centerline is sampled with resolution of 120 meters. Meanwhile, the ego history incorporates the positions, velocities, and accelerations of the vehicle over the previous two seconds, sampled at rate of 5Hz. Both and h are linearly projected to feature vectors of size 512, concatenated, and input to the MLP Oopen which has two 512-dimensional hidden layers. The output are the future waypoints for an 8-second horizon, spaced 0.5 seconds apart, expressed as Wüpen = Odpen (c, h). The model is trained using an L1 loss on our training dataset of 177k samples (described Section 4). By design, PDM-Open considerably simpler than existing learned planners [10, 12].
meter up to length
OLS vs. CLS. In Table 1, we benchmark the IDM and PDM-Open baselines using the nuPlan metrics. We present two IDM variants with different maximum acceleration val- ues (the default four PDM-Open variants based on different inputs. We ob- serve that reducing IDM's acceleration improves OLS but negatively impacts CLS. While IDM demonstrates strong closed-loop performance, PDM-Open outperforms IDM in open-loop even if it only uses the current ego state as in- put (first row). The past ego states (History) only yield little improvement and lead to a drop in CLS. Most im- portantly, adding the centerline significantly contributes ego-forecasting performance. A clear trade-off between CLS and OLS indicates a misalignment between the goals of ego-forecasting and planning. This sort of inverse correlation nuPlan is unanticipated, considering the increasing of ego-forecasting in current planning literature [3, 10, 12, 11]. While forecaacan is not necessary for driving performance, the nuPlan challenge requires both high OLS and CLSS
Centerline
CLS-NR 1
CLS-R +
History
OLS
1.0ms 2 and a lmmm) and
Method
IDM
[17] 1.0
77
76
38
a 0.1
54
48
51
38
34
72
PDM-Open
54
53
85
54
50
86
Table I: OLS-CLS Tradcoff. Baseline scores nuPlan with different inputs.
In Fig. 1, we illustrate the misalignment between the OLS and CLS metrics. In the depicted scenario, the rule-based IDM selects different lane in comparison the human driver. However, it maintains its position on the road throughout the simulation. This results in high CLS yet low OLS. Conversely, the learned PDM-Open generates predictions along the lane chosen by the human driver, thereby obtaining high OLS. Nonetheless, as errors accumulate in its short-term predictions during the simulation [66, 67], the model's 's trajectory veers off the drivable area, culminating in a subpar CLS.
3.3 Methods
We now extend IDM by incorporating several concepts from model predictive control, including forecasting, proposals, simulation, scoring, and selection, as illustrated in Fig. 2 (top). We call this model PDM-Closed. Note that first step, still require graph search to find sequence of lanes along the route and extract their centerline, in the IDM planner.

---

Select Centerline
Agent Forccast
IDM Proposals
Select Trajectory
Simulation
Scoring
D
X
PDM C1 osed
Input Scene:
Route Plan
Trajectory
Prediction
Output:
corraat
Centerline
Ego History
Position
Velocity
Acceleration
PDM-Offset:
PDM-Hybrid
Figure Architecture. PDM-Closed selects a centerline, forecasts the environment, and creates vary- ing trajectory proposals, which simulated and scored for trajectory selection. The JDM-Hybrid module predicts offsets using the PDM-Closed centerline, trajectory, and ego history, correcting only long-term waypoints and thereby limiting the learned model's influence in closed-loop simulation.
Forecasting, In nuPlan, the simulator provides an orientation vector and speed for each dynamic agent such as a vehicle or pedestrian. We leverage a simple yet effective constant velocity forecast- ing [68, 69, 70] over the horizon of seconds 10Hz.
Proposals. In the process of calibrating the IDM planner, observed trade-off when selecting sin- gle value for the target speed hyperparameter (vo), which either yielded aggressive driving behavior or insufficient progress across various scenarios. Consequently, we generate a set of trajectory proposals by implementing IDM policies at five distinct target speeds, namely, {20% 40%, 60%, 80%, 100%) of the designated speed limit. For each target speed, we also incorporate proposals with three lateral centerline offsets (+ Im and 0m), thereby producing N 15 proposals total. To circumvent computational demands in subsequent stages, the proposals have reduced horizon of H steps, which corresponds to 4 seconds at 10Hz.
Simulation. Trajectories in nuPlan are simulated by iteratively retrieving actions from an LQR controller [71] and propagating the ego vehicle with a kinematic bicycle model [72, 73]. We simulate the proposals with the same parameters and faster re-implementation of this two-stage pipeline. Thereby, the proposals are evaluated based on the expected movement closed-loop.
Scoring, Each simulated proposal is scored to favor traffic-rule compliance, progress, and comfort. By considering proposals with lateral and longitudinal variety, the planner can avoid collisions with agent forecasts and correct drift that may arise when the controller fails to accurately track the intended trajectory. Furthermore, our scoring function closely resembles the nuPlan evaluation metries. We direct the reader to the supplementary material for additional details.
Trajectory selection. Finally, coM-Cllsed selects the highest-scoring proposal which is extended to the complete forecasting horizon F with the corresponding IDM policy. If the best trajectory is expected to collide within 2 seconds, the output overwritten with an emergency brake maneuver.
Enhancing long-horizon accuracy. To integrate the accurate ego-forecasting capabilities of now propose hybrid version
PDM-Open with the precise short-term actions of PDM-Closed, of PDM, i.e., PDM-Hybrid. Specifically, PDM-Hybrid uses learned module PDM-Offset to predict offsets to waypoints from PDM- Cloed, as shown in Fig. 2 (bottom).
5

---

Method
Rep.
CLS-R 1
CLS-NR 1
OLS
Time
Urban Driver [10]
Polygon
50
53
64
GC-PGP [12]
Graph
55
59
83
100
PlanCNN [11]
Raster
72
73
43
IDM [17]
Centerline
77
76
38
27
PDM-Open
Centerline
54
50
PDM-Closed
Centerline
92
93
42
91
PDM-Hybrid
Centerline
92
93
84
96
PDM-Hybrid
Graph
92
93
84
172
Log Replay
GT
80
94
100
Table 2:
Val14 benchmark. We show the closed-loop score reactive/non-reactin (CLS-R/CLS-NR) open loop score (OLS) and runtime in ms for several planners. We specify the input representation (Rep.) used by each planner. PDM-Hybrid accomplishes strong ego-forecasting (OLS) and planning (CLS). *This is preliminary version of PDM-Hybrid that combined PDM-Closed with GC-PGP [12], and was used our online leaderboard submission (Table 3).
In practice, the LQR controller used nuPlan relies exclusively on the first seconds of the trajectory when determining actions in closed-loop. Therefore, applying the correction only to long-tern waypoints (i.e., beyond 2 seconds by default, which we refer to as the correction horizon C) allows PDM-Hybrid maintain closed-loop planning performance. The final planner outputs waypoints (up to the forecasting horizon F) [ Wyybrial that are given by:
w Bybrid Weloeed 1/c>-cooorreet (Wllosed: C, h).
Where and h are the centerline and history (identical to the inputs of PDM-Open). {WeLosed) ) the PDM-Closed waypoints added to the hybrid approach, and @offeet is an MLP. Its architecture is identical to Oppen except for an extra linear projection to accommodate Welozed as an additional input.
important to note that PDM-Hybrid designed with high modularity, enabling the substitution of individual components with alternative options when diverse requirements emerge. For example, show results with a different open-loop module in the supplementary material. Given its overall simplicity, one interesting approach to explore involves incorporating modular yer differentiable algorithms as components, seen in [34]. Exploring the integration of these modules within unified multi-task architectures is another interesting direction. We reserve such exploration for future work.
4 Experiments
We now outline our proposed benchmark and highlight the driving performance of our approach.
Val14 benchmark. We offer standardized data splits for training and evaluation. Training
all 70 scenario types from nuPlan, restricted to maximum of 4k scenarios per type, resulting in ~ 177k training scenarios. For evaluation, we use 100 scenarios of the 14 scenario types considered by the leaderboard, totaling 1,118 scenarios. Despite minor imbalance (all 14 types do not have 100 available scenarios), our validation split aligns with the online leaderboard evaluation (Table 2 and Table 3), confirming the suitability of our Val14 benchmark as a proxy for the online test set.
Baselines. We include several additional SoTA approaches adopting goo-ccrcag for planning in our study. Urban Driver [10] encodes polygons with PointNet layers and predicts trajectories with a linear layer after multi-head attention block. Our study uses an implementation of Urban Driver trained in the open-loop setting. GC-PCP [12] clusters trajectory proposals based on route-constrained lane- ame--grahh traversals before returning the most likely cluster center. Pl PlanCNN [11] predicts waypoints using a CNN from rasterized grid features without an ego state input. It shares several similarities Chauf feurNet [8], seminal work in the field. A preliminary version of PDM-Hybrid, which won the nuPlan competition, used GC-PGP its fgo-forcca component, and include this as baseline. We provide complete description of this version in the supplementary.

---

Results. Our results are presented in Table 2. PlanCNN achieves the best CLS among learned planners, possibly due to its design choice of removing ego state from input, trading OLS for enhanced CLS. Contrary to the community's growing preference for graph- and vector-based scene representations in prediction and planning (74, 11, 75, 76), these results show no clear disadvantage of raster representations for the closed-loop task, with PlanCNN also offering lower runtime. Surprisingly, the simplest rule-based approach in our study, IDM, outperforms the best learned observe PDM-Closed's ss advantages over IDM in terms of CLS: an
planner, PlanCNN. Moreover, improvement from 76-77 to 92-93 result of the ideas from Section 3. Surprisingly, PDM-Open achieves the highest OLS of 86 with a runtime of only 7ms using only a centerline and the ego state as input. We observe that PDM-Open improves on other methods in accurate long-horizon nollwoolo,,ng, as detailed further our supplementary material. Next, despite PDM-Closed's unsatisfactory 42 OLS, PDM-Hybrid successfully combines PDM-Closed with PDM-Open. Both the centerline and graph versions of PDM-Hybrid achieve identical scores in our evaluation. However, the final centerline version, using PDM-Open instead of GC-PGP, is more efficient during inference. Finally, the privileged approach of outputring the ground-truth ego future trajectory (log replay) fails to achieve a perfect CLS, in part due to the nuPlan framework's 's LQR controller occasionally drifting from the provided trajectory. PDM-Hybrid d compensates for this by evaluating proposals based on the expected controller outcome, causing it to match/outperform log replay in closed-low evaluation.
Challenge. The 2023 nuPlan challenge saw the prelim- inary (graph) version of PDM-Hybrid rank first out of 25 participating teams. The leaderboard considers the mean of CLS-R, CLS-NR, and OLS. While open-loop perfor- mance lagged slightly, closed-loop performance excelled, resulting in an overall SoTA score. Unfortunately, due the closure of the leaderboard, our final (centerline) ver- sion of PDM-Hybrid that replaces GC-PGP with the simpler PDM-Open module could not be benchmarked. All top con- tenders combined learned ego-forecasting with mule-based post-solvers or post-processing to boost CLS performance for the challenge [77, 78, 79]. Thus, we expect to see more hybrid approaches in the future.
CLS-R t
CLS-NK
Sccone T
OLS 4
Method
PDM- Hyb-Htbrid*
93
93
83
90
hoplan
89
85
pegasus_multi_pee
82
85
88
85
Urban Driver [10] IDM [17]
68
86
75
72
75
29
59
Table 3: 2023 nuPlan Challenge.
Importantly, near identical scores were recorded for our submission both Val14 benchmark (Table 2) and the official leaderboard (Table 3). Note that the Urban Driver and IDM results on the leaderboard are provided by the nuPlan team, so they likely use different training data and hyper-parameters than our implementations from Table 2.
Ablation Study. We delve into our design choices through an ablation study in Table 4. Table 4a displays PDM-Hybrid's closed-loop score reactive (CLS-R) and open-loop score (OLS) with varied correction horizons (C) from Os to 3s. Applying the waypoint correction all waypoints (i.e., C' == 0), outperforms PDM-Open in OLS (87 vs. 86, see Table 2) but leads to substantial drop in CLS-R compared to the default value of C = 2. On the other hand. a noticeable OLS decline occurs when initiating corrections deeper into the trajectory (e.g., C = 3), with minimal impact on CLS-R.
For PDM-Closed (Table 4b), we compare CLS-R and runtime (ms) with the base planner across three scenarios: removing lateral centerline offsets ("lat."), longitudinal IDM proposals ("lon."), and environment forecasting ("cast."), "). Our analysis reveals that eliminating proposals diminishes CLS-R R effectiveness but accelerates runtimes. Performance significantly drops when excluding the forecasting used for creating and evaluating proposals. However, the runtime remains nearly identical, showing the effectiveness of the simple forecasting mechanism.
As for PDM-Open (Table 4c), we test three variations: shorter centerline (30m vs. 120m), coarser centerline (every 10m vs. Im), and smaller MLP with reduced hidden dimension (from 512 to 256). Both a smaller MLP and reduced centerline length lead to performance degradation, but the impact remains relatively minor compared to disregarding the centerline altogether (Table 1, OLS=72). Meanwhile, the impact of a coarser centerline is negligible.
7

---

CLS-R f OLS 1
Method CL.S-R 1
Time
Method
OL.
0.0s
87
Base
Baseline
86
2.0s
92
84
No lat.
89
55
Shorter centerline
84
2.5s
92
84
No lon.
88
64
Coarser centerline
86
3.0s
72
cast.
Smaller MI.P
84
(a) PDM-Hybrid
(b) PDM-Closed
(c) PDM-Open
Table 4: Ablation Study. We show the closed-loop score reactive (CLS-R), open loop score (OLS) and runtime ms. We investigate (a) varying correction horizons for PDM-Hybrid. (b) ignoring sub-modules of PDM-Closed, and (c) the effects of input and architecture choices PDM-Open. The default configuration, highlighted in gray, achieves the best trade-offs.
Discussion
Although rule-based planning is often criticized for limited generalization, our results demonstrate strong performance in the closed-loop nuPlan task which best resembles real-world evaluation. Notably, open-loop success in part requires trade-off closed-loop performance. Consequently, imitation-trainet rgo-forecasting methods fare poorly in closed-loop. This suggests that rule-based planners remain promising and warrant further exploration. At the same time, given their poor performance out-of-the-box, there is room for improvement in imitation-based methods on nuPlan.
Integrating the strengths closed-loop planning and open-loop ego-forecasting. we present hybrid model. However, this does not enhance closed-looy driving performance; instead, it boosts open-loop performance while executing identical driving maneuvers. We conclude that considering precise open-loop ego-forecasting
prerequisite for achieving long-term planning goals is misleading.
Acknowledging the potential importance of ego-forecasting for interpretability and assessing human- like behavior, we propose focusing this evaluation on the short horizon (e.g., 2 seconds) relevant for closed-loop driving. The current nuPlan OLS definition, requiring a unimodal 8- 8-second ego-forecast may only be useful for alternate applications, like setting goals for background agents in data- driven traffic simulations allocating computational resources better, c.g. to prioritize perception or prediction in areas the ego-vehicle is expected to traverse. We discourage the use of open-loop metrics as a primary indicator of planning performance [18].
Limitations. While we significantly improve upon the established IDM model, PDM still does not execute lane-change maneuvers. Lane change attempts often lead to collisions when the ego-vehicle is between lanes, resulting in high penalty as per the nuPlan metrics. PDM relies HD maps and precise offboard perception [80, 9] that may be unavailable in real-world driving situations. While real-world deployment was demonstrated for learning-based methods [81, 33, 8], it remains significant challenge for rule-based approaches. Moreover, our experiments, aside from the held-out test set, have not specifically evaluated the model's generalization capabilities when encountering distributional shifts, such as unseen towns or novel scenario types. They were all conducted on important to recognize the limitations inherent in nuPlan's
single simulator, nuPlan. Therefore, data-driven simulation approach. When planner advances more rapidly than the human driving log, objects materialize abruptly in front of the ego-vehicle during simulation. For CLS-NR, vehicles move independently as observed in reality, disregarding the ego agent, leading to excessively aggressive behavior. Conversely, CLS-R R background agents rely on IDM and adhere strictly to the centerline, leading to unrealistically passive behavior. We see high value in developing a more refined reactive environment for future work.
Conclusion. In this paper, identify prevalent misconceptions in learning- based vehicle motion planning. Based on our insights, introduce PDM Hybbdd which builds upon IDM and combines it with learned goo-foreaatting component. It surpassed comprehensive set of competitors and claimed victory in the 2023 nuPlan competition.

---

Acknowledgements
Andreas Geiger was supported by the ERC Starting Grant LEGO-3D (850533), the BMWi in the project KI Delta Learning (project number 19A 190130) and the DFG EXC number 2064/1 project number 390727645. Kashyap Chitta was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen Center, FKZ: DIIS180394 We thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Kashyap Chitta. We also thank Pat Karnchancchari and Gianmarco Bernasconi for helping with technical issues in our leaderboard submissions, Niklas Hanselmann for proofreading, and Andreas Zell for helpful discussions.
References
[1] F. Codevilla, M. Miller, A. López, V. Koltun, and A. Dosovitskiy. End-to-end driving via conditional imitation learning. In Proc. IEEE International Conf. on Robotics and Automation (ICRA), 2018.
[2] F. Codevilla, E. Santana, A. M. López, and A. Gaidon. Exploring the limitations of behavior cloning for autonomous driving. Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2019.
[3] N. Rhinehart, R. McAllister, K. Kitani, and S. Levine. PRECOG: prediction conditioned on goals visual multi-agent settings. Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2019.
[4] W. Zeng, W. Luo, S. Suo, A. Sadat, B. Yang, S. Casas, and Urtasun. End-to-end interpretable neural motion planner. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.
[5] K. Chitta, A. Prakash, B. Jaeger, Yu, K. Renz, and A. Geiger. Transfuser: Imitation with transformer-basee sensor fusion for autonomous driving. IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMT), 2022.
[6] M. Bojarski, D. Testa, Dworakowski, Firner, B. Flepp, Goyal, L. D. Jackel, M. Mon- fort, U. Muller, Zhang, Zhang, J. Zhao, and K. Zieba. End end learning for self-driving cars. arXiv.org, 1604.07316, 2016.
[7] A. Kendall, Hawke, D. Janz, Mazur, Reda, M. Allen, V. Lam, Bewley, and A. Shah. Learning drive in day. arXivorrg, hts//180.00412,, 2018.
[8] M. Bansal, A. Krizhevsky, and A. Ogale. Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst. In Proc. Robotics: Science and Systems (RSS), 2019.
[9] H. Caesar, Kabzan, K. Tan, W. K. Fong, Wolff, Lang, L. Fletcher, O. Beijbom, and S. Omari. nuplan: closed-loop ml-based planning benchmark for autonomous vehicles. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR) Workshops, 2021.
[10] O. Scheel, L. Bergamini, M. Wolezyk, B. Osiáski, and P. Ondruska. Urban driver: Learning to drive from real-world demonstrations using policy gradients. In Proc. Conf. on Robot Learning (CoRL), 2021.
[11] K. Renz, K. Chitta, O.-B. Mercea, A. S. Koepke, Z. Akata, and A. Geiger. Plant: Explainable planning transformers via object-level representations. In Proc. Conf. Robot Learning (CoRL), 2022.
[12] M. Hallgarten, M. Stoll, and A. Zell. From Prediction to Planning With Goal Conditioned Lane Graph Traversals. arXiv.org. 2302.07753, 2023.

---

[13] A. Prakash, K. Chitta, and A. Geiger. Multi-modal fusion transformer for end-to-end au- tonomous driving. Proc. JEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.
[14) K. Chitta, A. Prakash, and A. Geiger. Neat: Neural attention fields for end-to-end autonomous driving, In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2021.
[15] F. Codevilla, A. M. Lopez, V. Koltun, and A. Dosovitskiy. On offline evaluation of vision-based driving models. In Proc. of the European Conf. on Computer Vision (ECCV), 2018.
[16] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun. CARLA: An open urban driving simulator. In Proc. Conf. on Robot Learning (CoRL), 2017.
[17] M. Treiber, A. Hennecke, and D. Helbing. Congested traffic states in empirical observations and microscopic simulations. Physical review E, 2000.
[18] Y. Hu, J. Yang, Chen, Li, C. Sima, X. Zhu, S. Chai, S. Du, T. Lin, W. Wang, L. Lu, X. Jia, Q. Liu, Dai, Y. Qiao, and H. Li. Planning-oriented autonomous driving. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2023.
[19] Thrun, M. Montemerlo, Dahlkamp, D. Stavens, A. Aron, J. Diebel, P. Fong, Gale, M. Halpenny, G. Hoffmaan,, K. Lau, C. M. Oakley, M. Palatucci, V. R. Pratt, P. Stang. S. Stro- hband, C. Dupont, L. Jendrossek, C. Koelen, C. Markey, C. Rummel, J. van Niekerk, E. Jensen, P. Alessandrini, G. R. Bradski,
B. Davies, S. Ettinger, A. Kaehler, A. V. Nefian, and P. Mahoney. Stanley: The robot that won the DARPA grand challenge. Journal of Field Robotics (JFR), 23 (9):6616692, 2006.
[20] A. Bacha, C. Bauman, R. Faruque, M. Fleming, C. Terwelp, C. Reinholtz, D. Hong, A. Wicks, T. Alberi, D. Anderson, et al. Odin: Team victortango's ' entry in the darpa urban challenge. Journal of Field Robotics (JFR), 25(8), 2008.
[21] J. Leonard, J. P. How, S. J. Teller, M. Berger, S. Campbell, G. A. Fiore, L. Fletcher, E. Frazzoli,
A. S. Huang, S. Karaman, O. Koch, Y. Kuwata, D. Moore, E. Olson, S. Peters, J. Teo, R. Truax, M. R. Walter, D. Barrett, A. Epstein, K. Maheloni, K. Moyer, T. Jones, R. Buckley, M. E. Antone, R. Galejs, S. Krishnamurthy, and J. Williams. A perception-driver autonomous urban vehicle. Journal of Field Robotics (JFR), 25(10))727-774, 2008.
[22] C. Urmson, J. Anhalt, D. Bagnell, C. Baker, R. Bittner, M. Clark, J. Dolan, D. Duggins, Galatali, C. Geyer, et al. Autonomous driving in urban environments: Boss and the urban challenge. Journal of Field Robotics (JFR), 25(8), 2008
[23] C. Chen, A. Seff, A. L. Kornhauser, and J. Xiao. Deepdriving: Learning affordance for direct perception in autonomous driving. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), pages 2722-27222 2730, 2015.
[24] A. Sauer, N. Savinov, and A. Geiger. Conditional affordance learning for driving in urban environments. In Proc. Conf. on Robot Learning (CoRL), 2018
[25] H. Fan, F. Zhu, C. Liu, L. Zhang, L. ZZuuang, D. Li, W. Zhu, J. Hu, H. Li, and Q. Kong. Baidu apollo EM motion planner. arXiv.org, 1807.08048, 2018.
[26] A. Sadat, M. Ren, A. Pokrovsky, Y. Lin, E. Yumer, and R. Urtasun. Jointly learnable behavior and trajectory planning for self- delf-driving vehicles. In Proc. IEEE International Conf. on Intelligent Robots and Systems (IROS), 2019.
[27] A. Kesting, M. Treiber, and D. Helbing, General lane-changing model mobil for car-following models. Transportation Research Record, 1999(1), 2007.
[28] R. Chekroun, T. Gilles, M. Toromanoff, S. Hornauer, and F. Moutarde. Mbappe: Mcts-built- around prediction for planning explicitly. arXiv.org, 2309.08452, 2023.
10

---

[29] A. Cui, A. Sadat, S. Casas, R. Liao, and R. Urtasun. Lookout: Diverse multi-future prediction and planning for self-driving. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2021.
[30] S. Casas, Sadat, and R. Urtasun. Mp3: unified model map, perceive, predict and plan. In Proc. IEEE Conf. on Computer Vision and Puttern Recognition (CVPR), 2021.
[31] A. Sadat, S. Casas, M. Ren, X. Wu, P. Dhawan, and R. Urtasun. Perceive, predict, and plan: Safe motion planning through interpretable semantic representations. In Proc. of the European Conf. on Computer Vision (ECCV), 2020.
[32] Z. Huang, Karkus, B. Ivanovic, Y. Chen, M. Pavone, and C. Lv. Dtpp: Differentiable joint conditional prediction and cost evaluation for tree policy planning in autonomous driving. arXiv.org, 2310.05885, 2023.
[33] T. Phan-Minh, F. Howington, T.-S. Chu, M. S. Tomov, R. E. Beaudoin, S. U. Lee, N. Li, C. Dicle, S. Findler, F. Suarez-Ruiz, et al. Driveirl: Drive in real life with inverse reinforcement learning. In Proc. IEEE International Conf. on Robotics and Automation (ICRA), 2023.
[34] Karkus, B. Ivanovic, S. Mannor, and M. Pavone. Diffstack: differentiable and modular control stack for autonomous vehicles. Proc. Conf. on Robot Learning (CoRL), 2022.
[35] M. H. Danesh, Cai, and D. Hsu. Leader: Learning attention over driving behaviors for planning under uncertainty. In Proc. Conf. Robot Learning (CoRL), 2022.
[36] Z. Huang, H. Liu, J. Wu, and C. Lv. Differentiable integrated motion prediction and planning with learnable cost function for autonomous driving. arXiv.org, 2207. 2047.1042, 2022.
[37] B. Wei, M. Ren, W. Zeng, M. Liang, B. Yang, and R. Urtasun. Perceive, attend, and drive: Learning spatial attention for safe self-driving. Proc. IEEE International Conf. on Roboties and Automation (ICRA), 2021.
[38] S. Hu, L. Chen, P. Wu, H. Li, J. Yan, and D. Tao. St-p3: End-to-end vision-based autonomous driving via Vision (ECCV), 2022.
feature learning. In Proc. of the European Conf. Computer
[39] H. Song, W. Ding, Y. Chen, S. Shen, M. Y. Wang, and Q Chen. Pip: Planning-informet trajectory prediction for autonomous driving. In Proc. of the European Conf. on Computer Vision (ECCV), 2020.
[40] N. Rhinchart, J. He, C. Packer, M. A. Wright, R. McAllister, J. E. Gonzalez, and S. Levine. Contingencies from observations: Tractable contingency planning with learned behavior models. In Proc. IEEE International Conf. on Robotics and Automation (ICRA), 2021.
[41] Y. Chen, P. Karkus, B. Ivanovic, X. Weng, and M. Pavone. Tree-structured policy planning with learned behavior models. Proc. IEEE International Conf. on Robotics and Automation (ICRA), 2023.
[42] Z. Huang, H. Liu, and C. Lv. Gameformer: Game-theoretic modeling and learning of transformer-basee interactive prediction and planning for autonomous driving. arXiv.org, 2303.05760, 2023.
[43] L. Chen, P. Wu, K. Chitta, B. Jaeger, A. Geiger, and H. Li. End-to-end autonomous driving: Challenges and frontiers. arXiv.rrg, 2306.16927, 2023.
[44] N. Rhinehart, R. McAllister, and S. Levine. Deep imitative models for flexible inference, planning, and control. In Proc. of the International Conf. on Learning Representations (ICLR), 2020.
11

---

[45] A. Filos, P. Tigas. R. McAllister, N. Rhinehart, S. Levine, and Y. Gal. Can autonomous vehicles identify, recover from, and adapt to distribution shifts? In Proc. of the International Conf. on Machine learning (ICML), 2020.
[46] H. Xu, Y. Gao, F. Yu, and T. Darrell. End-to-end learning of driving models from large-scale video datasets. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 3530-3538, 2017.
[47] D. Chen, Zhou, Koltun, and Krähenbühl. Learning by cheating. In Proc. Conf. on Robot Learning (CoRL), 2019.
[48] E. Ohn-Bar, A. Prakash, A. Behl, K. Chitta, and A. Geiger. Learning situational driving. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2020.
[49] A. Behl, K. Chitta, A. Prakash, E. Ohn-Bar, and A. Geiger. Label efficient visual abstractions for autonomous driving. In Proc. IEEE International Conf. Intelligent Robots and Systems (IROS), 2020.
[50] P. Wu, X. Jia, L. Chen, J. Yan, H. Li, and Y. Qiao. Trajectory-guidee control prediction for end- to-end autonomous driving: A simple yet strong baseline. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[51] D. Chen and P. Krähenbühl. Learning from all vehicles. In CVPR, 2022.
[52] B. Jaeger, K. Chitta, and A. Geiger. Hidden biases of end-to-end driving models. Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2023.
[53] N. Hanselmann, K. Renz, K. Chitta, A. Bhattacharyya, and A. Geiger. King: Generating safety-critical driving scenarios for robust imitation via kinematics gradients. In Proc. of the European Conf. on Computer Vision (ECCV), 2022.
[54] M. Vitelli, Y. Chang, Y. Ye, A. Ferreira, M. Wolczyk, B. Osiński, M. Niendorf, H. Grimmett, Q. Huang, A. Jain, et al. Safetynet: Safe planning for real-world self-driving vehicles using machine-learned policies. In Proc. IEEE International Conf. on Robotics and Automation (ICRA), 2022.
[55] S. Pini, C. S. Perone, A. Ahuja, A. S. R. Ferreira, M. Niendorf, and S. Zagoruyko. Safe real-world autonomous driving by learning to predict and plan with mixture of experts. In Proc. IEEE International Conf. on Robotics and Automation (ICRA), 2023.
[56] J. Cheng, Y. Chen, X. Mei, B. Yang, B. Li, and M. Liu. Rethinking imitation-based planner for autonomous driving. arXiv.org, 10443, 2023.
[57] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuscenes: A multimodal dataset for autonomous driving. Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020.
[58] J.-T. Zhai, Z. Feng. J. Du, Y. Mao, J.-J. Liu, Z. Tan, Zhang, X. Ye, and J. Wang. Rethinking the open-looy evaluation of end-to-end autonomous driving in nuscenes. arXiv.org, 2303.10430. 2023.
[59] M. Althoff, M. Koschi, and Manzinger. Commonroad: Composable benchmarks for motion planning on roads. In Proc. IEEE Intelligent Vehicles Symposium (IV), 2017.
[60] L. Bergamini, Y. Ye, O. Scheel, L. Chen, C. Hu, Pero, Osinski, H. Grimmett, and
P. Ondruska. Simnet: Learning reactive self-driving simulations from real-world observations.
In Proc. IEEE International Conf. Robotics and Automation (ICRA), 2021.
12

---

[61] S. Suo, S. Regalado, S. Casas, and R. Urtasun. Trafficsim: Learning to simulate realistic multi-agent behaviors. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.
[62] 7. Zhong, D. Rempe, D. Xu, Y. Chen, S. Veer, T. Che, B. Ray, and M. Pavone. Guided conditional diffusion for controllable traffic simulation. In Proc. IEEE International Conf. on Robotics and Automation (ICRA), 2023.
[63] D. Xu, Y. Chen, B. Ivanovic, and M. Pavone. Bits: Bi-level imitation for traffic simulation. In Proc. IEEE International Conf. on Robotics and Automation (ICRA), 2023.
[64] Z. Zhang, A. Liniger, D. Dai, F. Yu, and L. Van Gool. TrafficBots: Towards world models for autonomous driving simulation and motion prediction. In Proc. IEEE International Conf. Robotics and Automation (ICRA), 2023.
[65] L.. Feng, Q. Li, Peng, S. Tan, and Zhou. Trafficgen: Learning to generate diverse and realistic traffic scenarios. In Proc. IEEE International Conf. on Robotics and Automation (ICRA), 2023.
[66] Ross, G. J. Gordon, and D. Bagnell. reduction of imitation learning and structured prediction no-regret online learning. In Conference on Artificial Intelligence and Statistics (AISTA7S), 2011.
[67] A. Prakash, A. Behl, E. Ohn-Bar, K. Chitta, and A. Geiger. Exploring data aggregation in policy learning for vision-based urban autonomous driving. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020.
[68] S. Poddar, C. Mavrogiannis, and S. S. Srinivasa. From crowd motion prediction to robot navigation in crowds. arXiv preprint arXiv:2303.01424, 2023.
[69] C. Schöller, V. Aravantinos, F. Lay, and A. Knoll. What the constant velocity model can teach us about pedestrian motion prediction. IEEE Robotics and Automation Letters, 5(6):169-11703. 2020.
[70] H. Wu, T. Phong. C. Yu, P. Cai, S. Zheng, and D. Hsu. What truly matters in trajectory prediction for autonomous driving? arXiv preprint 1rXiv:2300.11136, 2023.
[71] Y. Tassa, N. Mansard, and E. Todorov. Control-limited differential dynamic programming. In Proc. IEEE International Conf. on Robotics and Automation (ICRA), 2014.
[72] R. Rajamani. Vehicle dynamics and control. Springer Science Business Media, 2011.
[73] P. Polack, F. Altché, B. d' Andréa Novel, and A. de La Fortelle. The kinematic bicycle model: A consistent model for planning feasible trajectories for autonomous vehicles? In Proc. JEEE Intelligent Vehicles Symposium (IV), 2017.
[74] N. Deo, E. M. Wolff, and O. Beijbom. Multimodal Trajectory Prediction Conditioned on Lane-Graph Traversals. In Proc. Conf. on Robot Learning (CoRL), 2021.
[75] N. Nayakanti, R. AI-Rfou, A. Zhou, K. Goel, K. S. Refaat, and B. Sapp. Wayformer: Motion forecasting via simple and efficient attention networks. In Proc. IEEE International Conf. on Robotics and Automation (ICRA), 2023.
[76] A. Cui, S. Casas, K. Wong, S. Suo, and R. Urtasun. Gorela: Go relative for viewpoint-invariant motion forecasting, In Proc. IEEE International Conf. Robotics and Automation (ICRA), 2023.
[77] Y. Hu, K. Li, P. Liang, J. Qian, Z. Yang, H. Zhang, W. Shao, Z. Ding, W. Xu, and Q. Liu. Imitation with spatial-temporaa heatmap: 2nd place solution for nuplan challenge. arXiv.org. 2306.15700, 2023.
13

---

[78] W. Xi, L. Shi, and G. Cao. An imitation learning method with data augmentation and post processing for planning autonomous driving. URL https//oppnndiielaa.commeeaaal AD23Challengee/Trcc 4 4_pegasus_ pegasus_weitao.poe
[79] Z. Huang, H. Liu, X. Mo, and C. Lyu. Gameformer planner: A learning-enabled interactive prediction and planning framework for autonomous vehicles. URL https///ppendiivlla..cmmm e2ead/ A
[80] C. R. Qi, Zhou, M. Najibi, P. Sun, K. Vo, B. Deng, and D. Anguelov. Offboard 3d object detection from point cloud sequences. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2021.
[81] O. Scheel, L. Bergamini, M. Wolezyk, B. Osinski, and P. Ondruska. Urban driver: Learning to drive from real-world demonstrations using policy gradients, In Proc. Conf. on Robot Learning (CoRL), 2021.
14


Reviews:
[{"review_id": "ZAQ1VwHunE", "reviewer": "Reviewer_qggz", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This research paper investigates the alignment between open-loop trajectory forecasting and closed-loop planning metrics on the nuPlan benchmark in the context of autonomous driving, with learning-based planners excelling at the former but a simple 20-year-old rule-based planner performing best for the latter. Notably, a centerline-only model surpasses methods employing elaborate scene representations, indicating limited long-horizon context value in open-loop forecasting. The paper proposes PDM-Hybrid, which combines rule-based and learned components, yielding state-of-the-art performance on both metrics with enhanced simplicity and efficiency. Extensive experiments demonstrate the effectiveness of this approach.  The paper makes key recommendations like focusing forecasting evaluation on short control-relevant horizons and avoiding reliance on open-loop metrics for planning assessment. Code, data and models are released to facilitate future autonomous driving research.", "summary_of_recommendation": "Overall, I'm positive about this work, but there are some limitations in terms of novelty, rigor, and technical advancements. The core revelation of misaligned metrics and learning method issues is an excellent contribution. However, the proposed PDM-Hybrid model only offers minor advances compared to established techniques. The experiments are fairly rigorous but lack extensive ablation studies and testing in diverse simulation environments to demonstrate generalizability. While the actionable recommendations for the field are impactful, the paper doesn't propose substantial new techniques to achieve these goals. Additionally, the lack of physical robot validation raises concerns about real-world applicability.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "+ Please clearly highlight the novelty of the proposed approach, what is the contributions, and what is being distinguish between incremental and novel aspects. Please also provide concluding remarks on this paper.\n+ Explicitly state the assumptions made in the problem formulation.\n+ The proposed PDM-Hybrid model seems incremental - how is it substantially different from prior techniques? What innovations does it introduce?\n+ Please comment on the computations cost of the proposed approach and on what computational platform was this approach implemented.\n+ How do you ensure safety guarantees for the learned components of your model?\n+ Does your approach account for imperfect state estimation or map errors that would occur in practice?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is well written and easy to follow. The paper's core contributions around revealing evaluation issues and limitations of learning methods are quite original. However, some ideas like using rule-based planners as a baseline have been explored before. The proposed PDM-Hybrid model is simple and high-performing but not an entirely new technique. The experimental results are comprehensive but significance is somewhat limited by focusing only on one simulator (nuPlan) and competition-style metrics. The findings may not generalize fully to other environments and tasks. \n\nThe revealing analysis and actionable recommendations do provide valuable new insights that stand to advance autonomous driving research. However, the proposed techniques themselves represent only an incremental advance, as evidenced by the simple model achieving state-of-the-art results. Substantial innovation of new planning and learning techniques is still needed for broader progress.\n\nStrengths:\n+ The analysis in the paper reveals fundamental misalignments between open-loop ego-forecasting and closed-loop planning assumptions. It also shows limitations of learning methods compared to a simple rule-based approach.\n+ The introduced PDM-Hybrid model achieves state-of-the-art results on nuPlan, winning the 2023 challenge. Its modular design is also flexible.\n+ The paper makes specific recommendations for future research, such as focusing ego-forecasting evaluation on short 2 second horizons relevant to control, rather than unrealistic long 8 second predictions.\n\nWeakness:\n+ The paper lacks clarity regarding its contributions, making it difficult to discern the originality of the work and distinguish between incremental and novel aspects. The absence of a conclusion section further compounds this issue. It is recommended that the authors include a dedicated paragraph outlining their contributions and incorporate a conclusion section to address these concerns.\n+ The modular PDM-Hybrid architecture exhibits limitations, particularly in the fact that its components are not learned end-to-end, potentially hindering the full utilization of crosstalk between different submodules.\n+ The absence of information regarding the computational platform used for the simulations is a notable omission. Moreover, the reliance on access to HD maps and perfect perception raises questions about the current real-world applicability of the proposed method. Addressing imperfect state estimation and maps is crucial. Therefore, the authors should include discussions on how their proposed method could be extended to real-world scenarios. Additionally, conducting further evaluations in diverse and interactive simulators would enhance the assessment of generalizability.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "U4nqusSgbq", "reviewer": "Reviewer_xfB2", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents an empirical analysis of the performance of vehicle motion planning approaches in the nuPlan simulator. \nIn doing so, the paper focuses on the difference in performance achieved between evaluating in a \"closed-loop scheme\" and an \"open-loop scheme\". While \"open-loop\" evaluation can be done on a static dataset by comparing generated actions of an algorithm with ground-truth labels, closed-loop evaluation requires an actual simulator where progress and collision rates are measured.  \n\nThe paper shows that there is a negative correlation between the two evaluation schemes when tested in the nuPlan simulator. Furthermore, the paper states that a simple rule-based planning baseline surpasses all SOTA learning-based methods. ", "summary_of_recommendation": "My recommendation is mainly based on the fact that this paper tackles the task of motion planning in a setting where perception is perfect. Tackling the motion planning task without considering the challenges of perception seems disconnected from the state of the art.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "How would the findings presented in this paper generalize to a more realistic driving scenario where perception is noisy and incomplete?\n\nHow does the reported metric relate to crash rates? How does this work compare to the state of the art in autonomous driving? Seeing that the best-recorded score in this paper is 93% (given perfect perception), I'm not sure if this correctly reflects SOTA. There are already companies that perform autonomous driving on real vehicles in real-world environments that obviously reach success rates substantially higher than 99%.  \n\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths\n========\n- The paper is clearly written and reads well. \n- The related work provides a nice overview of rule-based planning and approaches performing ego-forecasting.\n- The figures are well made and illustrate the task and its complexity well.\n- One of the approaches presented in the paper won the nuPlan 2023 challenge.\n\n\n\nConstructive Criticism\n=================\n- I am unfamiliar with the nuPlan challenge and do not know the level of complexity it poses. From the visualizations in the paper and the supplementary video, the tested environments look very simplistic, it appears that algorithms have perfect knowledge of the environment. Does this challenge contain a perception component? \n- While the study won the inaugural nuPlan challenge, it would be useful to know how the proposed method compares to other methods in more diverse and complex real-world scenarios. A comparison to additional datasets besides nuPlan could enhance the generalizability and robustness of the findings.\n- The discussion section reads as if there are only two classes of planning approaches for autonomous vehicles: rule-based methods and approaches based on imitation learning. I'm no expert in the field, but I assume there are approaches that combine interpretable (i.e. close to rule-based) decision-making approaches with data-driven environment models. It would be interesting if the paper could extend on this.\n- The paper refers to citation 59 for information about the evaluation metric's composition. Given that the metric is of such central nature to the findings of this paper, I suggest to include a description of it in the main text.", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "HDCPTWMR-L", "reviewer": "Reviewer_7G8b", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper describes the approach taken by the 2023 nuPlan winning team. It tracks two metrics: ego-forecasting and closed-loop performance. \n\nThe work puts forward three general approaches. The first one (PDM-Open) takes as input the centerline predicted by a rule-based planner (IDM), as well as the history of ego, and outputs the predicted ego trajectory. The second one (PDM-Closed) generates a collection of candidate trajectories by using a fixed set of hyperparameters and IDM. Candidate trajectories are then simulated, scored, and the best one is chosen for execution. The final (winning) approach (PDM-Hybrid) uses PDM-Closed to generate a plan and adds it as an input to a model similar to PDM-Open.\n\nPDM-Open excels at ego-forecasting but underperforms at closed-loop evaluation, PDM-Closed is the opposite, and PDM-Hybrid preserves the performance of PDM-Closed while recovering the performance of PDM-Open on ego-forecasting.", "summary_of_recommendation": "Overall I think the experiments are thorough and the results thought provoking. The main strengths of the paper are the simplicity of its approaches and its very strong empirical results. I do not think that the weaknesses in some of the analysis should detract from the paper’s overall strong contributions.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "For the results provided in Table 1, what is the input to PDM-Open for the version that has neither a centerline or history? Is it only the current state of ego?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I think this paper offers interesting insight into the problem of autonomous driving. Mostly it suggests to me that rule-based planners combined with simple learning methods might still have a lot to offer. That said I do have the following concerns about it.\n\nFirst, a key design decision in PDM-Hybrid is that the planner only modifies trajectories suggested by PDM-Closed beyond the horizon of the simulator’s closed-loop controller (LQR). This is what allowed the team to bring their score on ego-forecasting back up without impacting their closed-loop performance at all. I think this is clever, but I hesitate to cast this as any sort of insight into the problem of autonomous driving beyond pointing to the fact that ego-forecasting is probably a bad metric to use for future competitions. That said I think the authors acknowledge this towards the end of the paper. They might simply want to emphasize it earlier as well instead of introducing both metrics as equals involving a tradeoff.\n\nMy second concern with the paper is that it provides limited insight into what leads to good performance in the ego-forecasting task. With PDM-Open, the limited input (ego history and centerline) dramatically reduces the ability of the planner to learn any sort of policy (in contrast to what planners like PlanCNN - a competitive benchmark - are forced to do by removing ego from their inputs). It is therefore not surprising to me that this leads to accumulating error and poor performance in the closed-loop setting. What *is* surprising to me is that this leads to better performance in the ego-forecasting setting, and I think that the paper doesn’t really provide any insight as to why that might be the case.\n\nOverall the paper is well written and the experiment satisfactory. I also think that the methods proposed are simple enough that they are likely to be reproduced by the community.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "1-2ygssPtUA", "reviewer": "Reviewer_KEQU", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents an empirical study on the nuPlan dataset and benchmark. Specifically, the authors compare a series of planners ---including a proposed one that won the nuPlan challenge--- with respect to their open-loop and closed-loop performance. Through extensive ablation studies, the paper discusses key takeaways and directions for future work. One interesting insight is the observation that rule-based baselines exhibit strong performance in closed-loop settings in the nuPlan challenge, suggesting that they should not be prematurely rejected as non-generalizable approaches for some driving tasks.", "summary_of_recommendation": "Overall, this is a well written paper with compelling findings and well presented insights. With the incorporation of a few points of discussion (see above), I am in favor for acceptance.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "-Discuss the connection of the insights extracted in the study to the real-world and the possibility of future hardware deployment.\n\n-Discuss scaling to more complex scenarios and behaviors (e.g., intersections, aggressive drivers etc).", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths\n\n-The paper is well-written and easy to follow with nice explanations of the technical components and a helpful discussion of results.\n\n-The empirical study is quite extensive, containing ablation studies and in-depth discussions of the performance of the approaches evaluated.\n\n-The findings are quite intriguing, demonstrating the value of rule-based approaches for some types of driving tasks. Through its rigorous evaluation, the paper follows a few recent works demonstrating the value of well-understood, model-based approaches. See Scholler et al. What the constant velocity model can teach us about pedestrian motion prediction. RA-L 2020; Poddar et al. From Crowd Motion Prediction to Robot Navigation in Crowds. arXiv:2303.01424. 2023. I've felt like there is a point to be made about this, and the authors execute this quite well.\n\nWeaknesses\n\n-Section 3 could be organized a little better. I feel like the introductory headers in bold are helpful but a better organization through grouped subsections could help readability.\n\n-The scenarios highlighted are quite interesting but they feature relatively structured motion. Are there any insights for more interactive scenarios like (unsignalized) intersections? How would the PDM-based approach perform in such settings?\n\n-A discussion of how the insights extracted would likely port over (or not) to the real world would be interesting to add.\n\n***POST REBUTTAL***\n\nThe rebuttal addressed my comments and the additional supplementary material strengthened the paper. I remain in favor of acceptance.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper conducts an empirical analysis of vehicle motion planning approaches in the nuPlan simulator. It specifically examines the difference in performance between using a "closed-loop scheme" and an "open-loop scheme". While the reviewers have generally positive about this work, they did identify some writing-related issues that should be addressed in the final version.
Decision: Accept (Poster)