Future-Aware End-to-End Driving: Bidirectional Modeling of Trajectory Planning and Scene Evolution
Authors: Bozhou Zhang
Nan Song
jingyu li
Xiatian Zhu
Jiankang Deng
Li Zhang
Venue: NeurIPS 2025 poster ()
Keywords: autonomous driving
end-to-end
world model
Primary area: applications

Abstract: End-to-end autonomous driving methods aim to directly map raw sensor inputs to future driving actions such as planned trajectories, bypassing traditional modular pipelines. While these approaches have shown promise, they often operate under a one-shot paradigm that relies heavily on the current scene context, potentially underestimating the importance of scene dynamics and their temporal evolution. This limitation restricts the model’s ability to make informed and adaptive decisions in complex driving scenarios. We propose a new perspective: the future trajectory of an autonomous vehicle is closely intertwined with the evolving dynamics of its environment, and conversely, the vehicle’s own future states can influence how the surrounding scene unfolds. Motivated by this bidirectional relationship, we introduce **SeerDrive**, a novel end-to-end framework that jointly models future scene evolution and trajectory planning in a closed-loop manner. Our method first predicts future bird’s-eye view (BEV) representations to anticipate the dynamics of the surrounding scene, then leverages this foresight to generate future-context-aware trajectories. Two key components enable this: (1) future-aware planning, which injects predicted BEV features into the trajectory planner, and (2) iterative scene modeling and vehicle planning, which refines both future scene prediction and trajectory generation through collaborative optimization. Extensive experiments on the NAVSIM and nuScenes benchmarks show that SeerDrive significantly outperforms existing state-of-the-art methods.

Future-Aware End-to-End Driving: Bidirectional Modeling of Trajectory Planning and Scene Evolution
Bozhou Zhang 1.2, Nan Song 1.2, Jingyu Li Xiatian Zhu 3*, Jiankang Deng*, Li Zhang!
Schooo of Data Science, Fudan University
Shanghai Innovation Institute
3 University Surrey *Imperial College London
https:://githu..comm
Abstract
End-to-end autonomous driving methods aim to directly map sensor inputs future driving actions such as planned trajectories, bypassing traditional modular pipelines. While these approaches have shown promise, they often operate under a one-shot paradigm that relies heavily on the current scene context, potentially underestimating the importance of scene dynamics and their temporal evolution. This limitation restricts the model's ability to make informed and adaptive decisions in complex driving scenarios. We propose new perspective: the future trajectory of an autonomous vehicle is closely intertwined with the evolving dynamics of its environment, and conversely, the vehicle's own future states can influence how the surrounding scene unfolds. Motivated by this bidirectional relationship, we introduce SeerDrive, a novel end-to-end framework that jointly models future scene evolution and trajectory planning in a closed-loop manner. Our method first predicts future bird's s-eye view (BEV) representations anticipate the dynamics of the surrounding scene, then leverages this foresight to generate citure-context-wware trajectories. Two key components enable this: (1) future-aware planning, which injects predicted BEV features into the trajectory planner, and (2) iterative scene modeling and vehicle planning, which refines both future scene prediction and trajectory generation through collaborative optimization. Extensive experiments on the NAVSIM and nuScenes benchmarks show that SeerDrive significantly outperforms existing state-of-the-art methods.
1 Introduction
End-to-end autonomous driving [1] has emerged as a promising paradigm by jointly learning percep- tion [2, 4], prediction [5, 6, 7, 8J, and planning [9, 10, 11] in unified framework. Compared to traditional modular pipelines, this approach simplifies system design and enables planning-oriented optimization through holistic training. Recent advances [12, 13, 14, 15, 16] have demonstrated that directly generating future trajectories from raw sensor inputs can achieve strong performance, highlighting the potential of end-to-end methods for building scalable and efficient driving systems. These methods have been widely evaluated in both open-loop [17] and closed-loop [18] settings, across sallworldd datasets [19] and simulation platforms [20].
Despite these successes, most existing methods adopt a one-shot paradigm, in which sensor observa- tions, typically from the current time step, used directly predict trajectory several seconds into the future. In this setup, the model must rely heavily on the scene's current situation to infer the ego
*Li Zhang (lizhangfd @ fudan.edu.cn) and Xiatian Zhu Exiatian.zhir zhu @ surrey.ac.uk) are the corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Iterative
Sensor input
Sensor input
Future
scene feature
End-to-end planning network
End-to-end planning network
Future world modeling network
Current ego feature
Planned trajectories
Planned trajectories
Predicted future scene
(a) Previous methods
(b) Our SeerDrive
Figure 1: Paradigm comparison. (a) Prior end-to-end methods follow a one-shot paradigm, directly mapping sensor imputs planned trajectories based only the current scene. (b) In contrast, Seer- Drive predicts scene evolution with a world model and plans trajectories with planning model, enabling iterative interaction between the two in a closed-loop process.
vehicle's future motion (Figure (a)).. While effective in structured environments, this approach tends to underestimate the importance of how the scene may evolve over time, crucial factor in dynamic and interactive driving contexts. Furthermore, the ego vehicle's 's own future actions can significantly influence how the surrounding scene unfolds. These two aspects, the future scene and the agent's 's future behavior, are inherently coupled and should be modeled together. However, this bidirectional dependency remains underexplored in current end-to-end systems.
To address this gap, we draw inspiration from the emerging concept of world models, which offer the ability to learn environment dynamics and simulate future observations. We propose to leverage this capability not only to foresee how the driving scene will change, but also coordinate it with the ego vehicle's planned actions through mutual interaction.
In this paper, present SeerDrive, novel end-to-end driving framework that introduces paradigm shift by explicitly modeling the bidirectional relationship between scene evolution and trajectory planning (Figure 1(b)). To implement this paradigm, design two key components. Future- Aware Planning injects predicted future bird's '-eyeee-eeeee view (BEV) features into the planner, enabling trajectory generation informed by both current perception and anticipated dynamics. Iterative Scene Modeling and Vehicle Planning refines both the predicted scene and the planned trajectory through mutual feedback, supporting adaptive and temporally consistent decision-making. Together, these components enable context-aware planning in complex, dynamic environments.
paradigm for end-to-end driving that explicitly captures the bidirectional interaction between scene dynamics and the ego vehicle's future actions, challenging the conventional one-shot planning approach. (II) We instantiate this paradigm through unified framework, SeerDrive, which jointly models future BEV scene representations and vehicle trajectories through future-aware and iterative interaction mechanisms. (III) We conduct extensive experiments on both the NAVSIM and nuScenes benchmarks, demonstrating that SeerDrive achieves state-of-the-arr performance and validates the effectiveness of our proposed design.
Our contributions threefold. (I) We propose
2
Related work
End-to-end autonomous driving. End-to-end autonomous driving [12, 13, 14, 15] has attracted increasing attention, contrasts with traditional modular methods by integrating perception [2, 4], prediction [5, 7], and planning [9, 11] into a unified, differentiable framework that enables end-to-end optimization. Early methods [21, 22, 23, 24] often bypass intermediate tasks and directly infer planning trajectories or actions from sensor data, both in open-loop [17] and closed-loop [20] settings. UniAD [12) unifies perception, prediction, and planning into a differentiable framework and leverages transformer architecture to optimize the entire pipeline in planning oriented manner, achieving
coanine--orietedd

---

strong performance across all tasks. VAD [13] adopts vectorized representation to improve the efficiency of the end-to-end pipeline. VADv2 [25] introduces probabilistic planning paradigm with a large vocabulary and demonstrates impressive performance in closed-loop settings. SparseDrive [14] leverages a sparse scene representation to make better use of temporal information. Several other studies [26, 27] explore self-supervised learning to simplify the complex end-to-end pipeline.
Recent efforts have focused increasingly on more challenging end-to-end planning benchmarks [18, 19J. DiivusionDDrvve [15] proposes a truncated diffusion policy to achieve more accurate and diverse planning. GoalFlow [28] incorporates flow matching and goal-point guidance into end-to-ent autonomous driving.
[16] investigates the scaling law within a unified transformer- based architecture. Different from previous methods that focus solely on improving the planning process itself, our approach aims to jointly optimize future world modeling and planning in an adaptive manner to achieve better planning performance.
World model in autonomous driving, World models aim to predict future scene dynamics condi- tioned on the current environment and ego state. Some studies [29, 30, 31] address world modeling by training models to generate realistic videos that are consistent with physical principles. Drive- Dreamer [32] employs latent diffusion model guided by 3D boxes, HD maps, and ego states, and introduces an additional decoder for future action prediction. Drive- WM [33] generalizes this setup to multi-camera inputs and explores its application in the end-to-end planning task. Some other methods explore improving the generalization of dynamic world modeling by scaling up both data and model architecture. GAIA-I [34] models token sequences using an autoregressive transformer conditioned on past states, followed by a diffusion- -based decoder for realistic video synthesis. Vista [35] is trained on diverse web-collected driving videos and utilizes latent diffusion to produce high-resolution, long-horizon video outputs.
In contrast to methods that focus generating complex and realistic images, some approaches generate driving scenes in the bird's sird view. SLEDGE [36] introduces the first generative simulator designed for agent motion planning. GUMP [37] builds upon generative modeling capture dynamic traffic interactions, enabling diverse and realistic future scenario simulations. UMGen (38] generates ego-centric, multimodal driving scenes in an autoregressive, frame-by-frane manner. Scenario Dreamer [39] employs a vectorized latent diffusion model that directly operates on structured scene elements, coupled with an autoregressive Transformer for simulating agent behaviors in data-driven way. Following the simplicity and structured nature of BEV, we likewise adopt it for modeling future scene evolution.
Joint world modeling and planning. Several works [27, 40, 41, 42, 43) explore joint modeling and collaborative optimization of world models and planning models, employing various strategies. In the field of autonomous driving, some works extend the world model by introducing an action token. OccWorld [44] employs an occupancy-based representation and jointly generates future occupancy and ego actions in an autoregressive manner. Drive-OccWorlle [45] further extends this pipeline to operate directly from images, enabling 4D occupancy forecasting and planning. In contrast, some works start from the end-to-end autonomous driving pipeline and aim to enhance planning through world modeling. Existing methods either leverage world modeling as an additional supervision signal during training [26, 27], or use it to assist in selecting the best trajectory among multiple candidates [43]. Our approach conducts in-depth exploration of world modeling and end-to-end planning design, enabling deep interaction between the two processes and significantly improving planning performance.
3 Methodology
3.1 Preliminary
Task formulation. End-to-end autonomous driving maps sensor inputs (e.g., camera and LiDAR) to future ego trajectories, often using multi-modal outputs to capture diverse possible futures. Auxiliary tasks like detection, map segmentation, and agent motion prediction are commonly integrated to enhance scene understanding and support safer planning. World models autonomous driving aim to predict future scene evolution based on current observations, enabling agents to simulate and evaluate future outcomes. They offer structured representations of the environment, which help improve planning accuracy and decision reliability.
3

---

Update
Ego status
(b) End-to-end
Mulli-view images
planning network
Current ego feat.
Planned trajectories
Current ego feat.
BEV world
modeling network
Current BEV feat.
Future BEV feat.
Future BEV map
LiDAR
Iterative Scene Modeling and Vehicle Planning
(a) SeerDrive pipeline
Motion senonn-wwatt wawe Layer Normalization
Transformer
decocer
Current BEV feat.
Transformer decoder
Future BEV Reat.
Current feat.
Current ego feat.
Anchored endpoints
Future ego feat.
Future-aware End-to-end Planning Network
Figure 2: Overview of SeerDrive. (a) Multi-view images and LiDAR inputs are encoded to obtain the current BEV feature, while ego status is encoded to produce the current ego feature. These are then used to predict the future BEV feature. All three types of features are subsequently used for trajectory planning. (b) The end-to-end planning network generates future trajectories based on the current ego feature, current BEV feature, and future BEV feature. (c) The BEV world modeling network and the end-to-end planning network operate iteratively, updating the current ego feature to progressively improve planning performance.
Framework overview. As illustrated in Figure 2, the SeerDrive framework consists of two key components, (a) shows the overall pipeline, where multi-view images and LiDAR inputs are fused to obtain the current BEV feature, and the ego status is encoded as the current ego feature. These features are fed into the BEV world modeling and end-to-end planning networks, which operate iteratively to predict the future BEV map and generate planned trajectories. (b) details the end-to-end planning network, which incorporates the future BEV feature ceature-producet by the world modeling network-to enhance trajectory planning. (c) depicts the iterative interaction between world modeling and planning, enabling gradual refinement and improved planning performance.
3.2 Feature encoding
As illustrated in Figure (a), multiple types features are encoded. Given multi-view images I. and LiDAR features P. the encoder transforms these multi-modal sensor inputs into current BEV feature map Fourr E R xxxxxxx where H and W are the spatial dimensions the BEV feature, and C' is the number of feature channels. Following prior works [15, 43, 46], we adopt TransFuser [21] to obtain a unified BEV representation. Subsequently, a lightweight BEV decoder is employed to generate the current BEV semantic map Baur for supervision. Following prior works [14, 15, 43], the anchored multi-modal trajectories and ego status & are encoded with a simple multi-layer perceptron (MLP) encoder to produce the multi-modal ego feature Feure RM where M denotes ego
the number of trajectory modes. The process is shown below:
Four TransFuser (I, P),
Feur EgoEncoder (T, agoo
(I)
Bourr JEVVeecoeer(Pguur

---

3.3 Future BEV world modeling
Given the current BEV feature and ego feature obtained above, BEV world model employed to predict the future BEV representation. Instead of modeling future images, which is both challenging and computationally intensive, we follow recent works [27, 36, 43] that model structured and simplified BEV representations as more efficient alternative. The current BEV feature Fbev is first flattened along the spatial dimensions and then repeated across the modality dimension, resulting in an updated Fevv IR A xHW xC, It is then concatenated with the current ego feature egoo to form
the scene feature scene E (BMx(HW++1xx The BEV world model, implemented as a Transformer encoder, produces the future scene feature Ffut E (MM+()))) From this, the future BEV feature Hiut is extracted. A lightweight BEV decoder is then applied to generate the future BEV semantic map Bue for supervision. The overall process is illustrated below:
pfut scene
BEV BEVWorlddMoeee scene)
(2)
Brut BEVDecoder
3.4 Future-aware end-to-end planning
After obtaining the current BEV feature, current ego feature, and future BEV feature, the end-to-end planning network jointly reasons over the present scene and its future evolution to generate the planned trajectories. However, enabling the planning network to simultaneously consider both the current and future BEV features is non-trivial. Directly interacting the ego feature with both can lead to entangled representations, causing confusion in ego planning and degrading performance. To address this, we adopt decoupled strategy, where the ego feature interacts with the current and future BEV features independently, and the results subsequently fused for final trajectory planning.
As illustrated in Figure (b), the current ego feature Pego interacts with the current BEV feature Fourr through Transformer decoder. The updated ego feature is then passed through an MLP decoder to generate the planned trajectories T, for supervision. Since the future BEV feature represents the future scene and corresponds to the future ego state, we initialize the future ego feature Ffut using
the endpoints of the anchored trajectories. It then interacts with the future BEV feature Ffut through a Transformer decoder. Similarly, an MI.P decoder is applied to produce the planned trajectories Tb for supervision. The process is shown below:
peurr
TransformerDecodeer Fourr Feev )
F fftt ego
Transforme::eecodder (ffut agut E
T = EgoDecoder(FF))
To = EgoDecoder (ffutt) eggo
To incorporate the future ego feature into the current ego feature, we adopt motion- aware layer normalization (MLN) [4] to obtain a future-aware ego representation. This representation is then passed through an MI.P decoder to generate the final planned trajectories Tiinal
peur
MLN ego
peurr
pfut
Trinal EgoDecoder
Feurry
3.5 Iterative scene modeling and vehicle planning
As illustrated in Figure 2 (c), the BEV world modeling network and the end-to-end planning network operate in an iterative manner to progressively improve planning performance. This is motivated by the mutual dependency between future scene evolution and ego trajectories-fuurr traffic dynamics influence the ego's motion plans, while the ego's planned actions, in turn, shape the future scene. In the end-to-end planning network, the future BEV feature Ffut serves the planned trajectories. Mcanwhile, the refined ego feature Eearr is fed back into the BEV world modeling network to produce updated future BEV feature. This iterative process is repeated N times, yielding N pairs of predicted future semantic maps and ego trajectories, denoted as (B T((1) .T (..))..... fut (N) T( T( which are all supervised during training.
reference for generating
5

---

3.6 End-to-end learning
The model is trained in an end-to-end manner with a loss function comprising two compo- nents: the BEV semantic map loss mapp and the planned trajectory loss Limj. The semantic map loss includes the current BEV map loss map and the future BEV map losses from N it-
erations, denoted as L (1) Linp (N), The trajectory loss consists of N sets of trajectory planning losses, each containing three terms as described in the end-to-end planning network: (N) (N)
(Paraj Liraj - (1),,... ".,..., (Ctraj Ltraj Cera ) The overall loss function for end-to-end training is follows:
mmap= = X1 Cmpp + >2 >2
(1) +... _mpp (N)),
Ciraj = >s(( traj
(1) clinal (1) .... (f (N) + cb Liraj (N))),
(5)
Liotal = Lmap + Curaj.
where Å1, X2, and À are the balancing factors.
Experiments
4.1
Datasets and metrics
Datasets. We conduct experiments two large-scale real-world autonomous driving datasets: NAVSIM [19] and nuScenes [17]. NAVSIM, built upon nuPlan [47], is designed for non-reactive simulation in complex scenarios with dynamic intention changes. It contains 1,192 training/validation (navtrain) and 136 testing (navtest) scenarios, with 8-camera and LiDAR data at 2 Hz. nuScenes includes 1,000 scenes with 6-camera and LiDAR data at Hz. We follow the standard 700/150 train/validation split and evaluate planning in an open-loop setting.
Evaluation metries. For the NAVSIM dataset, we the PDM Score (PDMS), which comprises multiple sub-metrics: No At-Fault Collisions (NC), Drivable Area Compliance (DAC). Time-to- Collision (TTC), Comfort (Comf.), and Ego Progress (EP). For the nuScenes dataset, we follow the VAD [13] setting and report the L2 Displacement Error and Collision Rate.
4.2 Implementation details
Model settings. The model for the NAVSIM dataset uses both images and LiDAR as input, whereas the nuScenes model relies solely on images. For backbone networks, ResNet34 is employed for NAVSIM and ResNet50 for nuScenes. Regarding the number of camera views, 3 views are used in NAVSIM and 6 in nuScenes. The input image resolution is 1024x256 following TransFuser for NAVSIM, and 640x360 for nuScenes. The number of trajectory modes is set to 256 for NAVSIM and 6 for nuScenes. For NAVSIM, the planning horizon is 4 seconds with 8 future steps, and for nuScenes, 3 seconds with 6 steps. both settings, we predict the future BEV semantic map corresponding to the final planning step.
Training settings. The model is trained on NVIDIA GeForce RTX 3090 GPUs. For NAVSIM, the batch size 16 per GPU, with 30 training epochs and total training time of around 5 hours. For nuScenes, the batch size is 1 per GPU, with 12 epochs and a training time of about 12 hours. The learning rate is set to 10 for NAVSIM and 10- for nuScenes, both optimized using AdamW [48]. The loss balancing factors are set to \₁ = 10, X = 0.1. and X3 1 for NAVSIM, and all set to 1 for nuScenes.
4.3 Comparison with state of the art
As shown in Table we compare SeerDrive with several oottte----heerrrr methods on the NAVSIM dataset. Our approach achieves the highest PDM Score of 88.9. Under the same ResNet34 [49] backbone, SeerDrive outperforms recent methods, including the trajectory refinement model Hydra- NeXt [50], the online trajectory evaluation method WoTE [43], and the truncated diffusion policy method DiffusionDrive [15], demonstrating the effectiveness of our iterative world modeling and

---

Table 1: Performance comparison of planning the NAVSIM naviest split with closed-loop metrics. "C & L" represents the use both camera and LiDAR as sensor inputs. ResNet34 [49] is employed as the backbone for BEV feature extraction, following TransFuser [21]. The best and second best results are highlighted in bold and underline, respectively. i denotes the use of V2-99 [51] as the image backbone.
Method
Input
NC
DAC t
TTC
Comf. f
EP
PDMS f
VADv2-V8192 [25]
97.2
89.1
91.6
100
76.0
80.9
Byydra-MDP-Vssua [46]
C & L
97.9
91.7
92.9
100
77.6
83.0
UniAD [12]
Camera
97.8
91.9
92.9
100
78.8
83.4
LTF [21]
Camera
97.4
92.8
92.4
100
79.0
83.8
PARA-Drive [55]
Camera
97.9
92.4
93.0
99.8
79.3
84.0
TransFuser [21]
C L
97.7
92.8
92.8
100
79.2
84.0
LAW [26]
Camera
96.4
95.4
88.7
99.9
81.7
84.6
DRAMA [56]
98.0
93.1
94.8
100
80.1
85.5
MDP++ [57]
Camera
97.6
96.0
93.1
100
80.4
86.6
DiffusionDrrie [15]
C
98.2
96.2
94.7
100
82.2
88.1
WoTE [43]
C & L
98.5
96.8
94.9
99.9
81.9
88.3
Hydra-NeXI [50]
98.1
97.7
94.6
100
81.8
88.6
SeerDrive
C & L
98.4
97.0
94.9
99.9
83.2
88.9
GoalFlow f [28]
98.4
98.3
94.6
100
85.0
90.3
SeerDrive!
98.8
98.6
95.8
100
84.2
90.7
Table 2: Performance comparison of planning on the nuScenes validation split. ResNet50 [49] is used the backbone for all methods, except for UniAD [12], which adopts ResNet101. "w/o bev" indicates without future BEV injection, and "w/o iter" indicates without iterative world modeling and planning.
L2 (m) $
Col. Rate (%
+
Method
18
28
38
Avg.
18
28
38
Avg.
ST-P3 [22]
1.33
2.11
2.90
2.11
0.23
0.62
1.27
0.71
BEV-Planner [54]
0.28
0.42
0.68
0.46
0.04
0.37
1.07
0.49
VAD-Tiny [13]
0.46
0.76
1.12
0.78
0.21
0.35
0.58
0.38
LAW [26]
0.26
0.57
1.01
0.61
0.14
0.21
0.54
0.30
PARA-Drive [55]
0.25
0.46
0.74
0.48
0.14
0.23
0.39
0.25
VAD-Base [13]
0.41
0.70
1.05
0.72
0.07
0.17
0.41
0.22
GenAD [58]
0.28
0.49
0.78
0.52
0.08
0.14
0.34
0.19
UniAD [12]
0.44
0.67
0.96
0.69
0.04
0.08
0.23
0.12
BridgeAD [53]
0.29
0.57
0.92
0.59
0.01
0.05
0.22
0.09
MomAD [52]
0.31
0.57
0.91
0.60
0.01
0.05
0.22
0.09
SparseDrive [14]
0.29
0.58
0.96
0.61
0.01
0.05
0.18
0.08
SeerDrive w/o bev
0.22
0.49
0.73
0.48
0.02
0.05
0.16
0.08
SecrDrive w/o
0.28
0.47
0.81
0.52
0.02
0.07
0.20
0.10
SeerDrive
0.20
0.39
0.69
0.43
0.00
0.05
0.14
0.06
planning strategy. Furthermore, when replacing the ResNet-34 backbone with V2-99 [51], as done in GoalFlow [28], our method achieves better performance with significantly lower computational cost while supporting end-to-end training.
As shown in Table 2, we further evaluate our method on the nuScenes dataset. It achieves notable performance gains compared with recent state-of-the-art methods, including SparseDrive [14], Mo- mAD [52], and BridgeAD [53]. Although the nuScenes dataset contains relatively simple scenarios and imperfect evaluation metries [54], our method still achieves clear improvements, demonstrating the effectiveness of the two main designs.

---

4.4 Ablation study
Effects of components. As shown Table 3, we conduct an ablation study on the key components of our method, including Future-Aware Planning and Iterative Scene Modeling and Vehicle Planning. In the first row, both modules are removed, so the planned trajectories rely only on the current BEV feature and no iterative process is applied. This leads drop in PDMS from 88.9 87.1. In the second row, we remove the future BEV injection for planning, which also results in decreased performance, showing the importance of future BEV features for planning. In the third row, remove the iterative process, and performance drops as well, indicating its role in refining the results. The last row presents the full SeerDrive model with both modules included, achieving the highest performance.
Table Ablation study on the key components of our model. "Iter. S&V" refers to Iterative Scene Modeling and Vehicle Planning.
Future-aware plan
Iter. S&V
NC
DAC f
TTC
Comf.
EP
PDMS
98.3
95.6
94.5
100
81.1
87.1
98.2
96.6
94.3
100
82.0
87.9
98.2
96.5
94.4
100
82.5
88.1
98.4
97.0
94.9
99.9
83.2
88.9
Effects Future-Aware Planning. As shown in Table 4, we conduct an ablation study on the design of the future- end-to-end planning. The first two analyze different strategies for incorporating the future BEV feature. the first row, future BEV injection is removed entirely. In the second row, the decoupled strategy is omitted, and the network learns jointly from current and future BEV features. Both settings lead to notable performance degradation, highlighting the effectiveness of our proposed design. In the third and fourth rows, analyze how the current and future ego features are combined. the third row, the Motion-aware Layer Normalization (MLN) is removed, and the features are concatenated and passed through MLP for dimension alignment. In the fourth row, MLN is also removed, and the features are directly added. Both variations lead to notable drop in performance, demonstrating the effectiveness of MLN producing future-aware ego feature.
Table 4: Ablation study on the design future-aware end-to-end planning.
Type
NC f
DAC 1
TTC
Comf. f
EP 1
PDMS 1
w/o Future BEV
98.2
96.6
94.3
100
82.0
87.9
w/o Decouple
98.0
96.0
94.0
99.5
81.6
87.3
ML.22at
98.3
96.6
94.7
100
82.4
88.3
ML.2Add
98.2
97.0
94.1
100
82.9
88.5
SeerDrive
98.4
97.0
94.9
99.9
83.2
88.9
Effects of the number of iterations. As shown in Table 5, moderate number of iterations for scene modeling and vehicle planning achieves good balance between efficiency and performance. We observe that using two iterations yields the best trade-off.
Table Ablation study on the number iterations for scene modeling and vehicle planning.
Number
NC t
DAC
TTC 1
Comf.
EP
PDMS 1
98.4
96.6
94.5
100
82.2
88.1
98.4
97.0
94.9
99.9
83.2
88.9
98.4
97.2
94.9
100
82.5
88.7

---

Prediction steps of future BEV. We predict the future BEV only at the final planning step, as the last frame provides the most informative reference for determining the trajectory direction. This design yields an effective and efficient planning reference. As shown in Table 6, we further analyze this choice by predicting sequence of intermediate BEV representations and concatenating them as planner inputs. However, incorporating intermediate predictions brings no significant performance gains while increasing model complexity. Therefore, predicting only the final future BEV at the last planning step proves to be the more effective and efficient strategy.
Table 6: Ablation study on the number of prediction steps for future BEV.
Prediction steps
NC1
DAC
TTC f
Comf.
EP
PDMS
2--2--3s-
98.3
97.0
94.6
100
83.0
88.8
2s-4s
98.4
97.2
94.8
100
83.1
88.9
4s (Origin) 98.4
97.0
94.9
99.9
83.2
88.9
The performance of Tas The and Tfmal As shown in Table 7, we conduct an ablation study to separately evaluate the performance of Ta, Th, and Trina- The results show that Thnal significantly outperforms both Ta and Tb, indicating that each contributes meaningfully to the superior performance of the final trajectory.
Table 7: Ablation study on the performance of Ta. To. and Triall
NC 1
DAC t
TTC 1
Comf. 1
EP 1
PDMS t
98.3
96.6
94.4
100
82.7
88.3
Tb
98.2
96.4
94.2
99.9
83.0
88.2
Tinal
98.4
97.0
94.9
99.9
83.2
88.9
The use of the anchored endpoints initialize the future ego feature. The future BEV cor- responds to the final planning step. Thus, we initialize the future ego feature using the anchored endpoints, allowing it to encode prior knowledge of the ego vehicle's future state the final planning step. For more extensive evaluation, we analyze different initialization strategies for the future ego feature. As shown in Table 8, using anchored endpoints as a prior yields the best performance.
Table 8: Ablation study on the use of the anchored endpoints initialize the future ego feature.
NC f
DAC 1
TTC f
Comf. f
EP 1
PDMS f
Random Anchored trajectories
98.3
96.9
94.5
100
82.8
88.6
98.3
96.8
94.0
100
83.7
88.7
Anchored endpoints (Origin)
98.4
97.0
94.9
99.9
83.2
88.9
4.5 Qualitative results
As shown in Figure 3, we visualize two cases: a right turn and a left turn. In the bottom middle figure, the final planned trajectory generated by our model closely aligns with the ground-truth trajectory. The bottom right figure presents the planned multi-modal trajectories, capturing multiple possible future motions. The bottom left figure shows the predicted future BEV semantie maps, which reflect how the scene evolves and how the ego vehicle's 's position changes after the turning behaviors.
5 Conclusion
This paper presents SeerDrive, a unified end-to-end framework that combines future scene model- ing and trajectory planning. Unlike traditional one-shot approaches that rely only the current scene, SeerDrive predicts future BEV representations to guide planning. It introduces two core components: Future-Aware Planning, which incorporates predicted future context into trajectory

---

Front left
Front
Front right
Predicted future BEV semantic map
GT trajectory
Planned trajectory
Planned trajectory
Multi-modal trajectories
(a) Right-turn case
left
Front
Front right
Predicted future BEV semantic map
GT trajectory
Planned trajectory
Planned trajectory
Multi-modal trajectories
(b) Left-turn case
Qualitative results on the NAVSIM dataset. The visualization includes three front-facing camera views: front left, front, and front right, as well as model outputs including the predicted future BEV semantic map, the planned trajectory, and multi-modal predicted trajectories.
Figure 3:
generation, and Iterative Scene Modeling and Vehicle Planning, which refines both scene predictions and plans through repeated interaction. This design enables more informed and adaptive decisions. Extensive experiments on NAVSIM and nuScenes show approach achieves state-of titte-of-the-ale results.
Limitations and future work. The BEV world model adopts a transformer architecture specifically tailored framework, making it both effective and efficient for planning. However, it does not benefit from the generalization capabilities of foundation models. On the other hand, using off-the- shelf foundation models world models often suffers from slow inference speed and challenges in joint optimization with the planner. Therefore, developing tightly integrated paradigm of planning and world modeling represents promising direction for future work.
10

---

Acknowledgments
This work was supported in part by National Natural Science Foundation of China (Grant No. 62376060).
References
Li Chen, Penghao Wu, Kashyap Chitta, Bernhard Jaeger, Andreas Geiger, and Hongyang Li. End-to-end autonomous driving: Challenges and frontiers. IEEE TPAMI, 2024.
[2] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird's-eye-vice representation from multi-camera images via spatiotemporal trans- formers. In ECCV, 2022. 1,2
[3] Chenyu Yang. Yuntao Chen, Hao Tian, Chenxin Tao, Xizhou Zhu, Zhaoxiang Zhang, Gao Huang. Hongyang Li, Qiao, Lewei Lu, Bevformer v2: Adapting modern image backbones bird's-eye- view recognition via perspective supervision. In CVPR, 2023.
[4] Shihao Wang, Yingfei Liu, Tiancai Wang, Ying Li, and Xiangyu Zhang. Exploring object-centric temporal modeling for efficient multi-view 3d object detection. In ICCV, 2023. 1,2,5
[5] Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Motion transformer with global intention localization and local movement refinement. In NeurlPS, 2022. 1,2
[6] Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schicle. Mtr++: Multi-agent motion prediction with symmetric seene modeling and guided intention querying. IEEE TPAMI, 2024,
[7] Zikang Zhou, Jianping Wang. Yung- Hui Li, and Yu-Kai Huang. Query-centric trajectory prediction. In CVPR, 2023. 1,2
[8] Bozhou Zhang, Nan Song. and Li Zhang. Decoupling motion forecasting into directional intentions and dynamic states. In NeurlPS, 2024.
[9] Jie Cheng, Yingbing Chen, Xiaodong Mei, Bowen Yang, Bo Li, and Ming Liu. Rethinking imitation-based planners for autonomous driving. ICRA, 2024. 1,2
[10] Cheng, Yinghing Chen, and Qifeng Chen. Pluto: Pushing the limit of imitation learning- -hannd planning for autonomous driving. arXiv preprint, 2024.
[11] Yinan Zheng, Ruiming Liang, Kexin ZHENG, Jinliang Zheng, Liyuan Mao, Jianxiong Li, Weihao Gu, Rui Ai, Shengbo Eben Li, Xianyuan Zhan, and Jingjing Liu. Diffusion- based planning for autonomous driving with flexible guidance. In ICLR, 2025. 1,2
[12] Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li. Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oriented autonomous driving. In CVPR, 2023. 1, 2, 7, 23
[13] Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vectorized seene representation for efficient autonomous driving. In ICCV, 2023. 1,
23
[14] Wenchao Sun, Xuewu Lin, Yining Shi, Chuang Zhang. Haoran Wu, and Sifa Zheng. Sparsedrive: End-to- end autonomous driving via sparse scene representation. ICRA, 2025. 3,
[15] Bencheng Liao, Shaoyu Chen, Haoran Yin, Jiang, Cheng Wang, Sixu Yan, Xinbang Zhang, Xiangyu Li, Ying Zhang, Qian Zhang, and Xinggang Wang. Diflusiondrive: Truncated diffuusio model for end-to-end autonomous driving. In CVPR, 2025. 2, 2,4, 6, 7,
[16] Xiaosong Jia, Junqi You, Zhiyuan Zhang, and Junchi Yan. Drivetransformer: Unified transformer for scalable end-to-end autonomous driving. In ICIR, 2025. 1,3
[17] Holger Caesar, Varun Bankiti, Alex H Lang. Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal datasct for autonomous driving. In CVPR, 2020. 1, 2, 6, 21
[18] Xiaosong Jia, Zhenjie Yang, Qifeng Li, Zhiyuan Zhang, and Junchi Yan. Bench2drive: Towards multi- ability benchmarking of closed-loop end-to-end autonomous driving. In NewrIPS, 2024. 1, 22, 23
11

---

[19] Daniel Dauner. Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang, Zetong Yang. Hongyang Li. Igor Gilitschenski, Boris Ivanovic, Marco Pavone, al. Navsim: Data-driven non-reactive autonomous vehicle simulation and benchmarking. In NeurIPS, 2024. 1, 6. 21, 22, 23
[20] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In CoRI, 2017. 1,2
[21] Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-modal fusion transformer for end-to-end autonomous driving. In CVPR, 2021. 2,7,
[22] Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao. St-p3: End-to-end feature learning. In ECCV, 2022. 2,
vision-basee autonomous driving via
[23] Xiaosong Jia. Yulu Gao. Li Chen, Junchi Yan. Patrick Langechuan Liu, and Hongyang Li. Driveadapter: Breaking the coupling barrier of perception and planning in end-to-end autonomous driving. In ICCV, 2023, 2
[24] Xiaosong Jia, Penghao Wu, Li Chen, Jiangwei Xie, Conghui He, Junchi Yan, and Hongyang Li. Think twice before driving: Towards scalable decoders for end-to-end autonomous driving. In CVPR, 2023. 2
[25] Shaoyu Chen, Bo Jiang, Hao Gao, Bencheng Liao, Qing Xu, Qian Zhang, Chang Huang. Wenyu Liu, and Xinggang Wang. Vadv2: End-to-end vectorized autonomous driving via probabilistic planning, arXiv preprint, 2024. 7,7
[26] Yingyan Li. Lue Fan, Jiawei He, Yuqi Wang, Yuntao Chen, Zhaoxiang Zhang, and Tieniu Tan. Enhancing end-to-end autonomous driving with latent world model. In ICLR, 2025. 3, 7, 22
[27] Peidong Li and Dixiao Cui. Navigation- sparse scene representation for end-to-end autonomous driving. In ICLR, 2025. 3, 5,22
[28] Zehin Xing, Xingyu Zhang, Yang Hu, Bo Jiang, Tong He, Qian Zhang, Xiaoxiao Long, and Wei Yin. Goalflow: Goal-driven flow matching for multimodal trajectories generation end-to-end autonomous driving. In CVPR, 2025. 3.7
[29] Ruiyuan Gao, Kai Chen, Enze Xie, HONG Lanqing, Zhenguo Li, Dit-Ya Yan Yeung, and Qiang Xu. Magic- drive: Street view generation with diverse 3d geometry control. In ICLR, 2024. 3
[30] Yuqing Wen, Yucheng Zhao, Yingfei Liu, Fan Jia, Yanhui Wang, Chong Luo, Chi Zhang, Tiancai Wang, Xiaoyan Sun, and Xiangyu Zhang. Panacea: Panoramic and controllable video generation for autonomous driving. In CVPR, 2024. 3
[31] Jiachen Lu, Ze Huang, Zeyu Yang, Jiahui Zhang, and Li Zhang. Wovogen: World volume-aware diffusion for controllable multi-camera driving scene generation. In ECCV, 2024. 3
[32] Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jiagang Zhu, and Jiwen Lu. Drivedreamer: Towards worl-world- world models for autonomous driving. In ECCV, 2024. 3
[33] Yuqi Wang. Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. In CVPR, 2024. 3
[34] Anthony Hu, Lloyd Russell, Hudson Yco, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia- A generative world model for autonomous driving. arXiv preprint, 2023. 3
[35] Shenyuan Gao, Jiazhi Yang. Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability. In NewrlPS, 2024.
[36] Kashyap Chitta, Daniel Dauner, and Andreas Geiger. Sledge: Synthesizing driving environments with generative models and rule-based traffic. In ECCV, 2024, 3,5 5
[37] Yihan Hu, Siqi Chai, Zhening Yang, Jingyu Qian, Kun Li, Wenxin Shao, Haichao Zhang, Wei Xu, and Qiang Liu. Solving motion planning tasks with a scalable generative model. In ECCV, 2024. 3
[38] Yanhao Wu, Haoyang Zhang, Tianwei Lin, Lichao Huang, Shujie Luo, Rui Wu, Congpei Qiu, Wei Ke, and Tong Zhang, Generating multimodal driving scenes via next-scene prediction. In CVPR, 2025.
[39] Luke Rowe, Roger Girgis, Anthony Gosselin, Liam Paull, Christopher Pal. and Felix Heide. Scenario dreamer: Vectorized latent diffusion for generating driving simulation environments. In CVPR, 2025. 3
12

---

[40] Jingyu Li, Bozhou Zhang, Xin Jin, Jiankang Deng, Xiatian Zhu, and Li Zhang. Imagidrive: A unified amagination--nd-planniii framework for autonomous driving. arXiv preprint, 2025.
[41] Ruijie Zheng, Jing Wang, Scott Reed, Johan Bjorck, Yu Fang, Fengyuan Hu, Joel Jang, Kaushil Kundalia, Zongyu Lin, Loic Magne, al. Flare: Robot learning with implicit world modeling. arXiv preprint, 2025.
[42] Wenyao Zhang, Hongsi Liu, Zekun Qi, Yumnan Wang, Xingiang Yu, Jiazhao Zhang, Runpei Dong, Jiawei He, He Wang, Zhizheng Zhang, et al. Dreamvla: a vision- aisoo--anguage-ace model dreamed with comprehensive world knowledge. In NeurlPS, 2025. 3
[43] Yingyan Li, Yugi Wang, Yang Liu, Jiawei He. Lue Fan, and Zhaoxiang Zhang. End-to-end driving with online trajectory evaluation via bev world model. arXiv preprint, 2025. 3, 4, 5, 6, 7, 21,
[44] Wenzhao Zheng. Weiliang Chen, Yuanhui Huang. Borui Zhang. Yueqi Duan, and Jiwen Lu. Oceworld: Learning a 3d occupancy world model for autonomous driving. In ECCV, 2024.
[45] Yang, Jianbiao Mei, Yukai Ma, Siliang Du Wenqing Chen, Yijie Qian, Yuxiang Feng, and Yong Liu. Driving in the occupancy world: Vision-centric 4d occupancy forecasting and planning via world models for autonomous driving. In AAAI, 2025. 3
[46] Zhenxin Li, Kailin Li. Shihao Wang. Shiyi Lan, Zhiding Yu, Yishen Ji, Zhiqi Li. Ziyue Zhu, Jan Kautz, Zuxuan Wu, et al. Hydra-mdp: End-to-end multimodal planning with multi-target hydra-distillation. arXiv preprint, 2024, 4,7
[47] Holger Caesar, Juraj Kabzan, Kok Seang Tan, Whye Kit Fong. Eric Wolff. Alex Lang, Luke Fletcher, Oscar Beijbom, and Sammy Omari. nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles. arXiv preprint, 2021.
[48] Loshchilov. Decoupled weight decay regularization. In ICLR, 2019.
[49] Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 6,
[50] Zhenxin Li, Shihao Wang, Shiyi Lan, Zhiding Yu, Zuxuan Wu, and Jose M Alvarez. Hydra-next: Robust closed-loop driving with open-loop training. arXiv preprint, 2025. 6,
[51] Youngwan Lee and Jongyoul Park. Centermask: Real-time anchor-free instance segmentation. In CVPR, 2020. 7
[52] Ziying Song, Caiyan Jia, Lin Liu, Hongyu Pan, Yongchang Zhang, Junming Wang, Xingyu Zhang, Shaoqing Xu, Lei Yang. and Yadan Luo. Don't shake the wheel: Momentum-aware planning in end-to-end autonomous driving. In CVPR, 2025. 7, 23
[53] Bozhou Zhang, Nan Song, Xin Jin, and Li Zhang. Bridging past and future: End-to-end autonomous driving with historical prediction and planning. In CVPR, 2025. 7,23
[54] Zhiqi Li, Zhiding Yu, Shiyi Lan, Jiahan Li. Jan Kautz, Tong Lu, and Jose M Alvarez. Is ego status all you need for open-loop end-to-end autonomous driving? In CVPR, 2024.7
[55] Xinshuo Weng, Boris Ivanovic, Yan Wang, Yue Wang, and Marco Pavone, Para-drive: Parallelized architecture for real-time autonomous driving. CVPR, 2024.
[56] Chengran Yuan, Zhanqi Zhang. Jiawei Sun, Shuo Sun, Zefan Huang, Christina Dao Wen Lee, Dongen Li, Yuhang Han, Anthony Wong, Keng Peng Tee, et al. Drama: An efficient end-to-end motion planner for autonomous driving with mamba. arXiv preprint, 2024.7 7
[57] Kailin Li, Zhenxin Li, Shiyi Lan, Yuan Xie, Zhizhong Zhang, Jiayi Liu, Zuxuan Wu, Zhiding Yu, and Jose M Alvarez. Hydra-mdp++:: Advancing end-to-end driving via expert-guided hydra-distillation arXiv preprint, 2025. 7
[58] Wenzhao Zheng, Ruiqi Song, Xianda Guo, Chenming Zhang, and Long Chen. Genad: Generative end autonomous driving. In ECCV, 2024. 7
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See the experiment section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
The details of the model and experiments are provided in the main paper and the appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. ec/
Policy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nips... public/guider
sssissssn wwwssssoiccyyyyyc for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: :ttt:p:///eeripeee wtt.rripppccppppupeeeeee http://p.....p.lleccccccccceeeee
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our main idea is specifically designed for applications in real-world au- tonomous driving.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: See the experiment section.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https / 220 for what should or should not be described.
20

---

Appendix
A Notations
As shown in Table 9, we provide a lookup table for notations used in the paper.
Table 9; Notations used in the paper.
Notation
Description
I
multi-view images
P
LiDAR
T
anchored multi-modal trajectories
&
ego status
H&W
spatial dimensions of the BEV feature the number of trajectory modes the number of feature channels
M
C
N
the number of iterations current ego feature
Feurr
ego
F peurr
current BEV feature map
F scene
current scene feature
pfut
future ego feature
plut
future BEV feature map
bev
F plut
future scene feature
sceme
Beure
predicted current BEV semantic map predicted future BEV semantic map
Blut
Ta
planned multi-modal trajectories by current BEV planned multi-modal trajectories by future BEV final planned multi-modal trajectories
To
Trmal
B Implementation details
In addition to the implementation details provided the main paper, we offer further clarifications here. The feature channels C' are set to 256 for both the NAVSIM [19] and nuScenes [17] datasets. Regarding the spatial dimensions H
the BEV features, they are 8 x for NAVSIM and 100 100 for nuScenes. However, for future BEV map prediction, the features are uniformly downsampled to x 8 in nuScenes. For the anchored multi-modal trajectories, we follow previous works [14, 15, 43] and use the K-Means algorithm to cluster them from the ground-truth trajectories.
C Evaluation metrics
In the NAVSIM dataset, trajectories spanning 4 seconds 2Hz are first produced and then upsampled to 10Hz using an LQR controller. These refined trajectories are evaluated using set of closed- loop metrics, which include No At-Fault Collisions (5xc), Drivable Area Compliance (SDAC), Time to Collision with bounds (STrc), Ego Progress (SEP), Comfort (ScF), and Driving Direction Compliance (SDDC). The overall performance score computed by aggregating these individual metrics. However, due implementation limitations, SDDG is excluded from the final evaluation
SPdm SNC SDAC STTCX
5 x SEP 5 x SCE 2 x SDDC
(6)
12
21

---

D
Qualitative results
In addition the qualitative results, we provide additional visualizations in Figure 4 and Figure 5 to further demonstrate the effectiveness of our model.
E
Failure cases
Although our SeerDriiv demonstrates strong performance, it still encounters some failure cases.
As illustrated in Figure 6 (a), the model fails choose the correct lane after right turn. In the right subfigure of (a), which shows the multi-modal planning result, one of the predicted trajectories closely aligns with the ground truth. However, the classification score fails select it, suggesting that the multi-modal trajectory selection process requires further improvement.
As illustrated in Figure 6 (b), the model fails to infer the correct driving intention for right turn. Most of the planned trajectories instead tend to change lanes to the left and proceed straight. This suggests that incorporating high-level driving intentions, such explicit driving commands, is necessary for achieving more accurate planning outcomes.
F Discussions
Comparison with existing methods. Table 10 summarizes the differences between our work and prior approaches in terms how predicted future scenes from world models are utilized.
Table 10: Comparison between SeerDrive and existing methods.
Method
Usage of predicted future scene from world model
LAW [26]
Used as an auxiliary supervision signal during training
SSR [27]
Used as an auxiliary supervision signal during training
WoTE [43]
Used assist in selecting the best trajectory among multiple candidates
SeerDrive (Ours)
Used feature-level reference for planning, enabling iterative interaction with planner
Experiment complex scenarios. Following the reviewer's valuable suggestions, we evaluate our model on the test split of NAVSIM [19] under scenarios with more than 10 agents. As shown in Table 11, our model still demonstrates strong performance.
Table 11: Experiment on complex scenarios.
NC t
DAC
TTC 1
Comf.
EP
PDMS
Complex scenarios
98.2
96.8
94.3
99.9
83.1
88.5
Experiment beyond NAVSIM and nuScenes. Following the reviewer's valuable suggestions, we have further evaluated the complex Bench2 [18] dataset with many and high-entropy scenarios such as emergency braking, merging, and overtaking, along with longer planning horizon and closed-loop testing. As shown in Table 12, our model consistently achieves the best performance.
Uncertainty in future predictions and planning. To tackle the uncertainty caused by the multi- modal characteristies of future trajectories, our framework performs prediction and planning while explicitly considering the multi-modal nature of the future. The predicted future BEV feature is multi-modal and shares the same modality the ego feature used for planning. Accordingly, also predict a multi-modal future BEV representation. For supervision, we adopt a winner-taken takes-all strategy: the best trajectory and the corresponding future BEV selected using the same probability generated from the current ego feature, and the loss computed based on the selected mode.
Parameter size and inference cost. We evaluate our model on the NAVSIM [19] dataset. The model has 66 parameters, and the average inference time is 24 ms under the same configuration as in the main paper.
22

---

Table 12: Experiment the Bench2Drive [18] The first metric corresponds to open-loop testing, while the latter two are used for closed-loop evaluation.
Method
Avg. L2 Error +
Driving Score f
Success Rate (%) f
VAD [13]
0.91
42.35
15.00
UniyD-Tiny [12]
0.80
40.73
13.18
UniAD-Base [12]
0.73
45.81
16.46
MomAD [52]
0.82
47.91
18.11 18
BridgeAD [53]
0.71
50.06
22.73
SeerDrive (Ours)
0.66
58.32
30.17
Quantitative metric for predicted future BEV. the NAVSIM [19] dataset is 38.87.
The mloU of the predicted future BEV map on
23

---

Front left
Front
Front right
Predicted future BEV
semantic map
trajectory
Planned trajectory
Planned trajectory
Multi-modal trajectories
a
Front left
Front
Front right
8
81
Predicted future BEV semantic map
GT trajectory
Planned trajectory
Planned trajectory
Multi-modal trajectories
(b)
Figure 4: Qualitative results 1 on the NAVSIM dataset. The visualization includes three front-facine camera views: front left, front, and front right, as well as model outputs including the predicted future BEV semantic map, the planned trajectory, and multi-modal predicted trajectories.
24

---

Front left
Front
Front right
5
5
Predicted future BEV semantic map
GT trajectory
Planned trajectory
Planned trajectory
Multi-modal trajectories
Front left
Front
Front right
Predicted future BEV semantic map
GT trajectory
Planned trajectory
Planned trajectory
Multi-modal trajectories
dd)
Qualitative results 2 on the NAVSIM dataset. The visualization includes three front-facing camera views: front left, front, and front right, well model outputs including the predicted future BEV semantic map, the planned trajectory, and multi-modal predicted trajectories.
Figure 5:
25

---

39
Top CORE
b
Figure 6: Failure cases from the NAVSIM dataset. In the left figure, the ground-truth trajectory is shown green, while the top-ranked planning trajectory, selected based on the classification score, displayed in orange. The right figure illustrates the predicted mudti-modal trajectories.
26


Reviews:
[{"review_id": "Kqnwsi3FJs", "reviewer": "Reviewer_W3tz", "summary": "This paper addresses the limitations of existing end-to-end autonomous driving frameworks that typically operate in a one-shot manner, relying primarily on the current scene context without sufficiently considering the temporal evolution of the environment. \n\nTo address these issues, the authors introduce SeerDrive, a novel end-to-end framework designed to jointly model future scene evolution and trajectory planning in a closed-loop manner. The method consists of two principal components: \n(1) Future-aware planning, which incorporates predicted future bird’s-eye view (BEV) scene features directly into the trajectory planner. \n(2) Iterative scene modeling and vehicle planning, which jointly refines scene evolution predictions and planned trajectories through collaborative optimization. \n\nExtensive experiments on the NAVSIM and nuScenes benchmarks demonstrate that SeerDrive achieves notably better performance than current state-of-the-art methods.", "questions": "1. See weaknesses.\n\n2. What are the parameter size and inference cost of the proposed method?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The proposed SeerDrive framework shows improved performance and achieves state-of-the-art results in mainstream benchmarks.\nAblation studies demonstrate the effectiveness of the proposed modules.\n\n2. The paper is well-written, with clear notations, equations and figures, making it easy for readers to follow.\n\nWeaknesses:\n\n1. There are several existing methods leveraging scene future prediction as a latent world model to enhance planning, for example LAW [1], which leverages current sensor data and ego waypoints to predict future latent representation, then utilizes it as a prior for planning.\nWhat are the significant contributions of the proposed framework, comparing with the existing works?\n\n2. According to ablation studies, compared with the baseline, the proposed modules contribute limited improvement.\n\n3. The paper provides failure cases and limitation analysis, while the discussion seems to be shallow. It is beneficial to provide more case studies and an in-depth discussion to guide future work.\n\n[1] Enhancing end-to-end autonomous driving with latent world model. Li et al. ICLR 2025.", "was_revised": true, "final_justification": "The response effectively addresses my concerns about the originality compared to previous methods and the effectiveness of the proposed modules. The proposed approach to exploring the interaction between world modeling and planning seems well-justified, and the experiments on three popular benchmarks demonstrate its effectiveness.", "rebuttal": "We thank the reviewer for the detailed review and the suggestions for improvement. Below are our responses to the reviewer’s comments:\n\n### **Weakness 1: Comparison with existing methods.**\n\n- Thanks for this comment. As discussed in Line 104 of the main paper, LAW [3] leverages world modeling as auxiliary supervision during training. In contrast, our method establishes feature-level interaction between world modeling and end-to-end planning, uniquely enabling iterative and closed-loop communication between the two processes. This design facilitates deeper integration, leading to significant improvements in planning performance.\n\n- We summarize in the table below to highlight the differences between our and prior works. We will further clarify in the revised paper.\n\n| Method            | Usage of predicted future scene from world model                                          |\n|-------------------|--------------------------------------------------------------------------------------------|\n| LAW [3]           | Used as an auxiliary supervision signal during training                                   |\n| SSR [4]           | Used as an auxiliary supervision signal during training                                   |\n| WoTE [2]          | Used to assist in selecting the best trajectory among multiple candidates                 |\n| SeerDrive (Ours)  | Used as feature-level reference for planning, enabling iterative interaction with planner |\n\n### **Weakness 2: About ablation studies.**\n\n- We respectively disagree. As shown in Table 3 of the main paper, we conduct ablation studies on NAVSIM to evaluate the effectiveness of the proposed modules. By introducing Future-Aware Planning and Iterative Scene Modeling and Vehicle Planning, the PDMS (the most important metric) improves from 87.1 to 88.9, with a notable gain of +1.8, establishing a new state of the art (see Table 1 in main paper). Without these modules, the performance (87.1) is inferior to that of DiffusionDrive [1] (88.1), WoTE [2] (88.3), and Hydra-NeXt [5] (88.6).\n\n- As shown in Table 2 of the main paper, we also conduct ablation studies on nuScenes, which demonstrate clear performance gains from our proposed modules. In addition, we have now included another ablation experiment on nuScenes by removing all the proposed modules. The results below show a consistent and significant performance drop, further validating the effectiveness of our design.\n\n| Method | L2↓ 1s | 2s | 3s | Avg. | Col.↓ 1s | 2s | 3s | Avg. |\n|--------|--------:|----:|----:|------:|----------:|----:|----:|------:|\n| Baseline    | 0.40   |0.65| 1.02| 0.69  | 0.06    | 0.14| 0.37| 0.19 |\n| SeerDrive (Ours)    | **0.20**   | **0.39**| **0.69**|**0.43**  | **0.00**  |**0.05**|**0.14**| **0.06**  |\n\n### **Weakness 3: Case studies and discussions.**\n\n- Thanks for this suggestion. In addition to the discussion provided in Line 262 of main paper, we will further add an in-depth discussion as follows: \n\n_The BEV world model adopts a transformer architecture specifically tailored to our framework, making it both effective and efficient for planning. However, it does not benefit from the generalization capabilities of foundation models. On the other hand, using off-the-shelf foundation models as world models often suffers from slow inference speed and challenges in joint optimization with the planner. Therefore, developing a tightly integrated paradigm of planning and world modeling represents a promising direction for future work._\n\n- We will add in-depth limitation analysis to guide future work in revision.\n\n- Regarding failure cases, beyond those presented in Line 20 of the appendix, we will include additional examples along with more detailed analysis in the revised version of the paper.\n\n### **Question 2: Parameter size and inference cost.**\n\nThanks, and we will provide it in the revision as follows:\n\n_The model size is 66M, and the average inference time is 24 ms on the NAVSIM dataset, under the same configuration as in the main paper._\n\n>[1] DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving, CVPR 2025.\n\n>[2]  End-to-end driving with online trajectory evaluation via bev world model, ICCV 2025.\n\n>[3] Enhancing end-to-end autonomous driving with latent world model, ICLR 2025.\n\n>[4] Navigation-guided sparse scene representation for end-to-end autonomous driving, ICLR 2025.\n\n>[5] Hydra-NeXt: Robust Closed-Loop Driving with Open-Loop Training, arXiv 2503.12030"}, {"review_id": "luRugjWNhN", "reviewer": "Reviewer_LXsM", "summary": "The paper proposes SeerDrive, an end-to-end autonomous driving framework that integrates future scene understanding into trajectory planning. The key idea is to couple a bird’s-eye view (BEV) world model that predicts semantic features of the future scene with a trajectory planner, enabling the system to make informed driving decisions based on anticipated changes in the environment.\n\nUnlike prior work that either plans based on the current scene or uses separate modules for future prediction and planning, SeerDrive introduces a future-aware planning mechanism: it injects predicted future scene features directly into the planner and iteratively refines both the world model and the planner states over multiple steps before outputting the final trajectory.\n\nThe model is trained end-to-end using a joint loss over future BEV prediction and trajectory planning. Experiments on two datasets—NAVSIM (a closed-loop simulator) and nuScenes (open-loop real-world data)—show that SeerDrive achieves state-of-the-art driving performance, particularly in terms of safety and success rate under long-horizon planning. Ablation studies demonstrate the benefits of future feature injection and iterative refinement.", "questions": "- How strong is the predicted future BEV? Provide mIoU of predicted lane & actor masks at t+H to quantify BEV fidelity.\n- How does SeerDrive handle uncertainty in future predictions and planning?\n- How sensitive is SeerDrive to the choice of loss weights (λ₁–λ₃)?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The paper presents a well-engineered framework that effectively combines future scene prediction and trajectory planning in a closed-loop, end-to-end differentiable system.\n- The experimental evaluation is thorough on both the NAVSIM and nuScenes datasets. It shows state-of-the-art results in closed-loop driving metrics.\n- The proposed iterative refinement loop between the planner and world model is implemented cleanly and shown to improve performance over non-iterative baselines.\n- Ablation studies address several design choices such as feature injection and iteration count.\n- SeerDrive serves as a bridge between scene-centric world models and goal-directed planning, which are often treated separately.\n\nLimitations:\n- The world model only predicts the final future BEV at horizon H, not a sequence of intermediate steps, which may limit temporal expressiveness and robustness in dynamic scenes.\n- No uncertainty modeling is present in the predicted BEV or planning outputs, which can be critical in real-world driving.\n- The loss balancing between multiple objectives is hand-tuned, with no sensitivity analysis shown; this raises questions about stability and generalizability.\n- Gains over prior work like DiffusionDrive or WoTE are relatively small (e.g., +0.3–0.8 PDMS) and may not justify the additional complexity of iterative inference and dual-task training.", "was_revised": true, "final_justification": "The response effectively addresses several of my concerns, especially about uncertainty in future predictions and planning. The approach is interesting.", "rebuttal": "We thank the reviewer for the detailed review and the suggestions for improvement. Below are our responses to the reviewer’s comments:\n\n### **Limitation 1: The world model predicts the final future BEV at horizon H.**\n\n- **Explanation**: We predict the final future BEV at the horizon $H$, corresponding to the last planning step, as the future scene at the final frame serves as the primary reference with the most information for determining the final direction of the trajectory. This provides an effective and efficient reference for planning.\n\n- **Experiment**:  Regardless, as suggested, we have now analyzed this design choice by additionally predicting a sequence of intermediate BEV representations and concatenating them as references for the planner. As shown in the table below, incorporating intermediate predictions does not lead to significant performance improvements, while it increases model complexity. Thus, directly predicting the final future BEV at the last planning step is a more effective and efficient choice.\n\n| Prediction steps | NC ↑ | DAC ↑ | TTC ↑ | EP ↑ | PDMS ↑ |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| 1s-2s-3s-4s | 98.3 | 97.0 | 94.6 | 83.0 | 88.8 |\n| 2s-4s | 98.4 | 97.2 | 94.8 | 83.1 | 88.9 |\n| 4s (Origin) | 98.4 | 97.0 | 94.9 | 83.2 | 88.9 |\n\n### **Limitation 2 and Question 2: Uncertainty in future predictions and planning.**\n\nApologies for lack of clarity.\n- To tackle the uncertainty caused by the multi-modal characteristics of future trajectories, our framework performs prediction and planning while explicitly considering the multi-modal nature of the future. As described in Line 143 of the main paper, the predicted future BEV feature is multi-modal and shares the same modality as the ego feature used for planning. Accordingly, we also predict a multi-modal future BEV representation. For supervision, we adopt a winner-takes-all strategy: the best trajectory and the corresponding future BEV are selected using the same probability generated from the current ego feature, and the loss is computed based on the selected mode.\n\n- To address the uncertainty that may lead to errors in both prediction and planning, we propose the Iterative Scene Modeling and Vehicle Planning module. This design allows for iterative refinement of both future BEV prediction and trajectory planning, resulting in more accurate and consistent outcomes.\n\n- We will revise the paper to include a clearer and more detailed description of this design.\n\n### **Limitation 3 and Question 3: The choice of loss weights.**\n\n- **Explanation**: Thanks, and we will clarify. The loss balancing among multiple objectives is not specially hand-tuned. Instead, we follow the characteristics of the NAVSIM dataset and prior works [1][2], by scaling the current BEV loss by a factor of 10 and reducing the future BEV loss weight to 0.1. For nuScenes, we use uniform loss weights, setting all to 1. \n\n- **Experiment**:  As suggested, we have now conducted a sensitivity analysis, as shown in the table below. The results indicate that SeerDrive is not particularly sensitive to the choice of loss weights ($\\lambda_1$, $\\lambda_2$, $\\lambda_3$).\n\n| Loss weight | NC ↑ | DAC ↑ | TTC ↑ | EP ↑ | PDMS ↑ |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| $\\lambda_1$=1, $\\lambda_2$=1, $\\lambda_3$=1 | 98.2 | 96.9 | 94.5 | 82.6 | 88.5 |\n| $\\lambda_1$=5, $\\lambda_2$=0.5, $\\lambda_3$=1 | 98.4 | 96.9 | 94.8 | 83.1 | 88.8 |\n| $\\lambda_1$=10, $\\lambda_2$=0.1, $\\lambda_3$=1 (Origin) | 98.4 | 97.0 | 94.9 | 83.2 | 88.9 |\n\n### **Limitation 4: Gains over prior work.**\n\n- **Explanation**: We do not directly build upon previous methods such as DiffusionDrive [1] or WoTE [2], which are orthogonal to our approach. Nevertheless, our framework achieves state-of-the-art performance. Specifically, unlike DiffusionDrive [1], we do not adopt a diffusion-based policy for planning, and unlike WoTE [2], we do not employ multiple future BEV predictions for evaluation. We believe that incorporating such designs into our framework could potentially lead to further performance improvements.\n\n- **Gains compared to the baseline**: As demonstrated in Table 3 of the main paper, our proposed design improves the PDMS from 87.1 to 88.9, with a notable gain of +1.8. Additionally, as shown in Table 2, we conduct ablation studies on the nuScenes dataset, which clearly validate the effectiveness of our proposed modules through consistent performance improvements.\n\n- **Complexity**: Despite the iterative inference and dual-task training, our model does not introduce additional complexity. The average inference time is 24 ms and the parameter size is 66M, which is comparable to DiffusionDrive [1] and WoTE [2].\n\n### **Question 1: Quantitative metric for predicted future BEV.**\n\nGreat point! The mIoU of the predicted future BEV map on the NAVSIM dataset is 38.87. We will clarify.\n\n>[1] DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving, CVPR 2025.\n\n>[2] End-to-end driving with online trajectory evaluation via bev world model, ICCV 2025."}, {"review_id": "0gDGYO4Ryb", "reviewer": "Reviewer_TxmL", "summary": "This paper proposes SeerDrive, an end-to-end autonomous driving framework that explicitly models the bidirectional relationship between future scene evolution and ego trajectory planning. The method first predicts the future bird’s-eye-view (BEV) representation of the environment using a transformer-based world model, and then incorporates this predicted future into the planning module. Moreover, the authors propose an iterative refinement strategy where the planning and world modeling modules are jointly optimized in a closed-loop manner. Experiments on NAVSIM and nuScenes show that SeerDrive achieves state-of-the-art performance, and ablation studies demonstrate the effectiveness of its two core components: future-aware planning and iterative scene modeling.", "questions": "1. Is $\\mathcal{T}_{\\text{final}}$ a single trajectory or a set of trajectories? If it represents multiple candidates, how is the final trajectory selected from multiple candidates?\n\n2. Please provide an ablation study that separately evaluates the performance of $\\mathcal{T}_a$, $\\mathcal{T}_{b}$, and $\\mathcal{T}_{\\text{final}}$ as described in Section 3.5. This would help clarify the contribution of each trajectory output in the overall planning process.\n\n3. In the implementation, why is the future ego feature initialized using the anchored endpoints? Given that the endpoints are fixed while the future ego feature is updated iteratively, this design choice seems inconsistent and warrants further clarification.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper proposes a bidirectional modeling scheme between scene evolution and trajectory planning, challenging the prevalent one-shot planning paradigm. This is a fresh and timely perspective in end-to-end driving research.\n\n2. The framework is well-structured, with clear component design including BEV feature modeling, future-aware planning via decoupled interaction, and iterative refinement.\n\n3. SeerDrive shows competitive or superior performance on both NAVSIM and nuScenes benchmarks. The ablation studies are thorough and help validate each design component.\n\nWeakness\n1. The method bears resemblance to the existing WoTE [1] framework. The introduction and related work sections should more clearly emphasize the differences and connections between this work and WoTE [1].\n\n\n[1]:End-to-end driving with online trajectory evaluation via BEV world model.", "was_revised": true, "final_justification": "The authors’ rebuttal has addressed my concerns. I therefore maintain my positive assessment.", "rebuttal": "We thank the reviewer for the detailed review and the suggestions for improvement. Below are our responses to the reviewer’s comments:\n\n### **Weakness 1: Differences and connections with WoTE.**\n\n- As discussed in Line 104 of the main paper, WoTE [2] employs a BEV world model to generate future BEV, which are then used to select the best trajectory from a set of candidates proposed by the planner. In contrast, our SeerDrive framework establishes feature-level interaction between BEV world modeling and end-to-end planning, enabling uniquely iterative and closed-loop communication between the two components. \n\n- While both WoTE and SeerDrive leverage a BEV world model to support the planning process, their designs are fundamentally different and orthogonal. We will clarify these distinctions in the revised paper.\n\n### **Question 1: About $\\mathcal{T}\\_{\\text{final}}$.**\n\n- Apologies for the lack of clarity. $\\mathcal{T}\\_{\\text{final}}$ refers to a set of candidate trajectories. A probability score is assigned to each trajectory based on the current ego feature, and the final trajectory is selected according to the highest score, consistent with prior work [1]. \n- As described in Line 200 of the main paper, we also specify the number of trajectory modes. This clarification will be highlighted in the revised version of the paper.\n\n### **Question 2: Ablation study on the performance of $\\mathcal{T}\\_{\\text{a}}$, $\\mathcal{T}\\_{\\text{b}}$, and $\\mathcal{T}\\_{\\text{final}}$.**\n\nThanks. We conduct an ablation study to separately evaluate the performance of $\\mathcal{T}\\_{\\text{a}}$, $\\mathcal{T}\\_{\\text{b}}$, and $\\mathcal{T}\\_{\\text{final}}$, as shown in the table below. The results show that $\\mathcal{T}\\_{\\text{final}}$ significantly outperforms both $\\mathcal{T}\\_{\\text{a}}$ and $\\mathcal{T}\\_{\\text{b}}$, indicating that each contributes meaningfully to the superior performance of the final trajectory.\n\n|  | NC ↑ | DAC ↑ | TTC ↑ | EP ↑ | PDMS ↑ |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| $\\mathcal{T}\\_{\\text{a}}$ | 98.3 | 96.6 | 94.4 | 82.7 | 88.3 |\n| $\\mathcal{T}\\_{\\text{b}}$ | 98.2 | 96.4 | 94.2 | 83.0 | 88.2 |\n| $\\mathcal{T}\\_{\\text{final}}$ | 98.4 | 97.0 | 94.9 | 83.2 | 88.9 |\n\n### **Question 3: The use of the anchored endpoints to initialize the future ego feature.**\n\n- **Explanation**: As described in Line 202 of main paper, the future BEV corresponds to the final planning step. Thus, we initialize the future ego feature using the anchored endpoints, allowing it to encode prior knowledge of the ego vehicle’s future state at the final planning step.\n\n- **Clarification**: Note that the future ego feature is not updated iteratively — only the current ego feature undergoes iterative refinement.\n\n- **Experiment**: For more extensive evaluation, we have now analyzed different initialization strategies for the future ego feature. As shown below, using anchored endpoints as a prior yields the best performance. We will add this in revision.\n\n| Initialization strategy | NC ↑ | DAC ↑ | TTC ↑ | EP ↑ | PDMS ↑ |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| Random | 98.3 | 96.9 | 94.5 | 82.8 | 88.6 |\n| Anchored trajectories | 98.3 | 96.8 | 94.0 | 83.7 | 88.7 |\n| Anchored endpoints (Origin) | 98.4 | 97.0 | 94.9 | 83.2 | 88.9 |\n\n\n>[1] DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving, CVPR 2025.\n\n>[2] End-to-end driving with online trajectory evaluation via bev world model, ICCV 2025."}, {"review_id": "yWV1HqmrR3", "reviewer": "Reviewer_bKKZ", "summary": "This paper presents SeerDrive, an end-to-end driving framework that integrates trajectory planning and future scene prediction via bidirectional modelling. Unlike traditional one-shot models that plan based solely on the current state, SeerDrive predicts future Bird's-Eye View (BEV) semantic features and injects them into the planning process, creating a feedback loop between scene evolution and ego-motion. Experiments on NAVSIM and nuScenes demonstrate significant gains over state-of-the-art methods in planning accuracy.", "questions": "* Can SeerDrive handle rare or high-entropy scenarios (for example crowded intersections)? Have the authors tested the model on edge cases beyond NAVSIM and nuScenes?\n\n* Could we quantify the predictive accuracy of the future BEV semantic maps directly? This would help validate that the BEV predictions are not just beneficial but also semantically plausible.\n\n* What is the wall-clock inference time per iteration? Could this approach be deployed real-time?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### __Strengths__\n\n1. The paper demonstrates consistent improvements over competitive baselines on NAVSIM and nuScenes, with robust closed-loop metrics.\n\n2. The use of BEV maps rather than raw pixels for future prediction, offering a more efficient and interpretable modelling space.\n\n3. Good ablations on the future aware planning module, and the iterative refinement.\n\n### __Weaknesses__\n1. While used as an intermediate representation, the paper does not thoroughly evaluate or visualise the quality of the future BEV predictions in isolation (e.g. with intersection-over-union).\n\n2. The results are presented on NAVSIM and nuScenes, but it remains unclear how well SeerDrive generalises to longer horizons or more complex urban settings (e.g. large occlusions, heavy traffic).\n\n3. The model does not clearly demonstrate that more iterations lead to better representations or insights, especially since only 2 iterations are optimal, and going to 3 harms performance.", "was_revised": true, "final_justification": "After carefully reading the other reviews and the rebuttal, I've decided to increase my rating to borderline accept.", "rebuttal": "We thank the reviewer for the detailed review and the suggestions for improvement. Below are our responses to the reviewer’s comments:\n\n### **Weakness 1 and Question 2: Evaluation and visualization of future BEV prediction quality.**\nGreat suggestion! For visualization, we have already presented the predicted future BEV maps in Figure 3 of main paper, as well as in Figures 1 and 2 of appendix. Regarding the quality, we have now measured the mIoU of the predicted future BEV map on the NAVSIM dataset, standing at 38.87. We will include this result in the revision.\n\n### **Weakness 2 and Question 1: Testing on rare or high-entropy scenarios, longer horizons, complex urban environments, and edge cases beyond NAVSIM and nuScenes.**\n\n- **Experiment on complex scenarios**: Thanks. To conduct the suggested test, we test our model using the scenarios from the test split of NAVSIM with more than 10 agents. As shown in the table below, our model can still achieve strong performance. We have provided a visualization in Figure 1(b) of appendix and will add more in the revision.\n\n|   | NC ↑ | DAC ↑ | TTC ↑ | EP ↑ | PDMS ↑ |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| Complex scenarios | 98.2 | 96.8 | 94.3 | 83.1 | 88.5 |\n\n- **Experiment beyond NAVSIM and nuScenes**: For more extensive evaluation, we have further evaluated the more complex Bench2Drive [1] dataset with many rare and high-entropy scenarios such as emergency braking, merging, and overtaking, along with a longer planning horizon and closed-loop testing. As shown in the table below, our model consistently achieves the best performance. We will add this test.\n\n| Method                                   | Open-loop Avg. L2 Error ↓ | Closed-loop Driving Score ↑ | Closed-loop Success Rate (%) ↑ |\n|------------------------------------------|:----------------------------:|:------------------------------:|:----------------------------------:|\n| VAD [7]                  | 0.91                       | 42.35                        | 15.00                            |\n| UniAD-Tiny [8]              | 0.80                       | 40.73                        | 13.18                            |\n| UniAD-Base [8]              | 0.73                       | 45.81                        | 16.46                            |\n| MomAD [9]              | 0.82                       | 47.91                        | 18.11                            |\n| BridgeAD [10]              | 0.71                       | 50.06                        | 22.73                            |\n| **SeerDrive (Ours)**                     | **0.66**                   | **58.32**                    | **30.17**                        |\n\n### **Weakness 3: Iteration number and its impact on performance.**\n\n- **Explanation**: Our model design is, in fact, generic and not tied to a specific optimal iteration count. The ideal number of iterations largely depends on the characteristics and complexity of the target scenario. While more iterations generally lead to better representations, an excessive number can introduce computational burden and uncertainty. This doesn't conflict with our design; instead, our iterative architecture offers the flexibility to improve results by adding more iterations as scene complexity demands.\n\n- **Experiment**:  For instance, 2 iterations proved sufficient for NAVSIM given its comparatively lower complexity. However, for the more intricate Bench2Drive [6] dataset, our ablation study (see table below) clearly shows that 3 iterations yield superior results. This demonstrates the adaptability of our approach.\n\n| Iteration                                   | Open-loop Avg. L2 Error ↓ | Closed-loop Driving Score ↑ | Closed-loop Success Rate (%) ↑ |\n|:------------------------------------------:|:----------------------------:|:------------------------------:|:----------------------------------:|\n| 2                 | 0.74                       | 55.96                        | 27.27                            |\n| 3                 | 0.66                   | 58.32                    | 30.17                        |\n| 4                 | 0.68                       | 58.63                        | 29.98                            |\n\nUltimately, our framework allows for a flexible trade-off between accuracy and efficiency, letting users decide the optimal balance based on their specific needs and the complexity of the environment they're working with. We will discuss this in the revision.\n\n### **Question 3: Inference time and suitability for real-time deployment.**\n\nThanks and we will discuss this in revision: \nOur model achieves an average inference time of 24 ms on the NAVSIM dataset, comparable to DiffusionDrive [1] — a lightweight and deployable diffusion-based model. Meanwhile, our model outperforms DiffusionDrive [1] in terms of planning performance, indicating its suitability for real-time deployment with superior effectiveness.\n\n>[1] DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving, CVPR 2025.\n\n>[6] Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving, NeurIPS 2024.\n\n>[7] VAD: Vectorized Scene Representation for Efficient Autonomous Driving, ICCV 2023.\n\n>[8] Planning-oriented Autonomous Driving, CVPR 2023.\n\n>[9] Don't Shake the Wheel: Momentum-Aware Planning in End-to-End Autonomous Driving, CVPR 2025.\n\n>[10] Bridging Past and Future: End-to-End Autonomous Driving with Historical Prediction and Planning, CVPR 2025."}]

Author rebuttal:

Meta-review: The pre-rebuttal recommendation for this paper were rather divergent: 2 x Borderline Reject (`W3tz`, `bKKZ`), and 2 x Borderline Accept (`LXsM`, `TxmL`)  

The main concerns expressed by the reviewers were the following:
- limited performance improvements
- novelty w.r.t. prior methods, e.g., WOTE
- some limited studies on quality of BEV maps, prediction of intermediate BEV maps, tuning of the loss weights, number of iterations
- unclear generalization to longer horizons and complex urban settings,

The authors have submitted detailed responses to address some of the concerns expressed by the authors in particular regarding novelty, computational complexity and implementation details. In addition, new closed-loop results on Bench2Drive with longer and more complex scenarii, show performance improvements over different baselines.
After the rebuttal reviewer `LXsM` has increased their score to Accept and reviewer  `bKKZ` to Borderline Accept. 

Overall the reviewers appreciate the novelty of the contribution, bringing a fresh perspective in the area, the clean implementation, the consistent performance improvements and the ablation studies provided.

The meta-reviewer agrees with the assessment made by the reviewers and recommend this submission for acceptance. 
This work has several merits, however the reviewers have pointed out a few directions of improvement. We encourage the authors to take into consideration the useful advice from the reviewers.
Decision: Accept (poster)