From Forecasting to Planning: Policy World Model for Collaborative State-Action Prediction
Authors: Zhida Zhao
Talas Fu
Yifan Wang
Lijun Wang
Huchuan Lu
Venue: NeurIPS 2025 poster ()
Keywords: World model
Autonomous driving
End-to-end planning
Primary area: deep_learning

Abstract: Despite remarkable progress in driving world models, their potential for autonomous systems remains largely untapped: the world models are mostly learned for world simulation and decoupled from trajectory planning. While recent efforts aim to unify world modeling and planning in a single framework, the synergistic facilitation mechanism of world modeling for planning still requires further exploration. In this work, we introduce a new driving paradigm named Policy World Model (PWM), which not only integrates world modeling and trajectory planning within a unified architecture, but is also able to benefit planning using the learned world knowledge through the proposed action-free future state forecasting scheme. Through collaborative state-action prediction, PWM can mimic the human-like anticipatory perception, yielding more reliable planning performance. To facilitate the efficiency of video forecasting, we further introduce a parallel token generation mechanism, equipped with a context-guided tokenizer and an adaptive dynamic focal loss. Despite utilizing only front camera input, our method matches or exceeds state-of-the-art approaches that rely on multi-view and multi-modal inputs.  Code will be released at https://github.com/6550Zhao/Policy-World-Model.

From Forecasting to Planning: Policy World Model for Collaborative State-Action Prediction
Zhida Zhao
Yifan Wang Lijun Wang' Dalian University of Technology (770153907, oyontalas]@maill dlut. edu, en {wyfan, 1jwang, Ihchuan @dllut @dlut. cecluut.ddu..c. cn
Talas Fu"
Huchuan Lu
Abstract
Despite remarkable progress in driving world models, their potential for au- tonomous systems remains largely untapped: the world models are mostly learned for world simulation and decoupled from trajectory planning. While recent efforts aim to unify world modeling and planning in a single framework, the synergistic facilitation mechanism of world modeling for planning still requires further explo- ration. In this work, we introduce new driving paradigm named Policy World Model (PWM), which not only integrates world modeling and trajectory planning within unified architecture, but is also able to benefit planning using the learned world knowledge through the proposed action- free future state forecasting scheme. Through collaborative state-action prediction, PWM can mimic the human-like anticipatory perception, yielding more reliable planning performance. To facilitate the efficiency of video forecasting, we further introduce a parallel token generation mechanism, equipped with context-guided tokenizer and adaptive dynamic fo- cal loss. Despite utilizing only front camera input, our method matches or exceeds state-of-the-ar approaches that rely multi-view and multi-modal inputs. Code will be released at http:://iitubccom//5//////////o World-Model.
Introduction
Driving world models have recently garnered growing research interest, due to their capacity to simulate future environmental states, enabling autonomous systems anticipate complex traffic dynamics and enhance cecisionnmmkinn safety [1, 2]. This trend is particularly evident in video-based paradigms, driven by the accessibility of large-scale video datasets and the breakthroughs of the video generation technique [3-5]. Despite remarkable progress having been achieved, existing world model based autonomous driving methods [6-10] mostly operate decoupled manner (Figure (a)), i.e., the world models aim to predict the next state and the associated reward but cannot directly perform trajectory planning. A separate policy model still required for perceiving and planning. As result, the full potential of the world model in autonomous driving is largely restricted [11].
Several concurrent works have made the initial attempt towards integrating both world modeling and planning in a unified autoregressive model [12, 11]. where interleaved image and action token sequences are generated through the next-token prediction manner. Although the two tasks are jointly learned in these works, they are still independently conducted (Figure (b)), where world modeling prioritizes high-fidelity next frame prediction with photo-realistic details conditioned on input actions, and trajectory planning is performed through an end-to-end mapping from the visual observation to output actions without explicitly leveraging the learned world model. As such, the world modeling and trajectory planning are only unified in terms of model architecture and prediction
"Equal contribution
t Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Observation
Observation
State Prediction
Observation
Future Forecasting
Video
Unified Model
Model
Planner
Policy World Model
Planning
Navigation Command
Navigation
Planning
Planning
Command
(a) Conventional world model
(b) Unified world modeling and planning
(c) The proposed policy world model
with planner
Figure I: Comparison of video world models for autonomous driving (a) Conventional video world models [6-10] typically serve as data engines for stimulation in pixel space. (b) Unified world
models [1]-13] perform video generation and planning as separate tasks. (c) Our proposed policy world model performs planning based on the learned world knowledge.
manner, while their synergistic mechanism remains under-explored. It is still unknown whether and how this unification can further benefit autonomous driving.
In light of the above observations, we propose a new driving world model which not only unifies world modeling and trajectory planning in cohesive architecture, but is also able leverage the learned world knowledge to enhance planning efficacy (Figure (c)). Therefore, we name our model the Policy World Model (PWM). When human drivers plan, they frequently imagine possible future environment states to anticipate potential hazards. To mimic this "Anticipatory Perception" maneuver. PWM performs trajectory planning with an action-free future forecasting framework. Specifically, PWM is pre-trained to acquire world modeling ability through auto-regressive video generation on unlabeled video sequences. During fine-tuning and inference, given current and historical video frames, it first generates textual description to understand the current environment and then rolls out plausible future states via video generation based on the learned world knowledge. The current action is finally predicted by considering the generated description and forecasted future states as multi-modal rationales. The above procedure is implemented by an end-to-end auto- regressive Transformer, ensuring seamless collaboration between perception, prediction, and planning. Compared to conventional world models, PWM acting as a driving policy can fully unleash its world knowledge to directly perform decision maaiiiinmmmking.... rather than relying on model-based policy searching. In addition, the action-free video forecasting framework eliminates dependency on action-labeled data, not only ensuring training scalability but also allowing more fiexible future state rolling out.
To improve the efficiency of video forecasting, we enable parallel generation of all tokens within a single frame, which allows video synthesis through next- feammframe prediction and substantially accelerates the forecasting process. For this purpose, we employ a compact image token representation (28 tokens per image via context-guided compression and decoding), ensuring both efficiency and coherent visual generation. To ensure video generation quality, we design a novel dynamic focal loss for training PMW to focus temporally varying image regions. The combination of the above innovations allows PWM to generate high-quality future video frames at reasonable computational overhead. We evaluate PWM on the popular benchmarks including nuScenes [14]. NAVSIM [15]. Notably, approach achieves strong performance and significantly reduces the average collision rate compared to state-of-the-arr methods nuScenes. Additionally, using only camera inputs, achieve PDMS performance comparable to state-of-the-an methods that utilize both camera and LiDAR data on the NAVSIM benchmark. These findings both enhance autonomous driving safety and underscore the potential of learning from video-based environmental representations.
The main contributions of this work can be summarized follows:
We propose Policy World Model, which unifies world modeling and trajectory planning, and more importantly, able to benefit planning by unleashing the learned world knowledge through action-free future forecasting.
We develop a parallel video forecasting approach that accelerates prediction while preserving visual coherence, supported by a context-guided tokenizer for compact representation and dynamic focal loss for emphasizing temporally varying regions.

---

Our method, relying solely on front camera input, performs on par with or even surpasses ofate-of-the-art multi-view and multi-modal driving approaches on widely used benchmarks, while achieving efficient visual generation results, supporting safe and efficient planning.
2
Related Work
End-to-end Autonomous Driving. End-to-end autonomous driving has advanced remarkably rapidly, Modern systems [16-19] map raw sensor inputs directly to control actions, thereby simplifying the traditional pereeption-to-contee
pipeline. Early methods [20, 21] used structured bird's s-eye view (BEV) representations, while later work adopted dense 3D occupancy grids [22, 23] or sparse queries [24, 25] for richer, more efficient scene understanding, though handcrafted intermediate representations can limit generalization. More recently, autonomous driving research has begun to incorporate large language models (LLMs) [26, 27] and multimodal LLMs (MLLMs) [28-30]: driving scenes can be encoded as text for reasoning, achieving strong results in simulators [31-33], while video-based approaches leverage them to predict actions and answer queries [34-37].
Generative World Models. World models enable agents to understand and predict future environ- ments from experience [38-44]. In autonomous driving, various approaches have been proposed for world model construction. Bevworld [45] and WoTE [46] operate in the BEV latent space, while Drive-OceWii World [10] predicts future occupancy states to interact with planning modules. For video generation, generative world models synthesize controllable and high-quality driving scenes, typically through large-scale pretraining. These models fall into two main categories: diffusion- and Srtreessive.Dfffusion based methods include DriveDreamer [4] for controllable video genera- tion, Vista [8] for large-scale conditional synthesis, GLAD [47] for arbitrary-length videos, and DriveDreamer-2 [5] as well as Drive-WM [48] for multi-view generation. Autoregressive models such as GATA-1/2 [7, 49] unify frames, text, and actions into a single sequence for multi-condition control and long-horizon prediction, while InfinitYDrive [50] and DrivingWorld [51] extend to longer and continuous videos. However, token-by token-by-tolee by-token prediction makes these models computationally expensive and significantly less practical for efficiency-criticee downstream tasks.
Unified Generation and Understanding Models. Building on the rapid progress of LLMs and MLLMs, recent research increasingly emphasizes unifying both visual understanding and generative capabilities within a single MILM [52-54], often by carefully aligning the representations from large pretrained visual encoders with the embedding space of LLMs. Pioneering unified multimodal models (UMMs) [55-57] further extend this integration through powerful autoregressive or diffusion-basse modeling paradigms. Such holistic unification provides more principled and scalable framework to externalize the implicit world knowledge encoded in pretrained MILMs, thereby significantly enhancing their potential for comprehensive world modeling and downstream reasoning.
3 Method
We present Policy World Model (PWM), a unified model that integrates both world modeling and trajectory planning under a cohesive framework. Given historical video frames, PWM is able to stimulate future states by generating plausible future video frames in an action-free manner. Serving as a policy model, PWM can leverage its learned world knowledge to explicitly benefit planning via state-action collaborative prediction scheme. We implement PWM using end-to-end Transformer with an image tokenizer (See Figure 2), where the tokenizer is responsible for encoding input frames into a compressed token space with context guidance, and the Transformer learns to jointly represent the world and perform planning through rugo-regressive prediction. In the following, we present the model and training details of the tokenizer as well as PWM in Sec. 3.1 and Sce. 3.2, respectively.
3.1 Image Tokenizer with Context-Guided Compression and Decoding
Most driving world models [11, 7, 49] prioritize the quality of the video generation. order to capture high-resolution details, they typically adopt a long sequence of tokens to represent a single image, leading to significant computational and memory overhead during autoregressive generation. In contrast, PWM focuses using the learned world representation to benefit planning rather than pursuing photo-realistic details. To achieve balance between generation quality and efficiency, we employ new image tokenizer with context-guided compression and decoding.
3

---

Historical takens
Frozen
Nevigation/Egg Eggo Status
Text (Optional)
Trajectory D
Trainable
Future tokens
Frocasting
Action head
Encoder
2g
Decoder
Policy World
Encoder
Daw
Decoder
Action Queries
lww
Input
(a) Overview of the Policy World Model
(b) Dual-branch Image Tokenizer
Figure 2: Policy World Model Unified Forecasting and Planning. (a) PWM leverages its pre-trained world modeling generate future frames, enabling seamless collaboration between perception, prediction, and planning.(b) Future video frames IN are compressed into compact latent representations guided by initial frame I and [ represents the reconstructed frames.
Given a video clip I containing N frames, we aim to tokenize them into a compressed latent space by exploring their temporal consistency. To this end, we design the tokenizer two-branch encoder-decoder structure as depicted Figure 2 (b). To capture high-fidelity visual content, the high-resolution branch Qn (blue blocks) takes as input the initial frame I with full resolution. The encoder maps the initial frame into sequence L tokens Z₁ = [z], zf....,,2! further decoded back into the image space by the decoder. The encoder and decoder also produce a hicrarchy of multi-scale intermediate feature maps F. and Fa, respectively, which are used as guidance to facilitate token compression. In-parallel the high-resolution branch, the low-resolution branch Qi (orange blocks) will take all the downsampled frames a batch as input and performs tokenization and reconstruction for each frame independently. For frame L, compressed token sequence D, {d], di ...., dd } will be generated by the encoder, with L' << L. To transfer guidance information F. and Fa from the high-res to low-res branch, the encoder and decoder of the two branches are laterally connected via Pross-attentin layers. With the help of these guidance, the cow-resolution branch is able to effectively represent and reconstruct the input frames using significantly reduced number of image tokens.
) which is
In our experiment, we adopt pre-trained image tokenizer [55] as the high-resolution branch Qh- which is freezed during training. The low-resolution branch Qi implements learnable adaptation network mirroring the structure of Qn, augmented with additional random initialized cross-attention and downsampling blocks. The encoder of Qi is able tokenize input frame of 128 224 resolution into a compact feature map of x 7, giving rise to L' = 28 tokens per frame. We follow the standard VQ-GAN optimization strategy [58] to train the tokenizer. More implementation details are provided in the supplementary material.
3.2 Policy World Model
Most existing driving world models either solely focus world simulation [48, 6] or perform world representation trajectory planning as two separate tasks with a single model [1 1-13]. J. In contrast, our PWM serves as more cohesive paradigm, which not only integrates future state forecasting and planning within unified architecture, but also ensures the learned world modeling and forecasting ability explicitly benefit trajectory planning via a collaborative state-action prediction scheme. This is achieved by the following two unique techniques.
Learning World Modeling from Action-Free Video Generation. Most previous autonomous driving methods learn the video world model via action-conditioned video generation, which highly dependent on labeled training data. More importantly, they fail to perform any forecasting before the action predicted. Therefore, when the world model is integrated with the policy model in unified structure [1 [11], their future forecasting ability cannot be fully explored to benefit planning. To circumvent this issue, we propose to pretrain PWM on action-free video generation (Figure (a)). Formally, given video clip I1,,, first transform it into image tokens (Z,, D1.N } using the

---

arD, D2)
Dynamically Weight
Dynamically Weight
Des
Next Frame Prediciton
Policy World Model
Dg
Zyn
Dana
Dan2
Dan
Attention Mask
(a) World Modeling via Action-free Video Generation
(b) Token Generation & Attention
Figure 3: Pipeline for video world modeling. (a) World modeling is conducted action-free, highly compressed video data using dynamically enhanced parallel prediction. (b) Comparison of token prediction formats and attention interactions.
tokenizer in Sec, 3.1, where Z, = ( [zi,2i....,44 denotes the contextual tokens of the initial frame, and D, = {d] ... d denote the compressed tokens of the 1-th frame. PWM is then trained to generate these video token sequence an autoregressive manner. Thanks to the token compression scheme, PWM can simultaneously generate all the tokens of a image in-parallel, allowing us to replace the conventional next-token generation with the more efficient next-frame generation scheme (See Figure 3 (b)):
P(D I:N; Zi
II Po Zt), I
where 8 denotes the PWM parameters. To better model the spatial relationship between tokens of the same frame, employ the bi-directional attention within one frame, with the attention mask illustrated in Figure (b).
During training, we empirically observed that significant proportion (up to 50%) of the frame tokens are unchanged across adjacent frames, which makes the model tend to predict static tokens, impeding its temporal dynamic modeling ability. To alleviate this issue, introduce a Dynamic Focal Loss (DFL) that emphasize temporally varying image regions through spatial weighting. Specifically, define dynamic weighting function w(d), d) that measures the contribution of each token prediction based on its temporal change:
w(d), i a I d:.i] Bi[d; d a>
where I[|] is the indicator function and a. 3 are hyperparameters that control the relative importance of dynamic versus static token predictions. The overall DFL for the t-th frame is then formulated as:
LDF = w(d d) log (d Dec: Zt),
1=1
This design encourages the model to allocate more attention to dynamic regions, thereby enhancing its ability to capture meaningful spatio-temporal variations across frames.
End-to-End Planning with Future State Forecasting, In order to explicitly benefit planning with the attained world modeling ability, PWM is further fine-tuned for end-to-end planning in an auto- regressive manner. To this end, each data sample for fine-tuning prepared a multimodal sequence (Di:,, E,, X, D1+1:+nn A1:m): }. As shown in Figure 2, the inputs to PWM include the image tokens of observed frames D1:t and the current navigation command with ego status E, It then learns to autoregressively predict the textual tokens X, describing the current scenario and plausible future frame tokens D,+1ittn using its pre-trained world modeling ability as Eq.(3). Subsequently, m learnable action tokens are fed into PWM, and interact with the predicted latent features of the textual description and future forecasts. Their outputs are decoded into trajectory coordinates A 1:m by lightweight action head. For training, X, and D D1414tn supervised with cross-ent tross-entropy loss, and
5

---

Aim with L1 loss. Through the above collaborative state-action prediction scheme, PWM is able to explicitly leverage the forecasted future states, yielding more reliable trajectory planning.
Discussion of Collaborative State-action Prediction, Our use of collaborative denotes a causal and knowledge-sharing relationship within PWM, leveraging learned world knowledge better facilitate action prediction. This differs from prior methods that build world model P w (Dz+1:tt+ Drrr) and policy model Pe (a(A 1m (D.te) seperately. This is also disinct from recent attempts that integrate both world modeling and planning in unified architecture Pa D D.:e), where future frames and actions are still predicted independently via Po Po(D+11:++ D1:e) and PA D1:{) in downstream tasks, and knowledge is not explicitly shared between the two processes. In comparison, our PWM not only integrates world modeling and planning in a unified model A. but also unleashes the learned world knowledge through the proposed planning with future state forecasting scheme which can be expressed as:
1o D1.) Po(A 1.m D1s4n) - Po(D A Almm Du:e)
As shown in (4), world modeling and planning are not performed independently but in collabo- rative manner, allowing the PWM to mimic the human-like anticipatory ability based on the learned world knowledge and perform more reliable planning.
4 Experiment
4.1 Experimental Setups
Datasets. We adopt OpenDV-YouTube dataset [3], nuScenes [14], and NAVSIM [15] as main training datasets. OpenDV- YoenDV-YouTubee Tubeube contains 1747 hours of front-camera video at 10 Hz from 244 cities with scene description generated by BLIP2 [59]. nuScenes includes 1000 scenes (5.5 hours in total) with six camera views per scene, and is annotated with six 3-second waypoints and text descriptions [60]. NAVSIM built upon OpenScene [61], including 103k training and 12k test samples. We train the tokenizer on the the OpenDV-YouTube. YouTube. The PWM model is pretrained on OpenDV-YouTiue action-free video generation and then finetuned on nuScenes and NAVSIM for planning. We use only front-view camera video, with 10 Hz for OpenDV-You'Tute and NAVSIM, and 12 Hz for nuScenes.
for
Metrics. To assess the quality and consistency of future video generation. we employ three standard video synthesis metrics: FVD [62]. LPIPS [63]. and PSNR [64]. For the planning tasks, we use L2 error (m) and collision rate (%) on nuScenes, and Predictive Driver Model Score (PDMS) on NAVSIM. PDMS is composite metric designed to better align with closed-loop behavior, which combines five sub-scores: no-at-faull fault collisions (NC), drivable area compliance (DAC), -ome-to-oollisonn (TTC), comfort (Comf.), and ego progress (EP).
Implementation Details. Our model is initialized from Show-o [55]. During tokenizer training, the frozen branch processes 256 x 448 images to produce 448 tokens per frame, while the trainable branch uses 128 x 224 images, generating 28 tokens. Each branch has separate codebook of size 8192. We sample non-overlapping clips from the OpenDV-You You Tube dataset, using 1 % for validation and the rest for training. Training runs for 10 steps using Adam W on 6 NVIDIA A800 GPUs with a learning rate of 2 x 10-4 After training, the tokenizer is frozen. Next, we pre-train the autoregressive world model on OpenDV-YouTude for 3 10 steps using Adam with learning rate of 1 x 10-4, To prevent catastrophic forgetting, we also conduct auxiliary training on CC12M [65] and Web [66] for image captioning and text generation. For downstream tasks, we fine-tune the model using only the front-view camera. On nuScenes, we condition on I second of history to predict 11 frames and 6 waypoints over seconds. Training runs for 16 epochs on 2 A800 GPUs with batch size 8 and learning rate 3 x 10-5, On NAVSIM, we use 2 seconds of history to predict 10 frames and 8 waypoints across 4 seconds, training for 20 epochs with batch size 14. Dynamic Focal Loss is used throughout. No data augmentation is applied. More details are provided in the Appendix A.
4.2 Overall Comparison
On the nuScenes dataset, the relatively simple driving scenarios tend to induce an over-reliance on the vehicle's 's ego status [68, 74]. Therefore, in Table 1, we compare planning performance both without and with access to ego status across several popular methods. PWM achieves the lowest average collision rates (%) of 0.07 and 0.04 across two settings, respectively, surpassing previous

---

Table Comparison on the nuScenes validation split. Metrics are computed following the same protocol as [67]. For fair comparison, results are reported separately for settings without and with ego status (marked with "f"); results for UniAD and VAD are reproduced from BEV-Planner [68]. The best results are bolded.
L2(m)
Collision(
(6)
Method
Is
2s
3s
Avg.
2s
3s
Avg.
ST-P3 [20]
1.59
2.64
3.73
2.65
0.69
3.62
8.39
4.23
UniAD [21]
0.59
1.01
1.48
1.03
0.16
0.51
1.64
0.77
VAD-Base [67]
0.69
1.22
1.83
1.25
0,06
0.68
2.52
1.09
BEV-Planner [68]
0.30
0.52
0.83
0.55
0.10
0.37
1.30
0.59
Omni-Q [60]
1.15
1.96
2.84
1.98
0.80
3.12
7.46
3.79
LAW [69]
0.24
0.46
0.76
0.49
0.08
0.10
0.39
0.19
Drive-OeeWorld [10]
0.25
0.44
0.72
0.47
0.03
0.08
0.22
0.11
PWM(Ours)
0,41
0.75
1.17
0.78
0.01
0.01
0.18
0.07
UniADt [21]
0.20
0.42
0.75
0.46
0.02
0.25
0.84
0.37
VAD-Basef [67]
0.17
0.34
0.60
0.37
0.04
0.27
0.67
0.33
BEV-Planner [68]
0.16
0.32
0.57
0.35
0.00
0.29
0.73
0.34
OccWorld-Df [23]
0.39
0.73
1.18
0.77
0.11
0.19
0.67
0.32
Omni- Qf [60]
0.14
0.29
0.55
0.33
0.00
0.13
0.78
0.30
RDA-Driverf [70]
0.17
0.37
0.69
0.41
0.01
0.05
0.26
0.11
DiffusionDrivvft [17]
0.27
0.54
0.90
0.57
0.03
0.05
0.16
0.08
PWM(Ours)f
0.20
0.38
0.65
0.41
0.01
0.02
0.09
0.04
Table 2: NAVSIM NavTest split comparison. Overall Predictive Driver Model Score (PDMS) and sub-scores reflecting closed-loop performance, C: multi-view camera; SC: single-view camera; C&L: multi-view camera LiDAR; "-": visual input. The best results are bolded.
Method
Input
NC1
DACT
EPt
TTCT
Comf.
PDMS
Human
100.0
100.0
87.5
100.0
99.9
94.8
Constant Velocity
69.9
58.8
49.3
49.3
100.0
21.6
Ego Status MLP
93.0
77.3
62.8
83.6
100.0
65.6
VADv2 [18]
C&L
97.2
89.1
76.0
91.6
100.0
80,9
TransFuser [71]
C&L
97.7
92.8
79.2
92.8
100.0
84.0
DRAMA [72]
C&L
98.0
93.1
80.1
94.8
100.0
85.5
Hydra-MDP [19]
C&L
98.3
96.0
78.7
94.6
100.0
86.5
DiffusionDrive [17]
C&L
98.2
96.2
82.2
94.7
100.0
88.1
UniAD [21]
97.8
91.9
78.8
92.9
100.0
83.4
LTF [71]
C
97.4
92.8
79.0
92.4
100.0
83.8
PARA-Drive [73]
C
97.9
92.4
79.3
93.0
99.8
84.0
LAW [69]
C
96.4
95.4
81.7
88.7
99.9
84.6
DrivingGPT (11]
SC
98.9
90.7
79.7
94.9
95.6
82.4
PWM(OUrs)
SC
98.6
95.9
81.8
95.4
100.0
88.1
tftte-of-the-art models including Driv--OceWorla [10] and DiffusionDrive [17], with the former adopting a decoupled world modeling paradigm. On the more challenging NAVSIM dataset, as shown in Table 2, PWM delivers obvious superiority. Although rely only a single front camera view, our framework significantly outperforms prior camera-based models, including world modeling methods such DrivingGPT and LAW. It achieves PDMS score of 88.1, comparable to the state- of-the-art art method DiffusionDrive, which uses both camera and LiDAR inputs. Meanwhile, our model achieves higher time-to-colliste (TTC) score of 95.4 and no-at-fault collision (NC) score of 98.6.
4.3 Ablation Study
Impact of Action-free Video World Knowledge. We investigate the influence of learning from large-scale videos on planning performance Table and The first rows of the two tables show the results without pre-training, where models are fine-tuned the downstream task using the base model's weights [55]. All subsequent rows report the results after pre-training. shows that without pre-training, the model struggles to capture and predict dynamic scene changes and yields

---

Table 3: Impact of world modeling and dynamic focal loss on nuScenes and NAVSIM. "Pre- train" indicates training on Open- Youtube "ineo."Fine-iili FFine-tune indicates training on downstreaming benchmarks. "LDFL-p" and "DFi.." r indicate whether Dynamic Focal loss (DFL) was used Pre-train and Fine-tune, respectively. The video metrics and planning scores are reported.
(a) Ablation study on nuScenes Dataset.
Visual Forecast Quality PSNR1
Planning Metrics
Pre-train
CDIl
f
LPIPS|
FVD
Avg. (2m)) Avg.Col(%))
x
x
0.27
21.07
826.15
3.34
1.51
x
0.24
22.16
239.13
2,29
1.05
X
0.24
22.24
96.53
1.23
0.56
/
/
0.22
22.88
96.99
1.04
0.26
/
0.22
23.07
67.13
0.78
0.07
(b) Ablation study on NAVSIM Dataset.
Visual Forecast Quality PSNR
Planning Metries
Pre-train
LIFLUP
I.PISS.
FVD.
NC1
DACT
EP1
TTC
Comf.1
PDMS1
x
x
0.27
19.9
431.47
97.3
89.7
69.1
92.2
100.0
77.8
x
x
0.25
20.71
199.79
97.7
90.8
72.6
93.7
99.9
80.7
x
/
0.24
21.06
114.85
98.3
94.5
73.4
95.1
100.0
83.5
/
x
0.23
21.22
110.95
98.3
94.5
80.4
94.4
100.0
86.3
V
0.23
21.57
85.95
98.6
95.9
81.8
95.4
100.0
88.1
Trajectory
Figure 4: Decoded frame forecasting and corresponding BEV trajectory visualizations NVASIM (green: GT, orange: prediction).
the worst prediction and planning metrics. After pre-training on action-free video generation, the model significantly improves its ability predict future frames and achieves substantial gains in the planning task. In Table 5, we further compare the model's performance under different scales of video data on nuScenes. Without pre-training, adding the future-frame prediction task actually harms the model's planning capability, resulting in much worse performance than directly outputting trajectory waypoints. As the model exposed more data, from 0% to 50% and then to 100%, it progressively learns spatiotemporal modeling from driving videos, and the performance gap gradually narrows.
Effectiveness of Dynamic Focal Loss. We introduce Dynamic Focal Loss enhance the model's capability in capturing temporal dynamics for both video-frame prediction and planning. Tables 3a and 3b investigate the impact of applying this loss on prediction and planning performance for the nuScenes and NAVSIM datasets. Compared to omitting dynamic weight in both stages, applying in either pre-training or fine-tuning yields clear improvements in three generation metrics well as planning metrics. Notably, only using it during pre- pre-training achieves stronger results on LPIPS and PSNR than fine-tuning alone, indicating more effective spatiotemporal modeling from large- scale video. This also translates into improved planning. Applying the loss in both stages yields strongest improvements video prediction and planning, confirming that bolstering temporal dynamics synergistically enhances both capabilities. Additional evaluation metrics on the OpenDV. YouTube are reported in Table In Figure 4, we provide a qualitative visualization of the decoded future- frame predictions alongside the planned trajectories, illustrating their clear alignment.

---

Table 4: Ablation study on visual forcasting on nuScenes and NAVSIM benchmark.
(Fries))
Avg.1.2(m
nuScenes Avl(())(
Lateney
NCT
DACT
TTC1
NAVSIM
Comf.7
PDMST
Latency
0.50
0.13
0.88s
98.0
95.1
82.4
94.1
99.9
87.3
0,57x
0.82
0.10
1.1
98.4
95.4
81.5
94.8
100.0
87.7
0.691 121s
0.80
0.09
1,26 00.385
98.7
95.8
81.4
95.4
100.0
88.0
0.97) (0.40))
Table 5: Impact of different data scales of pre-training on nuScenes benchmark.
Table 6: Effect of dynamic focal loss on pre-training.
Usage
Avg.1.2(m)
Furecasted Frames 10
Avg. Avg.Col((.))]
Avg.1.2(m)]
Forecasted Frames =
(vl.Col((%)]
Con
LPIPS
Forecast Quality PSNR
FVDJ
2,95
1.26
1.62
0.23
20.29
50%
0.85
0.14
0.88
0.21
0.22
20.32
118.07
100%
0.78
0.07
0.80
1=2
6
1=8
1=10
GT
Wo DFL
DFL
GT
DFL
W DFL
Figure 5: Visualzation. Comparison of future frame predictions with and without Dynamic Focal Loss (DFL). The first row shows ground truth frames, the second row shows predictions without DFL, and the third row shows predictions with DFL. Sampled frames at t=2, 4, 6, 8, 10 are shown.
To further illustrate the impact of DFL, we also provide a visual comparison Figure 5. Each row corresponds to specific model configuration: the first row shows ground truth future frames, the second row presents prediction results without DFL, and the third row shows results with DFL. These visualizations demonstrate that the use of DFL helps the model better capture and represent dynamic scene elements over time, resulting in more accurate and temporally coherent predictions.
Influence of Video Forcasting on Planning. We evaluate the impact of forecasting and efficiency on downstream planning performance under different number of predicted future frames in Table Predicting 10 future frames achieves the best performance across both datasets. We speculate that shorter horizons capture insufficient temporal dynamics, resulting weaker planning. In contrast, longer horizons degrade prediction quality and may introduce hallucinations, especially given the limited perception from a single front-view camera, ultimately impairing decision-nin -aaiing. We evaluate frame token forecasting efficiency on single NVIDIA A800 GPU (batch size 1). As shown Table 4, the additional latency over the zero-horize baseline marginal, and the model achieves about 40 FPS without pixel-space decoding.

---

Predicted
Planning w forcasting
Planning wlo forcasting
Collision
Risky Behavior
Collision
Figure 6: Comparison of planning results with and without incorporating future prediction during training (green: GT, orange: prediction).
On the nuScenes benchmark, introducing future-frame prediction yields substantial reduction in average collision rate. On the more challenging NAVSIM dataset, we observe complementary trade-off: (i) when the model is trained without future prediction, it achieves a higher EP score, indicating that its planned trajectories advance farther along the route within the allotted horizon. By contrast, (ii) when the model does predict future frames during fine-tuning, it attains higher NC and TTC scores, demonstrating more effective avoidance potential collisions, and a higher DAC score, showing that its trajectories remain better confined to drivable areas. Furthermore, in Figure 6 we present a qualitative analysis of how future-frame prediction affects planning on challenging NAVSIM. The first column shows the current observation, the second shows the model's tenth fnth-frrmme prediction under forecasting fine-tuning, and the third and fourth columns respectively overlay the planned trajectories from the "with" and "without" prediction variants. From these findings, infer that future-frame forcasting can induce a more conservative planning strategy, sacrificing some progress (EP) in order to secure higher safety margins (NC and TTC). As result, the model favors lower-risk routes rather than maximizing forward progress.
5 Conclusion
In this paper, propose the Policy World Model (PWM), unified framework that integrates world modeling and trajectory planning for autonomous driving. By introducing action-free video generation and multi-modal reasoning. PWM forecast future scenes and make informed decisions without relying on action-labeled data. Our design significantly improves planning performance and efficiency, achieving competitive results using only monocular camera input. This work highlights the potential of using compact, anticipatory video- based representations to drive safer and more scalable autonomous systems.
Limitations and Future Work. Although video-based PWM demonstrates strong performance, relying solely on single-view inputs can compromise robustness under poor visibility conditions. Furthermore, its short planning horizon limits its applicability in horizoo scenarios. In future work, aim to further explore efficient integration of multi-view inputs and enhance long-term forecasting capabilities to improve generalization and real-world readiness.
6 Acknowledgement
This paper is supported by the National Natural Science Foundation of China (62422610, A23380 62441231, 62293542,, 62276045, 62576072), Liao Ning Science and Technology Plan No. No.2023JH26/1 02000062 Dalian Science and Technology Innovation Fund (CQ00)1 Natural Science Foundation Zhejiang (IDD25F020001), Ningbo Key R&D project (20257039), and Ningbo Science and Technology Innovation Project (2024Z294).
10

---

References
[1] Feng, Tuo, Wenguan Wang, Yi Yang. A survey of world models for autonomous driving. arXiv preprint arXiv:2501.1 //260, 2025.
[2] Guan, Yanchen, Haicheng Liao, Zhenning Li, et al. World models for autonomous driving: An initial survey. IEEE Transactions on Intelligent Vehicles, 2024.
[3] Yang, Jiazhi, Shenyuan Gao, Yihang Qiu, et al. Generalized predictive model for autonomous driving. In CVPR, pages 14662-14672. 2024.
[4] Wang, Xiaofeng, Zheng Zhu, Guan Huang, et al. Drivedreamer: Towards real-world-drive world models for autonomous driving. In ECCV, pages 55-72. 2024.
[5] Zhao, Guosheng, Xiaofeng Wang. Zheng Zhu, et al. Drivedreamer--2: LIm-enhanced world models for diverse driving video generation. In Toby Walsh, Julie Shah, Zico Kolter, eds., AAAI, pages (0412-1042)) 2025.
[6] Jia, Fan, Weixin Mao, Yingfei Liu, al. Adriver-i: A general world model for autonomous driving, arXiv preprint arXiv:2311.33544 2023.
[7] Hu, Anthony, Lloyd Russell, Hudson Yeo, et al. Gaia-l: A generative world model for autonomous driving. arXiv preprint ar3iv:2300. 17080, 2023.
[8] Gao, Shenyuan, Jiazhi Yang, Li Chen, al. Vista: A generalizable driving world model with high fidelity and versatile controllability. In NeurlPS. 2024.
[9] Wang, Yuqi, Ke Cheng, Jiawei He, et al. Drivingdojo dataset: Advancing interactive and knowledge- enriched driving world model. In NeurlPS, vol. 37, pages 13020-13034. 2024.
[10] Yang. Yu, Jianbiao Mei, Yukai Ma, et al. Driving in the occupancy world: Vision- centric 4d occupancy forecasting and planning via world models for autonomous driving. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, pages 9327-9335. 2025.
[11] Chen, Yuntao, Yuqi Wang. Zhaoxiang Zhang. Drivinggpt: Unifying driving world modeling and planning with multi-modal autoregressive transformers. arXiv preprint arXiv:2412. 18607, 2024.
[12] Zheng, Wenzhao, Zetian Xia, Yuanhui Huang, et al. Doe-1: Closed-loop autonomous driving with large world model. arXiv preprint arXiv:2412.09627, 2024.
[13] Bartoccioni, Florent, Flias Ramzi, Victor Besnier, al. Vavim and vavam: Autonomous driving through video generative modeling. arXiv preprint axXiv:2502.15672, 2025.
[14] Caesar, Holger, Varun Bankiti, Alex H. Lang, et al. nuscenes: A multimodal dataset for autonomous driving. In CVPR, pages 11618-11628. 2020.
[15] Dauner, Daniel. Marcel Hallgarten, Tianyu Li, et al. NAVSIM: data-driven non-reactive autonomous vehicle simulation and benchmarking. In NeurlPS, vol. 37, pages 28706-28719. 2024.
[16] Ren, Allen Z, Justin Lidard, Lars L Ankile, et al. Diffusion policy policy optimization. arXiv preprint 0tti::2409000888 2024.
[17] Liao, Bencheng, Shaoyu Chen, Haoran Yin, et al. Diffusiondrive: Truncated diffusion model for end-to-end autonomous driving. In CVPR. 2025.
[18] Chen, Shaoyu, Bo Jiang, Hao Gao, et al. Vadv2: End-to-end vectorized autonomous driving via probabilistic planning. arXiv preprint arXiv:2402. 13243, 2024.
[19] Li, Zhemxin, Kailin Li, Shihao Wang, et al. Hydra-mdp: End-to-end multimodal planning with multi-target hydra-distillation. arXiv preprint arXiv:2406.06977 2024.
[20] Hu, Shengchao, Li Chen, Penghao Wu, et al. St-p3: End-to-end vision-based autonomous driving via
spatial-temporal feature learning. In European Conference Computer Vision, pages 533-549. Springer, 2022.
[21] Hu. Yihan, Jiazhi Yang, Li Chen, et al. Planning-oriented autonomous driving, In CVPR, pages 17853- 17862. 2023.
[22] Tong, Wenwen, Chonghao Sima, Tai Wang, et al. Scene as occupancy. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 8406 8415. 2023.
11

---

[23] Zheng, Wenzhao, Weiliang Chen, Yuanhui Huang, et al. Occworld: Learning 3d occupancy world model for autonomous driving. In ECCV, vol. 15071. pages 55-72. 2024.
[24] Sun, Wenchao, Xuewu Lin, Yining Shi, et al. Sparsedrive: End-to-end autonomous driving via sparse scene representation. arXiv preprint arXiv:2405. 196200 2024.
[25] Zhang, Diankun, Guoan Wang, Runwen Zhu, et al. Sparsead: Sparse query-centric paradigm for efficient end-to-end autonomous driving. arXiv preprint arXiv:2404.0689 2024.
[26] Touvron, Hugo, Thibaut Lavril, Gautier Izacard, et al. Llama: Open and efficient foundation language models. arXiv preprini 2302::230211977.,, 2023.
[27] Yang, An, Anfeng Li, Baosong Yang. et al. Qwen3 technical report. arXiv preprint arXiv:2505.09350 2025.
[28] Liu, Haotian, Chunyuan Li, Qingyang Wu, et al. Visual instruction tuning. Advances in neural information processing systems, 36:34892-344 144488992399,,, 34916, 2023.
[29] Chen, Zhe, Weiyun Wang, Yue Cao, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, arXiv preprint arXiv:24/2.05271, 2024.
[30] Bai, Shuai, Keqin Chen, Xuejing Liu, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 2025.
[31] Shao, Hao, Yuxuan Hu, Letian Wang. et al. Lmdrive: Closed-loop end-to-end driving with large language models. In CVPR, pages 15120-155 15130. 2024.
[32] Zhang, Zaibin, Shiyu Tang, Yuanhang Zhang, ct al. Ad-h: Autonomous driving with hicrarchical agents. arXiv preprint 2rXiv:2406.034777 2024.
[33] Liu, Wenru, Pei Liu, Jun Ma. Dsdrive: Distilling large language model for lightweight end-to-end autonomous driving with unified reasoning and planning. arXiv preprint 0rXiv:2505.05366 2025.
[34] Yuan, Jianhao, Shuyang Sun, Daniel Omeiza, al. Rag-driver: Generalisable driving explanations with reumentdl- in-context learning in multi-modal large language model. arXiv preprint arXiv:2402. 10828, 2024.
[35] Sima, Chonghao, Katrin Renz, Kashyap Chitta, et al. Drivelm: Driving with graph visual question answering. In ECCV, vol. 15110, pages 256-274, 2024.
[36] Xu, Zhenhua, Yujia Zhang, Faze Xie, al. Drivegpt4: Interpretable end-to-end autonomous driving via large language model. RAI,, 9(10):188--88 88-881966113,1 2024.
[37] Chen, Long, Oleg Sinavski, Jan Hünermann, et al. Driving with Ilms: Fusing object-level vector modality for explainable autonomous driving. In 2024 IEEE International Conference on Robotics and Automation (ICRA). pages 14093 14100. IEEE, 2024.
[38] Ha, David, Jürgen Schmidhuber. World models. arXiv preprint orXiv:1803. 10122, 2018.
[39] Hafner, Danijar, Timothy Lillicrap. Mohammad Norouzi, et al. Mastering atari with discrete world models. arXiv preprint arXiv.2010.02/93, 2020.
[40] Singer, Uriel, Adam Polyak, Thomas Hayes, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209. 14792, 2022.
[41] He, Yingging, Tianyu Yang, Yong Zhang, et al. Latent video diffusion models for high-fidelity long video generation. arXiv preprint arXiv:2211. 132211, 2022.
[42] Zhou, Daquan, Weimin Wang, Hanshu Yan, et al. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint 12 2022.
[43] Khachatryan, Levon, Andranik Movsisyan, Vahram Tadevosyan, et al. Text2video-zero: Text-to-image -matt diffusion models are zero-shot video generators. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15954-15964. 2023.
[44] Liu. Yixin, Kai Zhang. Yuan Li, et al. Sora: A review on background, technology, limitations. and opportunities of large vision models. arXiv preprint arXiv:2402.17117 2024.
[45] Zhang, Yumeng, Shi Gong, Kaixin Xiong, et al. Beyworld: A multimodal world model for autonomous driving via unified bev latent space. arXiv preprint arXiv:2407.05679 2024.
12

---

[46] Li. Yingyan, Yuqi Wang. Yang Liu, et al. End-to-end driving with online trajectory evaluation via bev world model. In ICCV. 2025.
[47] Xie, Bin, Yingfei Liu, Tiancai Wang, et al. Glad: A streaming scene generator for autonomous driving. arXiv preprint arxiv:2003. 00004,, 2025.
[48] Wang, Yuqi. Jiawei He, Lue Fan, al. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. In CVPR, pages 14749-144 -9499914999.. 2024.
[49] Russell, Lloyd, Anthony Hu, Lorenzo Bertoni, et al. Gaia-2: A controllable multi-view generative world model for autonomous driving. arXiv preprint arXiv:2503.2053: 2025.
[50] Guo, Xi, Chenjing Ding, Haoxuan Dou, al. Infinitydrive: Breaking time limits in driving world models. arXiv preprint arXiv:2412.01522 2024.
[51] Hu, Xiaotao, Wei Yin, Mingkai Jia, et al. Drivingworld: Constructingworrld model for autonomous driving via video gpt. arXiv preprint arx::224 19505, 2024.
[52] Li, Hao, Changyao Tian, Jie Shao, et al. Synergen-vl: Towards synergistic image understanding and generation with vision experts and token folding. In Proceedings of the Computer Vision and Patern Recognition Conference, pages 29767-29779. 2025.
[53] Dong, Runpei, Chunrui Han, Yuang Peng. et al. Dreamllm: Synergistic multimodal comprehension and creation. arXiv preprint 2309 2023.
[54] Tong, Shenghang, David Fan, Jiachen Zhu, et al. Mctamorph: Multimodal understanding and generation via instruction tuning. arXiv preprint 2rX1v:241..14164, 2024.
[55] Xie, Jinheng, Weijia Mao, Zechen Bai, et al. Show-o: One single transformer to unify multimodal understanding and generation. In ICLR. 2025.
[56] Team, Chameleon. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.098/ 0,9188 2024.
[57] Zhou, Chunting, Lili Yu, Arun Babu, et al. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408. 1039, 2024.
[58] Esser, Patrick, Robin Rombach, Bjorn Ommer. Taming transformers for high-rcsolution image synthesis. In CVPR, pages 12873-12883. 2021.
[59] Li. Junnan, Dongxu Li, Silvio Savarese, et al. Blip-2: Bootstrapping language- image pre- traininii with frozen image encoders and large language models. In ICML, pages 19730-19799 19742, PMLR, 2023.
[60] Wang, Shihao, Zhiding Yu, Xiaohui Jiang, et al. Omnidrive: A holistic Ilm-agent framework for autonomous driving with 3d perception, reasoning and planning. In CVPR. 2025.
[61] Peng. Songyou, Kyle Genova, Chiyu Jiang, et al. Openscene: 3d scene understanding with open vocabular- ies. In CVPR, pages 815-824. 2023.
[62] Unterthiner, Thomas, Sjoerd Van Steenkiste, Karol Kurach, et al. Towards accurate generative models of video; A new metric challenges. arXiv preprint 11Xiv:18/2.01117,, 2018.
[63] Zhang, Richard, Phillip Isola, Alexei Efros, et al. The unreasonable effectiveness of deep features as a perceptual metrie. In CVPR, pages 586-595.10 2018.
[64] Huynh- Huynh-Thu, Quan, Mohammed Ghanbari. Scope of validity of psnr in image/vidco quality assessment. Electronics letters, 80000800,,-, 2008
[65] Changpinyo, Soravit, Piyush Sharma, Nan Ding, et al. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, pages 3558-3568. 2021.
[66] Penedo, Guilherme, Hynek Kydlícek, Loubna Ben Allal, et al. The fineweb datasets: Decanting the web for the finest text data at scale. In NeurlPS, vol. 37, pages 30811 10811-30849, 2024.
[67] Jiang, Bo, Shaoyu Chen, Qing Xu, et al. Vad: Vectorized scene representation for efficient autonomous driving. In CVPR, pages 3340-8350. 2023.
[68] Li. Zhiqi, Zhiding Yu, Shiyi Lan, et al. Is ego status all you need for open-loop end- and-to-end autonomous driving? In CVPR, pages 14864 148441447...00 2024.
13

---

[69] Li, Yingyan, Lue Fan, Jiawei He, et al. Enhancing end-to-end autonomous driving with latent world model. In ICLR. 2025.
[70] Huang. Zhijian, Tao Tang. Shaoxiang Chen, et al. Making large language models hetter planners with crasoning-decisoon alignment. In ECCV, pages 73-90. 2024.
[71] Chitta, Kashyap, Aditya Prakash, Bernhard Jaeger, et al. Transfuser: Imitation with transformer-based sensor fusion for autonomous driving. IEEE TPAMI, 45(11):12878-1289, 2022.
[72] Yuan, Chengran, Zhanqi Zhang, Jiawei Sun, al. Drama: An efficient end-to-end motion planner for autonomous driving with mamba. arXiv preprint arXiv:2408.0m 03600, 2024.
[73] Weng, Xinshuo, Boris Ivanovic, Yan Wang, et al. Para-drive: Parallelized architecture for real-time autonomous driving. In CVPR, pages 15449-15458. 2024.
[74] Zhai, Jiang-Tian Ze Feng, Jinhao Du, et al. Rethinking the open-loop evaluation of end-to-end autonomous driving in auscenes. arXiv preprint Xxii:2305.10430 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The contribution and scope of the paper accurately reflected in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification:
We discussed the limitations of work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

We provide complete set of hypotheses and complete proofs for each theoretical result.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide all the information to enable readers to reproduce our method, and the code will also be provided.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide full implementation information as well as code and metadata to faithfully reproduce our method and experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
Moolli//giies/Codeeeee
onPolicy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / wttpp.ccpppplliipppppppppppppppp wwtpccccppcccceececccccccccctttt cooollccguidess.
seabbmmmsssooPae ssionoolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have detailed all the training and test details required to understand the results in our paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: we have reported appropriate information about the statistical significance of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided detailed computational resource requirements to reproduce the experiment.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that NeurlPS Code of Ethics https : http://eerrisssceeee httpge///aarrrss.cccpbbb.//eeees cttppppcccehhihicccii.........e
didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer:
[Yes]
Justification:
All aspects of the article comply with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The potential social impacts discussed in paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The methods and datasets in our paper do not involve the risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have mentioned and cited the models and datasets used in the article.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datastss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The assets introduced in the article have sufficient documentation and provided together with the assets, which will be made public later.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not include any crowdsoureing experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not include any crowdsourcing experiments research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The core methods proposed in our paper does not use L.M..
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: httpp:://neurppa.s wwtpe///ee/eeee//////0/0/// for what should or should not be described.
21

---

Supplementary Material Overview
This supplementary material provides additional implementation details, experimental setups, and extended results to complement the main paper.
Section A describes architecture and training details of our image tokenizer and policy world model, including both pre-training and fine-tuning stages.
Section presents additional experiments on the nuScenes.
Section C provides visualizations of the latent predictions and qualitative comparisons.
A Implementation and Experimental Details
A.1 Image Tokenizer
Architecture details. We adopt dual-branch architecture to map input images of different resolutions into a shared latent space. The trainable branch, denoted as Q1, is initialized with the same structure as the frozen high-resolution branch Qn. To enhance its modeling capacity, we insert additional self-attention layers after the multi-level downsampling stages both the encoder and decoder. These layers are designed to better capture spatial relationships within the feature maps. Furthermore, we incorporate cross cross-attention layers to enable Qn, to guide the learning of Qt. This alleviates the burden on Qi to extract contextual information, allowing it to focus more modeling temporal variations and dynamic changes. Specifically, multi-scale features from Qi are used as queries, while the corresponding features from Qn, serve as keys and values. The attention is applied independently across scales. In the encoder of Qi, self-attention and cross-attention is applied at resolutions of introduce lightweight
8 x 14; in the decoder of Qt, it is applied at 8 x 14, 16 x 28. Additionally, MLP layer to perform 4x downsampling and upsampling of latent token sequence length from Q in the encoder and decoder, respectively. This serves to further compress and reconstruct the latent representations efficiently.
More training details. We sample the original YountbY dataset into non-overlapping clips, each containing 40 frames spanning 4 seconds. During training, each sample randomly selects a continuous 30-frame segment, from which 2 frames are randomly chosen high-resolution context frames. Subsequently, future frames are randomly sampled as low-re low-resolution video inputs. We optimize the model using the AdamW optimizer with 500 warm-up steps. Image reconstruction is supervised using the L1 loss. Since pixel-wise differences between future frames and initial context frames tend to increase over time, we apply time-dependent weighting the reconstruction loss, assigning greater emphasis to later frames to reflect their higher prediction difficulty and encourage better long-term modeling. We assign weight of 2.0 to the perceptual loss and weight of 1.0 the discriminator loss to balance perceptual quality and realism during optimization.
A.2 Policy World Model
A.2.1 Pre-training setup
Although we sample two consecutive nigh-resolution initial frames for the tokenizer, only one high- resolution frame is actually fed into the model per second without supervision. This strategy does not significantly affect the model performance, while effectively reducing the input token sequence length and improving training efficiency. Each video clip is randomly cropped into a 24-frame continuous segment from the original 40-frame clip to define the prediction horizon. As shown in Figure I(a), mark the beginning and end of each high-resolution frame sequence using two special tokens, "<lsoil>" " and "<lcoil>", following the Show-o convention. For the compressed low-resolution frames, we introduce two additional special tokens, "<lsodi>" and "<leodi>", to indicate the start and end positions of the low-resolution frames.
During training, the video autoregressive prediction task serves as the primary objective, while image-text captioning and pure language modeling tasks are incorporated to preserve the model's s capability in understanding and generating language. These three tasks are mixed in a batch ratio of 3:1:1, respectively. The loss weights are set 1.0 for the video task and 0.5 for both the captioning and text-only tasks. The warm-up steps are set to 1000.
22

---

Initial frame tokens
future frame tokens
planning
<|soi|p


</sod]p
</eod|>


(a) Details of frame tokens
(b) Details of action tokens
Figure 1: Detailed structure of input sequences. (a) illustrates the format of high-resolution and low-resolution video frames. (b) depicts the input configuration used for trajectory prediction.
A.2.2 Fine-tuning setup.
Details on nuScenes. We consider two configurations to investigate the influence of ego status on planning performance. Structurally, when ego status included as input, we use a two-layer MLP with SiLU activation to project the ego state into the model's s latent space. For navigation commands, we randomly initialize three learnable embeddings to represent "Go Straight," "Turn Left," and "Turn Right." The "<lacti>" token is used to prompt trajectory prediction, as illustrated in Figure 1(b). The action head implemented as an MLP with SiLU activation. Our framework is designed to seamlessly support multi-moda outputs, including both video and language. Given that nuScenes is one of the most widely used open-loop datasets and that several prior works have proposed textual annotations based on it, we adopt a simplified setting where the action description serves the target for language generation, using fixed prompts. For example: "In this quiet nighttime driving situation, the vehicle should maintain moderate speed and continue straight, adhering to lane discipline." This setup demonstrates the flexibility of our framework and lays the groundwork for future research to explore
expressive and diverse multi-moda outputs.
During training and evaluation, the last one second of each scene lacks future frames, which makes unsuitable for future frame prediction. Therefore, in the training set, we manually exclude these final segments to ensure supervision consistency. For the validation set, we also discard the last one second of each scene when evaluating video generation metrics. However, to ensure a fair comparison with previous works on planning, we retain these segments when computing planning-re reeltedd metrics.
Details on NAVSIM. We follow the official practice by concatenating the ego status and navigation command into single vector, which then projected into the model space via MLP layer. Since this dataset does not provide textual descriptions, we do not include any language modeling tasks during training or inference. Additionally, some video frames required for prediction are not included in the NAVSIM dataset. To address this, we resample the missing frames from the original nuPlan dataset. Other aspects of the training procedure remain largely consistent with that of nuScenes.
B
More experiments
B.1 Set up in Dynamic Focal Loss
To effectively improve the video modeling and generation capability of the Policy World Model, we propose Dynamic Focal Loss (DFL) that emphasizes temporally varying image regions through spatial weighting. We conduct an ablation study on the key hyperparameters and 3 on nuScenes, where a controls the weight for spatial tokens that change over time, and / controls the weight for those that remain unchanged across consecutive frames. As shown in Table we fix 1.0 and vary 3 to explore their relative influence.
We observe that when < B and / is set to small value (e.g., B == 0.1), the large disparity in task weights leads to overfitting in the future frame prediction task during training. while the planning task has not yet fully converged. This imbalance ultimately results in degraded overall performance, indicating the need to better coordinate the training progress the two tasks for more effective joint optimization. On the other hand, when < B, the Dynamic Focal Loss mechanism tends to fail, leading to significant drop the quality of future frame prediction, which in urrn negatively impacts the performance of the downstream planning task.
23

---

Table Ablation study on the hyperparameters the Dynamic Focal Loss. The dynamic weight is fixed to 1.0 across all experimental settings.
Visual Forecast Quality
Planning Metrics
LPIPS
PSNR
FVD4
Avg.L2(m))
Avg. Co)(
0.1
0.23
22.69
65.07
0.82
0.12
0.4
0.22
23.07
67.13
0.78
0.07
0.7
0.22
23.11
71.84
0.84
0.09
1.0
0.22
22.88
96.99
1.04
0.26
2.0
0.22
22.65
93.83
1.27
0.27
Table 2: Impact of the Textual Generation Task on nuScenes validation split. We also conduct an ablation study to evaluate the impact of the textual generation task on planning performance. Specifically, compare different settings: (1) "None": without incorporating any text generation task, (2)"S: with scene description prediction, and (3) "Action": with action description prediction.
Visual Forecast Quality
Planning Metrics
Text Task Type
LPIPS
PSNR
FVD4
Avg.1.2(m)
Avg.Col(%)
None
0.22
23.12
65.45
0.77
0.09
Scene
0.22
22.97
67.52
0.77
0.08
Action
0.22
23.07
67.13
0.78
0.07
B.2
Visualization and Analysis of Temporal Representations in Predicted Video Frames.
As shown in Figure 2, we visualize the 2D UMAP projection of forecasted latent video frames over time on the nuScenes validation split. Specifically, we extract future predicted driving latents from each 20-second- projected into 2D coordinate point according its temporal order. Different colors are used to indicate the temporal progression from Os to 20s.
scene and concatenate all samples across scenes. Each predicted frame is then
We observe that, although the future representations conditioned on different input frames, the resulting projected embeddings consistently exhibit smooth and coherent temporal dynamics across the full prediction horizon. This phenomenon suggests that our Policy World Model is capable of learning a robust internal representation of the temporal evolution of driving scenes. Importantly, also demonstrates that the model can effectively decouple the dynamics from specific visual content in the input, capturing underlying motion patterns that are consistent and semantically meaningful, regardless of the particular observation used as guidance.
generated independently for each sample and
B.3 Impact of the Textual Generation Task on Planning Performance.
To isolate the impact of different types of text prediction planning performance of nuScenes, we design three settings. The first setting excludes any text prediction task, which also serves as the baseline in the NAVSIM dataset. The second setting provides a detailed description of the scene environment, while the third offers brief prediction of the ego vehicle's future behavior. For action descriptions, we filter out information related multi-view perspectives and retain only brief summary supervision. As shown in Table 2, our model, incorporating scene- or action-level textual generation tasks does not lead to significant improvements in future frame forecasting downstream planning metrics, suggesting a limited effect in our specific setting.
Visualizations
C.1
Comparison visualization
In Figures 3 and 4, we provide additional qualitative results to compare planning with and without future frame prediction. On the left, we show sequence of video frames where red-bordered frame indicates the current observation, and the unframed ones predicted future frames. We

---

Scene-0035
Scene-0038
Scene-0039
Scene-0092
Scene-0102
Scene-0107
Scene-0277
Scane-0559
Scene-0623
Scene-0629
Scene-068 -6me-0030
Scene-0636
Scene-0783
Scane-0800
Scene-090S
Scene-0911
Scene-09
Scene- 0930
05
20s
Figure 2: The UMAP projection reveals temporal changes in frame-level latent representations, highlighting the model's 's ability to capture scene dynamics across different environments.
Real observation
Predicted future
Result w forecasting Result w/o forecasting
Figure More comparison of planning results with and without incorporating future prediction during training (green: GT, orange: prediction).
uniformly sample three future frames for visualization. On the right, we present the corresponding BEV (bird's-eep s-eye view) planning outcomes. These comparisons highlight how incorporating future frame prediction can enhance planning quality by enabling better anticipation of dynamic scene changes. In Figure 5, show additional visual comparisons illustrating the effect of Dynamic Focal Loss on future frame prediction.
25

---

Real observation
Predicted future
Result w forecasting
Result w/ forecasting
Figure 4: Visualzation. More comparison of planning results with and without incorporating future prediction during training (green: GT, orange: prediction).
26

---

10
GT
Wo DFL
W DFL
GT
Wo DFL
DFL
GT
DFL
DFL
GT
DFL
W DFL
Figure 5: Visualzation. More comparison of future frame predictions with and without Dynamic Focal Loss (DFL.. The first row shows ground truth frames, the second row shows predictions without DFL, and the third row shows predictions with DFL. Sampled frames at (=2, 4, 6, 8, 10 are shown.
27


Reviews:
[{"review_id": "5KGNOuks5L", "reviewer": "Reviewer_n8oM", "summary": "The paper introduces Policy World Model (PWM), an autoregressive Transformer that jointly learns a video‑based world model and performs trajectory planning, leveraging action‑free future‑frame generation instead of action‑labelled data. It adds a dual‑branch image tokenizer that compresses each frame to 28 tokens and enables frame‑level parallel prediction, plus a Dynamic Focal Loss to focus learning on temporally changing regions. Combined, these innovations let a single front‑camera model match or exceed multi‑sensor state‑of‑the‑art baselines on nuScenes and NAVSIM, while running at 40 FPS without pixel decoding.", "questions": "1. Compare with **Drive-OccWorld**[1] (the SOTA work also employs a world model for end-to-end planning) on motivation, theoretical analysis, and experimental performance. Clearly point out your novelty and contribution.\n2. Explain why PWM is ''**unified**''? In L181-188, future states are first obtained through **action-free** RGB frame prediction. Trajectories are then forecasted based on **predicted textual description and future forecasting results**. These two predictions are clearly **separate**. It's more like employing an additional decoder to **infer trajectories from videos**.\n3. Provide more ablation results **without extra data** (Open-YouTube)  pre-training as follows. Otherwise, there will be concerns that performance gains come from additional large-scale and more comprehensive training samples. \n| Pre-train      | L_DFL-p | L_DFL-f    |\n| :---        |    :----:   |          ---: |\n| $\\times$     | $\\checkmark$       | $\\checkmark$   |\n| $\\times$   | $\\checkmark$        | $\\times$      |\n| $\\times$     | $\\times$       | $\\checkmark$   |\n4. Compute and latency details. Please provide a comparison with SOTA works like LAW / DiffusionDrive / Drive-OccWorld on (i) full model size and (ii) end‑to‑end inference latency including token decoding.\n5. Failure‑case analysis. Provide qualitative or quantitative analysis of cases where future‑frame hallucinations mislead planning (e.g., poor weather, heavy occlusion) and give some possible solutions to further ensure safety.\n\n[1] Yang, Yu, et al. \"Driving in the occupancy world: Vision-centric 4d occupancy forecasting and planning via world models for autonomous driving.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 9. 2025.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. PWM is technically sound, with clear pre‑training → fine‑tuning stages, thorough ablations isolating each design choice, and consistent performance improvements across metrics and datasets. \n2. It cuts nuScenes collision rates to 0.07 % and reaches a PDMS of 88.1 on NAVSIM using only a monocular camera, outperforming or tying LiDAR‑ and multi‑view methods. \n3. The tokenizer and loss yield a 16 × token reduction and demonstrable speed‑accuracy gains, showing practical efficiency. \n4. The manuscript is well‑written and organized, making replication feasible. \n\nWeaknesses:\n1. Lacks analysis and comparison with the existing state-of-the-art policy world model **Drive-OccWorld**[1]. The originality of **unified prediction** remains uncertain.\n2. Despite the claim of a unified generation-planning framework, the prediction of trajectory and the prediction of future states are still independent. In L181-188, future states are first obtained through **action-free** RGB frame prediction. Trajectories are then forecasted based on **predicted textual description and future forecasting results**. These two predictions are clearly **separate**. It's more like employing an additional decoder to **infer trajectories from videos**.\n3. Extra data (Open-Youtube) used for training, raising concerns about the fairness of performance comparisons.\n4. The performance gain from textual generation is limited, which hurts the necessity of the multimodal component.\n5. While unification is useful, the method is incremental over prior autoregressive world‑model frameworks. It's more like employing an additional decoder to infer future trajectories from predicted videos.\n\n[1] Yang, Yu, et al. \"Driving in the occupancy world: Vision-centric 4d occupancy forecasting and planning via world models for autonomous driving.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 9. 2025.", "was_revised": true, "final_justification": "The authors addressed most of my concerns during the rebuttal phase. I believe the work is novel and provides valuable insights, but I remain concerned about the insufficient content on end-to-end planning and the lack of a clear explanation for collaborative state-action prediction (e.g., token interaction process).", "rebuttal": "We are glad that the strengths (technically sound, consistent performance improvements, thorough ablations, practical efficiency) of our work can be acknowledged by the reviewer. We thank the rveiwer for the constructive suggestions. Detailed responses are as follows.\n\n**W1/Q1: PWM vs. Drive-OccWorld**\n1. Motivation: \nDrive-OccWorld aims to design an independent occupancy-based world model to support control interactions. In contrast, our Policy World Model focuses on integrating the world model and the policy into a unified architecture, which can leverage its pretrained world knowledge to inform and guide decision-making. This integration enables planning to benefit directly from future predictions generated by the world model itself.\n\n2. Theoretical Analysis: \nAs shown in Fig.1-2 of the Drive-OccWorld paper, their occupancy prediction model and trajectory planner are two stand-alone model, which belongs to the type-(a) paradigm shown in our Fig.1 (a), albeit with occupancy grids instead of video inputs.\nOur approach aligns with the type-(c) paradigm in our Fig.1 (c), where the PWM model can not only simulate the future states as a world model but also acts as a policy model for decision based on the forcasted future. \n\n3. Experimental Performance: \nWe conduct additional experiments on nuScenes, where all methods are evaluated under the same settings: without ego-state inputs and using the BEV-Planner evaluation protocol. Drive-WM and OccWorld results are taken from the OccWorld paper.\nOccWorld achieves the best L2 error, possibly benefiting from its strategy of sampling multiple candidate trajectories, which could increase the likelihood of overlapping with the ground-truth. In contrast, our PWM predicts a single trajectory without candidate selection. Despite this, PWM achieves the lowest collision rate, indicating that our unified model produces safer and more consistent predictions.\n\n| Method       | Modality   |           | L2 (m) ↓                         |      |   |   | Collision (%) ↓                           |      |       |\n|:------------:|:----------:|:----------------------:|:-------------------:|:----:|:-----:|:------------------------:|:-------------------:|:----:|:-----:|\n|              |            | 1s                     | 2s                  | 3s   | avg   | 1s                       | 2s                  | 3s   | avg   |\n| Drive-WM [1] | video      | 0.43                   | 0.77                | 1.20 | 0.80  | 0.10                     | 0.21                | 0.48 | 0.26  |\n| OccWorld [2] | occupancy  | 0.25                   | 0.44                | 0.72 | 0.47  | 0.03                     | 0.08                | 0.22 | 0.11  |\n| PWM (Ours)   | video      | 0.41                   | 0.75                | 1.17 | 0.78  | 0.01                     | 0.01                | 0.18 | 0.07  |\n\n**W2/W5/Q2: Clarification on Our Unified Generation-Planning Framework.**\nWe appreciate the reviewer’s observation regarding the design of our generation-planning framework. However, we would like to clarify a key misunderstanding: in our unified framework, future state prediction and trajectory planning are not handled as independent stages nor via separate models. \nInstead, we enable the world model itself to act as an executor for planning, where both the prediction of video frames and the prediction of actions are accomplished by a single model. Specifically, as illustrated in Figure 2 of the paper, our lightweight action decoder does not infer trajectories directly from predicted videos. Instead, it serves to map the action embeddings output by the Policy World Model into a low-dimensional trajectory space. These action embeddings are generated conditioned on the model’s autoregressive prediction context, which includes predicted textual descriptions and future forecasting embeddings. \n    \n**W3: Concerns About Use of OpenDV-Youtube**\nSeveral existing methods also incorporate external datasets to enhance image generation or captioning capabilities before comparing against traditional baselines such as UniAD and VAD. For example, GenAD and VaVAM utilize the OpenDV-Youtube dataset, Doe-1 leverages BDD100k, and DrivingGPT includes data from the nuPlan dataset.\nWe use the OpenDV-Youtube dataset primarily to equip the model with future video prediction capabilities, which is a core component of our approach. This dataset is only used during the action-free pretraining phase and is not directly involved in optimizing planning performance. Meanwhile, the pretraining process for video generation training does not require annotations, which is a key advantage of our method in terms of annotation costs and scalibility.\nMoreover, we believe a key value of world models lies in their ability to model environmental dynamics in open-world scenarios. Achieving this goal naturally requires learning from rich and diverse visual data to capture temporal regularities, much like how language models benefit from large-scale pretraining. Our approach follows this trajectory to advance the development of planning-centric world models.\n    \n**W4: Concern regarding the necessity of the multimodal component.**\nThe direct performance gain from textual generation in our current implementation is indeed limited. However, textual generation is not intended for performance improvement but rather serves broader goals—aligning with trends (e.g., DriveLM, Omnidrive) using multimodal models to boost interpretability and human-AI interaction in autonomous driving.\nMotivated by these trends, our method includes a language generation component to demonstrate the model’s ability to produce high-level textual summaries of driving scenes and decision contexts, aiding explainability and future human-in-the-loop applications. We will provide more explanations in the final version.\n   \n**Q3: More ablation results without extra data (OpenDV-YouTube) pre-training.**\nWe would like to clarify the meaning of different configurations in Table 3. Specifically, “Pretrain” refers to training on the Open-Youtube dataset, while “L_DFL-p” and “L_DFL-f” denote whether the Dynamic Focal Loss is applied during the pretraining and fine-tuning phases.\nAccordingly, L_DFL-p becomes inapplicable without pretraining, as its application is predicated on the existence of a pretraining stage. Therefore, the first two configurations listed in Q3 (with Pretrain = ×, L_DFL-p = √) are not feasible in our framework.\nTo address the concern regarding the necessity of extra data, we provide an ablation where we disable pretraining (Pretrain = ×, L_DFL-p = ×) and only apply L_DFL-f (L_DFL-f = √) during fine-tuning on the nuScenes dataset. This configuration shows that even without large-scale pretraining, applying L_DFL during fine-tuning alone improves future frame prediction and leads to measurable gains in planning performance. These results support the effectiveness of our loss design independent of external data scale.\n  \n| Pre-train | L_DFL-p | L_DFL-f | LPIPS ↓ | PSNR ↑ | FVD ↓  | Avg.L2 (m) ↓ | Avg.Col (%) ↓ |\n|:---------:|:-------:|:-------:|:-------:|:------:|:------:|:-------------:|:---------------:|\n| ✗         | ✓       | ✓       |   --    |  --    |  --    |      --       |       --        |\n| ✗         | ✓       | ✗       |   --    |  --    |  --    |      --       |       --        |\n| ✗         | ✗       | ✓       |  0.26   | 21.14  | 514.88 |     2.95      |      1.26       |\n| ✗         | ✗       | ✗       |  0.27   | 21.07  | 826.15 |     3.34      |      1.51       |\n  \n    \n**Q4：Compute and latency details.**\nWe compare the model size and inference latency (including token decoding) of our method with both traditional lightweight models (e.g., UniAD, DiffusionDrive, LAW, Drive-OccWorld) and multimodal large models (e.g., DriveLM-Agent) on a single NVIDIA 4090 GPU using the nuScenes dataset.\nAs expected, multimodal large models such as DriveLM-Agent and Ours generally have larger model sizes than lightweight baselines, given their use of large-scale language or vision-language backbones.\nIn terms of inference latency, in addition to differences in model scale, variations also stem from different output requirements. For instance, Drive-OccWorld involves iterative reconstruction of future occupancy grids before planning, which adds substantial computational overhead. LLM-Driver and DriveLM-Agent require autoregressive generation of relatively long natural language outputs prior to decision making. Our method, while generating shorter textual outputs, also produces compact future-frame embeddings. \n    \n| Metric           | UniAD  | DiffusionDrive | LAW    | Drive-OccWorld | PWM (Ours) | DriveLM-Agent |\n|:----------------:|:------:|:--------------:|:------:|:--------------:|:----------:|:-------------:|\n| Params           | 131.9M | 60.0M          | 40.2M  | 56.3M          | 1.5B       | 3.9B          |\n| Inference Latency| 555.6ms| 122.0ms        | 51.2ms | 657ms          | 1059ms     | 6250ms        |\n\n**Q5: Failure‑case analysis.**\nDue to restrictions on uploading external links or supplementary PDFs on the rebuttal platform, we provide a textual description of failure cases where future-frame hallucinations have misled the planner. These failure cases can be mainly attributed to Limited Field-of-View Induced Hallucination.\nAs our method operates on monocular observations, certain scenarios—such as intersections or turns—pose a challenge. The limited field of view may prevent the model from capturing distant or occluded dynamic agents (e.g., oncoming vehicles), leading to hallucinated clear paths and subsequently unsafe planning decisions.\nFuture work will enhance dynamic capture via advanced modules and adopt multi-view inputs to reduce hallucinations and improve safety."}, {"review_id": "FhjPPpU6F1", "reviewer": "Reviewer_4cZK", "summary": "The paper introduces PWM, a novel driving policy pretaining recipe that leverages action-free front-view driving videos in a scalable way. During the pretraining stage, it learns general world knowledge through a future state forecasting task. During the finetuning stage, it add action predictions conditioned on the forecasted states. To facilitate the efficiency, the paper further presents a context-guided image tokenizer to compress each frame to only 28 tokens. The resulting policy achieves state-of-the-art performance on both nuScenes and NAVSIM benchmarks.", "questions": "Q1) Compared to interleaved video-action prediction, do you think that multi-frame video and multi-step action prediction will lead to better planning results, similar to the findings in [1]?\n\nQ2) The current model still requires visual prediction during planning. However, one might wonder if the policy model could bypass visual prediction at inference while still leveraging the pretrained world dynamics knowledge [2]. Do you think this idea is applicable to the proposed architecture?\n\n[1] GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation. arXiv 2024\n\n[2] Unified Video Action Model. RSS 2025", "limitations": "Please see the Weaknesses part.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\nS1) The proposed method is well-motivated and highly streamlined.\n\nS2) The authors provide solid results to demonstrate the effectiveness of each component.\n\nS3) Beyond the common public datasets, the authors leverage the large-scale OpenDV to demonstrate the promising scalability of their method.\n\n# Weaknesses\n\nW1) **Inference cost.** Since the planning pipeline involves visual prediction, efficiency could be a major concern. It would be beneficial to compare the inference speed with some typical end-to-end planning methods and report the speedup powered by the proposed context-guided tokenizer.\n\nW2) **Confusing summarization of related work.** In 3.2, the authors mention that previous methods (DrivingGPT, Doe-1, VaVAM) fail to perform forecasting before action prediction. However, in their sequential prediction pipeline, the observation token is placed before the corresponding action token and future observation tokens do not theoretically affect the current action planning. Do you mean their methods can only utilize action-labeled data? For VaVAM, it also has demonstrated clear benefits from action-free generative pretraining. Therefore, I'm a little confused about the authors’ explanation of the limitations of previous work and the advantages of the proposed pipeline in leveraging driving videos.\n\nW3) **Interleaved variant.** It would be beneficial to see how the model performs when the image tokens and action tokens are predicted in an interleaved manner during finetuning.\n\nW4) **Minor typos.** In Line 100, \"Driving into the Future\" can be abbreviated as \"Drive-WM\". In Line 107, \"Poicy\" should be \"Policy\". In Line 152, \"Most Prior\" should be \"Most prior\". Plus, \"Open-YouTube\" dataset should be \"OpenDV-YouTube\" dataset.\n\nW5) **Discuss MLLM.** Since the model is initialized from Show-o, it would be great for the authors to include a brief discussion about recent advancements in multimodal understanding and generation.", "was_revised": true, "final_justification": "PWM presents a valuable, unified, scalable framework for autonomous driving research. Thus, I will maintain my initial rating.", "rebuttal": "We sincerely appreciate the reviewer’s thoughtful and constructive suggestions. Detailed responses are as follows. \n\n**W1: Inference cost**\n\nThe inference cost of autonomous driving systems is indeed a crucial metric for evaluating the safety and practicality of the system. In our response to Reviewer n8oM's comment on w3, we provided a table comparing the inference costs between modern state-of-the-art autonomous driving methods. Our inference latency is comparable with MLLM-based competitors with leading driving performance. Detailed results will be added to the final version of our paper.\n\n**W2: Differences from previous work**\n\nDrivingGPT and Doe-1 employ an autoregressive approach that alternates between multiple modalities for training and inference. This contrasts significantly with the staged training paradigm proposed in our work, which first focuses on world knowledge learning without actions, followed by end-to-end planning task learning with future predictions.\nThe work of VaViM and VaVAM centers around integrating an action expert model with a video generation model for application in autonomous driving. During the pretraining phase, VaViM and VaVAM utilize an autoregressive training method for video generation, but their primary focus is on representation modeling and learning. In the fine-tuning stage, the video generation capability is not explicitly leveraged in planning tasks, and their inference process does not involve predicting future states. In contrast, our approach requires future state prediction during inference, which is then used as the basis for planning action predictions.\n\n**W3/Q1: Interleaved variant**\n\nWe understand your intent as conducting an ablation experiment to verify the interdependencies between the world model's future predictions and the autonomous driving planning tasks. In response, we provide in Table 4 the impact of different levels of future state predictions during fine-tuning on downstream planning tasks. It is evident that at the appropriate level of future state prediction (10 frames), the planning task achieves optimal performance. We analyze this result as follows: at lower levels of future state prediction, the model is unable to predict future states accurately or sufficiently, leading to a decline in planning task performance; at higher levels, the uncertainty of the model's future state predictions becomes too high, causing incorrect guidance for the planning task.\n\n**W5: Discusson on MLLM**\n\nThank you for your suggestion. Undoubtedly, the development of MLLMs forms a crucial foundation for our work. We will include a dedicated section on related work in the main paper.\n\n**Q2:Bypass visual prediction**\n\nThank you for your question. Our framework proposes the paradigm of using a world model to predict future states, while simultaneously performing planning tasks based on the predicted future states. This approach has not been fully explored in the field of autonomous driving, which is the primary motivation behind our PWM proposal. The paradigm you mentioned, which does not explicitly predict future states but directly outputs actions, has been discussed in works such as VaViM and VaVAM. Additionally, in our work, Table 4 presents the performance of PWM in planning tasks without visual predictions in the first row.\n\n\n**W4: Typos**\n\nThank you for your careful feedback. We will correct these typos in the main paper."}, {"review_id": "e7yjayODUD", "reviewer": "Reviewer_WR4K", "summary": "This paper introduces the Policy World Model (PWM), a novel driving paradigm that unifies world modeling and trajectory planning within a single, end-to-end architecture. The key innovation of PWM lies in its ability to leverage learned world knowledge for planning through an action-free future state forecasting scheme. To enhance efficiency in video forecasting, the authors propose a new image tokenizer with context-guided compression and decoding, alongside a dynamic focal loss to prioritize temporally varying regions. The PWM is pre-trained on large-scale unlabeled video datasets for action-free video generation and then fine-tuned for end-to-end planning.", "questions": "See in Weaknesses.", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper includes thorough ablation studies investigating the impact of action-free video world knowledge and the effectiveness of the dynamic focal loss. These studies clearly demonstrate the benefits of each proposed component, providing strong empirical support for the design choices. \n2. This paper proposes a dual-branch image tokenizer to achieve a balance between generation quality and efficiency.\n3. PWM integrates world modeling and trajectory planning into a unified architecture, explicitly enabling the learned world knowledge to benefit planning. \n\n\nWeaknesses:\n\n1. While the ablation studies effectively demonstrate the utility of each proposed component, the final overall performance of the method does not appear to be outstanding when compared to the state-of-the-art results presented, as shown in Table 1. It is recommended to highlight the best-performing method more clearly in the tables for easier comparison.\n2. In Appendix B.1, it's stated that \"when $\\alpha \\ge \\beta$, the Dynamic Focal Loss mechanism tends to fail.\" However, in Equation (2) within the main paper, the condition is given as $\\alpha > \\beta$. This discrepancy in the relationship between $\\alpha$ and $\\beta$ needs clarification.\n3. To more intuitively compare the performance of world model designs in autonomous driving, it might be beneficial to evaluate using a closed-loop simulation environment like CARLA, focusing on metrics such as episode return. This would provide a more holistic assessment of how the world model contributes to actual driving behavior and safety.\n\nTypos:\n\nIn line 158 and line 164, \"Figure 3(b)\" should be \"Figure 3(a)\".", "was_revised": true, "final_justification": "This paper is valuable and innovative, and the authors have largely resolved my concerns. Therefore, I am inclined to recommend acceptance.", "rebuttal": "Thank you for your detailed analysis and valuable feedback; we greatly appreciate your professional insights. Detaield comments are as follows.\n\n**W1: About Performance of Our Method**\n\nIn our experimental setup, the collision rate metric is relatively crucial. The Nuplan benchmark paper [1] introduces the L2 metric to measure \"shape similarity\" in imitation learning, but it has a weak correlation with safety. On the other hand, the collision rate metric directly measures the safety of behavior and has a significant impact on the deployment and robustness of autonomous driving systems. In Table 1, we compare our method with the current state-of-the-art autonomous driving algorithms. Our method achieves the most advanced performance in terms of the collision rate metric, while simultaneously achieving state-of-the-art results for the L2 metric.\n\nOur method uses less input information. In the evaluation shown in Table 1, we achieved results comparable to the state-of-the-art methods while using less input information. Specifically, our method only utilizes a single front-view visual input, whereas other methods in the table rely on multi-view visual inputs or even radar inputs.\n\nWe highlight the optimal results and enhance the clarity of presentation. We will classify the results based on the amount of input information to help avoid any doubts regarding the performance of our method.\n\n\n\n**W3: Closed-loop Evaluation**\n\nOur experiments on the NAVSIM dataset and simulator demonstrate that the closed-loop results can be effectively validated. NAVSIM is based on real-world data and uses a simulator to unfold the predicted future trajectories into a short-term (typically 4 seconds) small-scale BEV (Bird’s Eye View) simulation. This design is similar to a “Pseudo-Simulation” approach. The PDMS (Predictive Driver Model Score) rating system of NAVSIM is inspired by closed-loop scoring systems, such as the CLS in nuPlan. This rating system, through multi-dimensional comprehensive evaluation, ensures that the open-loop assessment results are highly consistent with the complete closed-loop simulation metrics. We compared the open-loop evaluation of NAVSIM (PDMS) with the real closed-loop simulation (CLS), and the results showed that PDMS achieved a Pearson and Spearman correlation coefficient of approximately 0.8 (significantly higher than the traditional prediction error-based score of ~0.7) [2].\n\nUndoubtedly, it is essential to conduct experiments on widely recognized simulation platforms such as CARLA. But due to time and engineering constraints, the relevant experiments and results are still under development. We will provide the complete experimental results in the near future.\n\n**W2: Typos**\n\nThank you for your pointing out the typo. The expression α > β in the main paper is correct. In Appendix B.1, it should be α ≤ β.\n\n\n\n[1] Karnchanachari N, Geromichalos D, Tan K S, et al. Towards learning-based planning: The nuplan benchmark for real-world autonomous driving[C]. 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024: 629-636.\n\n[2]Dauner D, Hallgarten M, Li T, et al. Navsim: Data-driven non-reactive autonomous vehicle simulation and benchmarking[C]. Advances in Neural Information Processing Systems, 2024, 37: 28706-28719."}, {"review_id": "xOcFqpRNMg", "reviewer": "Reviewer_NcJp", "summary": "This paper proposes PWM (Policy World Model), a unified architecture that integrates world modeling and trajectory planning. By tightly coupling the world model with the planner, PWM enables the world model to directly benefit the planning process. To enhance efficiency, the paper also introduces a novel image tokenizer. The proposed method demonstrates strong performance on the NAVSIM and nuScenes benchmarks.", "questions": "1. As shown in Tables 3 and 4, the performance gain from pretraining (77.8 → 86.3) is more substantial than that from forecasting (87.3 → 88.1). This suggests that the key factor may not lie in the forecasting capability of the world model itself, but rather in the effectiveness of video pretraining in a world model-like framework.\n2. Empirically, autoregressive learning tends to perform better when the signal-to-noise ratio is higher. In this work, 28 tokens are used to encode the future frames. Have you explored how varying the number of tokens affects the performance?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength:**\n\n1. The paper is clearly presented, with a straightforward and compelling motivation.\n2. Table 3 reveals a positive correlation between visual quality and planning performance, which is an interesting and noteworthy observation.\n3. PWM achieves performance comparable to existing BEV-based methods within an autoregressive framework on the NAVSIM benchmark, highlighting its potential.\n4. The proposed dual-branch tokenizer proves to be highly effective for video compression, significantly reducing the number of tokens.\n\n**Weakness:**\n\n1. The main concerns lie in computational efficiency and the lack of support for multi-sensor inputs.", "was_revised": true, "final_justification": "This paper presents PWM, which achieves performance comparable to existing BEV-based methods within an autoregressive framework on the NAVSIM benchmark, demonstrating its potential. The paper also investigates the impact of visual prediction quality on policy learning, offering meaningful insights. I recommend accepting this paper.", "rebuttal": "Thank you for your insightful review and recognition of PWM's strengths and contributions. Detaield responses are as follows.\n\n**W1: Concerns of computational efficiency and inadequate multi-sensor input support.**\n\nEfficiency:  In our response to Reviewer n8oM's comment (W3), we compare the inference costs between state-of-the-art autonomous driving methods. Our inference latency is comparable with MLLM-based competitors with leading driving performance. Detailed results will be added to the final version of our paper.\n    \nMulti-sensor / multi-view support: While our current work does not explicitly focus on multi-view or multi-sensor fusion, we consider the autoregressive framework inherently extensible and well-suited for integrating multiple modalities. Although this is not the primary focus of our current study, we regard it as a promising direction for future research. Notably, concurrent works such as Gaia-2 have demonstrated the feasibility of multi-view autoregressive generation, which we find both insightful and inspiring for extending our method.\n\n**Q1: Pretraining vs. Forecasting in Performance Gain.**\n\nTo better disentangle the effects of video pretraining and future frame forecasting, we conduct additional ablation studies on the NAVSIM dataset (containing 100k training samples). As shown in the following table, pre-training is indeed crucial for our model to learn world modeling capabilities. With our proposed forecasting mechanism, we can observe an additional driving performance gain, which indicates that the contributions of forcasting and pre-training are orthogonal to the final performance.     \n    \n    \n| Pre-train | Forecast  | NC ↑        | DAC ↑       | EP ↑        | TTC ↑       | Comf. ↑     | PDMS ↑      |\n|:---------:|:--------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|\n| ✗         | ✗        | 97.3        | 94.5        | 81.4        | 92.1        | 100.0       | 85.6        |\n| ✓         | ✗     | 98.0 (+0.7) | 95.1 (+0.6) | 82.4 (+1.0) | 94.1 (+2.0) | 99.9 (-0.1) | 87.3 (+1.7) |\n| ✓         | ✓       | 98.6 (+1.3) | 95.9 (+1.4) | 81.8 (+0.4) | 95.4 (+3.4) | 100.0 (+0.0)| 88.1 (+2.5) |\n\n**Q2: The choice of token quantity and impact on performance.** \n    \nIn our implementation, the tokenizer used in the baseline model encodes a 128×224 image into a compressed feature map of shape 8×14, resulting in 112 tokens. In our work, we adopt a more compact representation by further compressing the spatial dimensions of the future frames, resulting in a 4×7 grid of tokens (i.e., 28 tokens) per frame.\n    \nWe acknowledge that exploring the effect of token count on downstream planning performance is a meaningful direction. However, such an investigation requires retraining the tokenizer, followed by full-scale pretraining and finetuning for each token configuration, which is computationally expensive. As a result, we have not yet completed this ablation.\n     \nHowever, we note that prior work [1] has studied the effect of token count in visual tokenizers designed for generation tasks. An interesting observation as shown in Figure 4 of [1] is that increasing the number of tokens improves generation quality. When these tokens are directly used for perceptual tasks (e.g. ImageNet-1K linear probing) the performance tends to degrade. This suggests a potential trade-off: using more tokens helps capture texture-level information, while using fewer tokens yields more compact and semantically concentrated representations, which may be more beneficial for high-level tasks such as planning.\n\n\n[1]  Yu Q, Weber M, Deng X, et al. An image is worth 32 tokens for reconstruction and generation[C]. Advances in Neural Information Processing Systems, 2024, 37: 128940-128966."}]

Author rebuttal:

Meta-review: ### Summary and Recommendation Justification

This paper introduces the Policy World Model (PWM), a novel and well-motivated paradigm for autonomous driving that tightly integrates world modeling and trajectory planning into a single, unified architecture. The core contribution is a system where planning directly benefits from the world model's learned ability to forecast future states, enabling a form of human-like anticipatory perception. The authors complement this architectural innovation with practical efficiency improvements, including a highly compressed image tokenizer and a dynamic focal loss. The proposed method achieves state-of-the-art performance on key safety metrics, notably reducing collision rates on the nuScenes benchmark while using only a single front-facing camera, outperforming methods that rely on richer multi-modal inputs.

After effective rebuttal period, reviewers reached a clear consensus for acceptance. The reviewers consistently praised the work for its compelling motivation, technical soundness, and the novelty of its unified framework. The strong empirical validation, supported by thorough ablation studies that isolate the contribution of each component, was another frequently cited strength.

### Detailed Justification

**Strengths:**

1.  **Conceptual Novelty and Significance:** The central idea of a truly unified world model that also acts as the policy is a significant conceptual advance. As highlighted by multiple reviewers (4cZK, WR4K, NcJp), this is a departure from prior works where world models and planners are either decoupled or treated as independent downstream tasks within a larger architecture. By enabling the planner to directly leverage the model's own future-state predictions, PWM presents a valuable and scalable framework for end-to-end planning.

2.  **Strong Empirical Performance:** The paper provides robust experimental evidence for its claims. PWM matches or exceeds the performance of state-of-the-art methods on both the nuScenes and NAVSIM benchmarks. Crucially, as the authors clarified during the rebuttal, it achieves the lowest collision rate while using significantly less input data (monocular vs. multi-view/multi-modal). This demonstrates not just high performance but also high data efficiency.

3.  **Technical Soundness and Rigorous Evaluation:** The reviewers commended the thoroughness of the ablation studies, which convincingly demonstrate the effectiveness of the action-free pre-training, the forecasting mechanism, the novel tokenizer, and the dynamic focal loss. The two-stage training paradigm (pre-training on large-scale unlabeled video, fine-tuning for planning) is well-motivated and demonstrates a promising path for leveraging vast amounts of readily available driving data.

### Addressing Initial Concerns

Initial reviews raised several important questions regarding the method's novelty, the precise nature of its "unification," fairness in comparisons, and computational cost. The authors' rebuttal was exemplary in addressing these points:

*   **Novelty and Unification:** The authors effectively clarified the distinction between PWM and prior work like Drive-OccWorld, positioning their approach as a more deeply integrated paradigm where planning emerges from the world model's internal representations, rather than being a separate downstream module. The discussion with Reviewer n8oM helped refine the understanding of "collaborative state-action prediction" as a causal knowledge-sharing mechanism within a single model.
*   **Fairness and Computational Cost:** The authors justified their use of extra pre-training data by pointing to common practices in the field and provided a new ablation that isolated the gains from their proposed loss function, independent of the extra data. Furthermore, they provided a detailed comparative analysis of model size and inference latency, showing that PWM's cost is competitive within the landscape of modern MLLM-based driving agents.
*   **Pre-training vs. Forecasting:** In response to Reviewer NcJp, the authors provided a new, detailed ablation that successfully disentangled the performance gains from pre-training and forecasting, demonstrating that both contribute orthogonally to the final performance.

The productive discussion phase led to a convergence of opinions, with reviewers confirming that their concerns had been largely resolved and either raising their scores or solidifying their positive assessment.

In short, this paper introduces a promising new architecture, supports it with strong empirical results, and provides valuable insights into the synergy between world modeling and planning.
Decision: Accept (poster)