Title: From Forecasting to Planning: Policy World Model for Collaborative State-Action Prediction
Venue: NeurIPS 2025
Accepted: yes
Keywords: World model

Autonomous driving

End-to-end planning
Abstract: Despite remarkable progress in driving world models, their potential for autonomous systems remains largely untapped: the world models are mostly learned for world simulation and decoupled from trajectory planning. While recent efforts aim to unify world modeling and planning in a single framework, the synergistic facilitation mechanism of world modeling for planning still requires further exploration. In this work, we introduce a new driving paradigm named Policy World Model (PWM), which not only integrates world modeling and trajectory planning within a unified architecture, but is also able to benefit planning using the learned world knowledge through the proposed action-free future state forecasting scheme. Through collaborative state-action prediction, PWM can mimic the human-like anticipatory perception, yielding more reliable planning performance. To facilitate the efficiency of video forecasting, we further introduce a parallel token generation mechanism, equipped with a context-guided tokenizer and an adaptive dynamic focal loss. Despite utilizing only front camera input, our method matches or exceeds state-of-the-art approaches that rely on multi-view and multi-modal inputs.  Code will be released at https://github.com/6550Zhao/Policy-World-Model.


Peer reviews:
comments: SUMMARY:
This paper proposes PWM (Policy World Model), a unified architecture that integrates world modeling and trajectory planning. By tightly coupling the world model with the planner, PWM enables the world model to directly benefit the planning process. To enhance efficiency, the paper also introduces a novel image tokenizer. The proposed method demonstrates strong performance on the NAVSIM and nuScenes benchmarks.

STRENGTHS AND WEAKNESSES:
**Strength:**

1. The paper is clearly presented, with a straightforward and compelling motivation.
2. Table 3 reveals a positive correlation between visual quality and planning performance, which is an interesting and noteworthy observation.
3. PWM achieves performance comparable to existing BEV-based methods within an autoregressive framework on the NAVSIM benchmark, highlighting its potential.
4. The proposed dual-branch tokenizer proves to be highly effective for video compression, significantly reducing the number of tokens.

**Weakness:**

1. The main concerns lie in computational efficiency and the lack of support for multi-sensor inputs.

QUESTIONS:
1. As shown in Tables 3 and 4, the performance gain from pretraining (77.8 → 86.3) is more substantial than that from forecasting (87.3 → 88.1). This suggests that the key factor may not lie in the forecasting capability of the world model itself, but rather in the effectiveness of video pretraining in a world model-like framework.
2. Empirically, autoregressive learning tends to perform better when the signal-to-noise ratio is higher. In this work, 28 tokens are used to encode the future frames. Have you explored how varying the number of tokens affects the performance?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Policy World Model (PWM), a novel driving paradigm that unifies world modeling and trajectory planning within a single, end-to-end architecture. The key innovation of PWM lies in its ability to leverage learned world knowledge for planning through an action-free future state forecasting scheme. To enhance efficiency in video forecasting, the authors propose a new image tokenizer with context-guided compression and decoding, alongside a dynamic focal loss to prioritize temporally varying regions. The PWM is pre-trained on large-scale unlabeled video datasets for action-free video generation and then fine-tuned for end-to-end planning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper includes thorough ablation studies investigating the impact of action-free video world knowledge and the effectiveness of the dynamic focal loss. These studies clearly demonstrate the benefits of each proposed component, providing strong empirical support for the design choices. \n2. This paper proposes a dual-branch image tokenizer to achieve a balance between generation quality and efficiency.\n3. PWM integrates world modeling and trajectory planning into a unified architecture, explicitly enabling the learned world knowledge to benefit planning. \n\n\nWeaknesses:\n\n1. While the ablation studies effectively demonstrate the utility of each proposed component, the final overall performance of the method does not appear to be outstanding when compared to the state-of-the-art results presented, as shown in Table 1. It is recommended to highlight the best-performing method more clearly in the tables for easier comparison.\n2. In Appendix B.1, it's stated that \"when $\\alpha \\ge \\beta$, the Dynamic Focal Loss mechanism tends to fail.\" However, in Equation (2) within the main paper, the condition is given as $\\alpha > \\beta$. This discrepancy in the relationship between $\\alpha$ and $\\beta$ needs clarification.\n3. To more intuitively compare the performance of world model designs in autonomous driving, it might be beneficial to evaluate using a closed-loop simulation environment like CARLA, focusing on metrics such as episode return. This would provide a more holistic assessment of how the world model contributes to actual driving behavior and safety.\n\nTypos:\n\nIn line 158 and line 164, \"Figure 3(b)\" should be \"Figure 3(a)\".\n\nQUESTIONS:\nSee in Weaknesses.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces PWM, a novel driving policy pretaining recipe that leverages action-free front-view driving videos in a scalable way. During the pretraining stage, it learns general world knowledge through a future state forecasting task. During the finetuning stage, it add action predictions conditioned on the forecasted states. To facilitate the efficiency, the paper further presents a context-guided image tokenizer to compress each frame to only 28 tokens. The resulting policy achieves state-of-the-art performance on both nuScenes and NAVSIM benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\nS1) The proposed method is well-motivated and highly streamlined.\n\nS2) The authors provide solid results to demonstrate the effectiveness of each component.\n\nS3) Beyond the common public datasets, the authors leverage the large-scale OpenDV to demonstrate the promising scalability of their method.\n\n# Weaknesses\n\nW1) **Inference cost.** Since the planning pipeline involves visual prediction, efficiency could be a major concern. It would be beneficial to compare the inference speed with some typical end-to-end planning methods and report the speedup powered by the proposed context-guided tokenizer.\n\nW2) **Confusing summarization of related work.** In 3.2, the authors mention that previous methods (DrivingGPT, Doe-1, VaVAM) fail to perform forecasting before action prediction. However, in their sequential prediction pipeline, the observation token is placed before the corresponding action token and future observation tokens do not theoretically affect the current action planning. Do you mean their methods can only utilize action-labeled data? For VaVAM, it also has demonstrated clear benefits from action-free generative pretraining. Therefore, I'm a little confused about the authors’ explanation of the limitations of previous work and the advantages of the proposed pipeline in leveraging driving videos.\n\nW3) **Interleaved variant.** It would be beneficial to see how the model performs when the image tokens and action tokens are predicted in an interleaved manner during finetuning.\n\nW4) **Minor typos.** In Line 100, \"Driving into the Future\" can be abbreviated as \"Drive-WM\". In Line 107, \"Poicy\" should be \"Policy\". In Line 152, \"Most Prior\" should be \"Most prior\". Plus, \"Open-YouTube\" dataset should be \"OpenDV-YouTube\" dataset.\n\nW5) **Discuss MLLM.** Since the model is initialized from Show-o, it would be great for the authors to include a brief discussion about recent advancements in multimodal understanding and generation.\n\nQUESTIONS:\nQ1) Compared to interleaved video-action prediction, do you think that multi-frame video and multi-step action prediction will lead to better planning results, similar to the findings in [1]?\n\nQ2) The current model still requires visual prediction during planning. However, one might wonder if the policy model could bypass visual prediction at inference while still leveraging the pretrained world dynamics knowledge [2]. Do you think this idea is applicable to the proposed architecture?\n\n[1] GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation. arXiv 2024\n\n[2] Unified Video Action Model. RSS 2025\n\nLIMITATIONS:\nPlease see the Weaknesses part.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Policy World Model (PWM), an autoregressive Transformer that jointly learns a video‑based world model and performs trajectory planning, leveraging action‑free future‑frame generation instead of action‑labelled data. It adds a dual‑branch image tokenizer that compresses each frame to 28 tokens and enables frame‑level parallel prediction, plus a Dynamic Focal Loss to focus learning on temporally changing regions. Combined, these innovations let a single front‑camera model match or exceed multi‑sensor state‑of‑the‑art baselines on nuScenes and NAVSIM, while running at 40 FPS without pixel decoding.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. PWM is technically sound, with clear pre‑training → fine‑tuning stages, thorough ablations isolating each design choice, and consistent performance improvements across metrics and datasets. \n2. It cuts nuScenes collision rates to 0.07 % and reaches a PDMS of 88.1 on NAVSIM using only a monocular camera, outperforming or tying LiDAR‑ and multi‑view methods. \n3. The tokenizer and loss yield a 16 × token reduction and demonstrable speed‑accuracy gains, showing practical efficiency. \n4. The manuscript is well‑written and organized, making replication feasible. \n\nWeaknesses:\n1. Lacks analysis and comparison with the existing state-of-the-art policy world model **Drive-OccWorld**[1]. The originality of **unified prediction** remains uncertain.\n2. Despite the claim of a unified generation-planning framework, the prediction of trajectory and the prediction of future states are still independent. In L181-188, future states are first obtained through **action-free** RGB frame prediction. Trajectories are then forecasted based on **predicted textual description and future forecasting results**. These two predictions are clearly **separate**. It's more like employing an additional decoder to **infer trajectories from videos**.\n3. Extra data (Open-Youtube) used for training, raising concerns about the fairness of performance comparisons.\n4. The performance gain from textual generation is limited, which hurts the necessity of the multimodal component.\n5. While unification is useful, the method is incremental over prior autoregressive world‑model frameworks. It's more like employing an additional decoder to infer future trajectories from predicted videos.\n\n[1] Yang, Yu, et al. \"Driving in the occupancy world: Vision-centric 4d occupancy forecasting and planning via world models for autonomous driving.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 9. 2025.\n\nQUESTIONS:\n1. Compare with **Drive-OccWorld**[1] (the SOTA work also employs a world model for end-to-end planning) on motivation, theoretical analysis, and experimental performance. Clearly point out your novelty and contribution.\n2. Explain why PWM is ''**unified**''? In L181-188, future states are first obtained through **action-free** RGB frame prediction. Trajectories are then forecasted based on **predicted textual description and future forecasting results**. These two predictions are clearly **separate**. It's more like employing an additional decoder to **infer trajectories from videos**.\n3. Provide more ablation results **without extra data** (Open-YouTube)  pre-training as follows. Otherwise, there will be concerns that performance gains come from additional large-scale and more comprehensive training samples. \n| Pre-train      | L_DFL-p | L_DFL-f    |\n| :---        |    :----:   |          ---: |\n| $\\times$     | $\\checkmark$       | $\\checkmark$   |\n| $\\times$   | $\\checkmark$        | $\\times$      |\n| $\\times$     | $\\times$       | $\\checkmark$   |\n4. Compute and latency details. Please provide a comparison with SOTA works like LAW / DiffusionDrive / Drive-OccWorld on (i) full model size and (ii) end‑to‑end inference latency including token decoding.\n5. Failure‑case analysis. Provide qualitative or quantitative analysis of cases where future‑frame hallucinations mislead planning (e.g., poor weather, heavy occlusion) and give some possible solutions to further ensure safety.\n\n[1] Yang, Yu, et al. \"Driving in the occupancy world: Vision-centric 4d occupancy forecasting and planning via world models for autonomous driving.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 9. 2025.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}