Plan-R1: Safe and Feasible Trajectory Planning as Language Modeling
Authors: Xiaolong Tang
Meina Kan
Shiguang Shan
Xilin Chen
Venue: ICLR 2026 Poster (2026)
Keywords: Trajectory Planning
Reinforcement Learning
Autonomous Driving
Primary area: applications to robotics, autonomy, planning
TL;DR: We propose Plan-R1, a two-stage framework that decouples planning principle alignment from behavior learning to overcome the limitations of expert data. With VD-GRPO to preserve safety-critical signals, Plan-R1 achieves SOTA results on nuPlan.

Abstract: Safe and feasible trajectory planning is critical for real-world autonomous driving systems.
However, existing learning-based planners rely heavily on expert demonstrations, which not only lack explicit safety awareness but also risk inheriting undesirable behaviors such as speeding from suboptimal human driving data.
Inspired by the success of large language models, we propose Plan-R1, a two-stage trajectory planning framework that decouples principle alignment from behavior learning.
In the first stage, a general trajectory predictor is pre-trained on expert data to capture diverse, human-like driving behaviors.
In the second stage, the model is fine-tuned with rule-based rewards using Group Relative Policy Optimization (GRPO), explicitly aligning ego planning with principles such as safety, comfort, and traffic rule compliance.
This two-stage paradigm retains human-like behaviors while enhancing safety awareness and discarding undesirable patterns from demonstrations.
Furthermore, we identify a key limitation of directly applying GRPO to planning: group-wise normalization erases cross-group scale differences, causing rare, high-variance safety-violation groups to have similar advantages as abundant low-variance safe groups, thereby  suppressing optimization for safety-critical objectives.
To address this, we propose Variance-Decoupled GRPO (VD-GRPO), which replaces normalization with centering and fixed scaling to preserve absolute reward magnitudes, ensuring that safety-critical objectives remain dominant throughout training.
Experiments on the nuPlan benchmark demonstrate that Plan-R1 significantly improves planning safety and feasibility, achieving state-of-the-art performance, particularly in realistic reactive settings.
Our code is available at https://github.com/XiaolongTang23/Plan-R1.

Published as a conference paper at ICLR 2026
PLAN-R1: SAFE AND FEASIBLE TRAJECTORY PLAN- NING AS LANGUAGE MODELING
Xiaolong Tang, Meina Kan, Shiguang Shan, Xilin Chen Institute of Computing Technology, Chinese Academy of Sciences University of Chinese Academy of Sciences (tangxias aonnxiaallon ao long22s, kanmeina, sgshan, Eichen)eict.as
ABSTRACT
Safe and feasible trajectory planning critical for real-world autonomous driv- ing systems. However, existing learning- based planners rely heavily on expert demonstrations, which not only lack explicit safety awareness but also risk in- heriting undesirable behaviors such as speeding from suboptimal human driv- ing data. Inspired by the success of large language models, we propose Plan- R1, two-stage trajectory plamning framework that decouples principle align- ment from behavior learning. In the first stage, a general trajectory predic- tor is pre-trained expert data to capture diverse, human-like driving behav- iors. In the second stage, the model is fine-tuned with rule-based rewards us- ing Group Relative Policy Optimization (GRPO), explicitly aligning ego plan- ning with principles such as safety, comfort, and traffic rule compliance. This two-stage paradigm retains human-like behaviors while enhancing safety aware- ness and discarding undesirable patterns from demonstrations. Furthermore, we identify a key limitation of directly applying GRPO planning: group-wise normalization erases gross-group scale differences, causing rare, high-variance safety-violation groups to have similar advantages as abundant low-variance safe groups, thereby suppressing optimization for safety-critical objectives. To ad- dress this, we propose Variance- Mtriance-Decoupled GRPO (VD-GRPO), which replaces normalization with centering and fixed scaling to preserve absolute reward mag- nitudes, ensuring that safety-critical objectives remain dominant throughout train- ing. Experiments the nuPlan benchmark demonstrate that Plan-RI signif- icantly improves planning safety and feasibility, achieving state-of-the-art per- formance, particularly in realistic reactive settings. Our code is available com Cwttuu..oo//iiaoo
https:: :atps://gge //tp://gtttb././/////
INTRODUCTION
Trajectory planning is fundamental component of autonomous driving systems, directly influenc- ing vehicle safety, efficiency, and the ability to navigate complex and dynamic environments. In
recent years, learning-based planning approaches (Hu et al. 2023 Xu al.) (024) have attracted increasing attention due to their strong adaptability, competitive performance, and minimal reliance on manually designed rules. These methods offer promising solutions for generating trajectories that can respond effectively to various traffic scenarios and rapidly changing road conditions.
However, due to the complexity and diversity of real-world driving scenarios, most existing planning methods, whether based on imitation learning (IL) (Cheng et al. [2024b al Zheng et al. 2025
Tang
al. 2025) or reinforcement learning (RI) (Zhang et al. 2025 Huu al. 2024 Li et al. 2025) rely
heavily on expert demonstrations for supervision. This dependency introduces two key limitations:
(i) expert data rarely covers negative scenarios such as collisions or off-road driving, leaving the model unable to explicitly learn how to avoid them, and (ii) demonstrations are not always optimal and may contain undesirable behaviors. For example, we find that over 10% of the nuPlan (Caesar et al. 2021) training scenes exhibit speeding, and some contain uncomfortable maneuvers crit- ically low time-to-colision (TTC) values. As a result, models trained purely on expert data risk inheriting these undesirable behaviors without clear notion of safety, as shown in Figure

---

Published as a conference paper at ICLR 2026
To address these limitations, draw inspiration from the success of large language models (LLMs) (Shao et al. 2024, Ouyang et al., 2022). which typically adopt a two-stage training paradigm: pre-training
general-purpose predictor via next-token prediction, followed by RL fine-tuning to align outputs with desired objectives (e.g., format). Our key insight is that trajec- tory planning can be formulated in similar way, i.e,, first trained a trajectory predictor on expert driving data, and then fine-tuned using RL to align the trajectory with explicit plan- ning principles, such safety, comfort, and rule compliance. This decoupled paradigm separates principle alignment from behavior learning, enabling the model to retain human-like behaviors, en- hancing safety awareness and discarding undesirable patterns learned from the expert data.
Specifically, we introduce Plan-R1, two-stage dual-model framework for principle-alignee trajec- tory planning. the pre-training stage, trajectories are discretized into motion tokens across time and space (Seff et al. 2023 Philion et al., 2023), and general motion predictor is trained with next- motion-token prediction to capture diverse, human-like multi-agent behaviors. In the fine-tuning stage, unlike prior methods that rely on human preference data (Huang et al. 20224 or additional expert demonstrations (Li et al. 2025), which may introduce biases or undesirable behaviors, we use rule-based rewards that provide consistent and unbiased supervision. To ensure realistic multi-agent interactions and stable optimization, we introduce a dual-modes design: a trainable ego planner ex- plores alternative decisions, while frozen copy of the pre-trained model serves reactive world model to predict the responses of surrounding agents. This separation enables ego-centric policy up- dates without destabilizing non-ego behaviors, yielding stable, interaction- aware joint predictions.
For RL optimization, we adopt Group Relative Policy Optimization (GRPO) (Shao et al. 2024), which has shown strong performance across various domains (Huang et al. 2025). However, we find that its default design is not well suited for trajectory planning. Unlike mathematical reasoning tasks where the goal simply to produce the correct answer, trajectory planning must jointly op- timize multiple. potentially conflicting objectives with carefully designed priorities. For example, collision avoidance must always take precedence over comfort. After pre-training, nearly 80% of trajectory groups contain no safety violations, and their reward variance is dominated by non-safety objectives such as comfort. Standard GRPO normalizes rewards independently within each group, erasing natural scale differences across groups. This causes rare, high-variance safety-violation groups to have normalized advantages comparable to abundant safe groups (Figure 4), dilut- ing safety-critical gradients and shifting optimization toward non-safety objectives. To address this, we propose Variance- Decoupled GRPO (VD- VD-GRRO), which replaces per-group normalization with centering and fixed scaling. By preserving absolute reward magnitudes, safety-critical groups naturally generate larger gradients, ensuring that high-priority safety objectives remain dominant and continue to improve even late-stage training when such cases become extremely rare.
By addressing both behavior-prineipe decoupling and long-tailes safety optimization, Plan-RI pro- vides a unified and scalable solution for safe and feasible trajectory planning. Results on the nuPlan benchmark show that Plan-R1 significantly improves both safety and feasibility, achieving state-of- the-art performance and demonstrating strong generalization challenging interactive scenarios.
The primary contributions of this paper are:
We introduce a new perspective that formulates trajectory planning as a principle-aligned pre- diction task, decoupling planning principle alignment from behavior learning to overcome the limitations of expert data.
We propose Plan-R1, a two-stage dual-model framework that first pre- trains a motion predictor on expert data capture diverse driving behaviors, and then fine-tunes it with rule-based reinforce- ment learning, requiring no additional expert or preference data.
We identify a key limitation of GRPO: per-group normalization erases cross-group scale differ- ences, diluting safety-critic signals. We address it with Variance- Decoupled GRPO (VD-GRPO) to preserve absolute reward magnitudes, prioritizing safety-critical objectives during optimization.
Plan-RI RI achieves state-of-ther -hee---r-thh-tttt performance on the nuPlan benchmark, significantly improving both safety and feasibility, particularly in challenging reactive settings.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
2.1 LEARNING-BASED TRAJECTORY PLANNING
Learning-bast -batin trajectory planning methods can be broadly grouped into imitation learning (IL), re- inforcement learning (RL), and hybrid ILL approaches. Pure II. and RL. have been extensively ex- plored for trajectory planning and achieved notable progress (Bansal et al. Schee et al. 2022
Tang et al. 2025 Cheng et al., 2024b a Zhang et al. 2025 Li et al. 2024, Cusumano-Towwe 2025). However, they both have inherent limitations: IL relies heavily on expert demonstrations, which rarely cover safety-critical events and often contain suboptimal behaviors such as speeding; RL enables behavior discovery beyond demonstrations but suffers from severe sample inefficiency, complex multi-objective reward design, and poor luman-likeness in large, dynamic environments.
et al,,
Recent works therefore explore hybrid IL+ approaches. One line of research adopts imitation- regularized RL, where the distance to expert trajectory is incorporated into the optimization objec- tive either reward signal or a regularizer (Zhang al.) [025 Lu et al. 2023). For example, BC-SAC (Lu et al., 2023) jointly optimizes a behavior cloning loss and a Soft Actor-Critic loss improve safety. These methods stabilize training but remain heavily dependent
expert data, inheriting its biases and undesirable behaviors. Another line follows a pre-training + fine-tuning paradigm inspired by LLMs, such as Gen-Drive (Huang al.) 2024) and TrajHF (Li al. 2025). Gen-Drive trains a reward model from collected preference data and fine-tunes the planner via RL. TrajHF fine-tunes the planner using human driving preference data to align with desired driving styles. However, collecting preference data is expensive and may introduce new biases, limiting scalability and reliability, In contrast, our Plan-RI also adopts a two-stage framework but replaces preference data with rule-based rewards, which provide consistent and unbiased supervision. This allows us to aligm planning with safety and other principles while preserving the diverse human- like behaviors, achieving safe and robust trajectory planning without extra human supervision.
2.2
GROUP RELATIVE POLICY OPTIMIZATION (GRPO)
GRPO (Shao et al. 2024) is reinforcement learning algorithm originally developed for mathemat- ical reasoning. samples multiple rollouts under the same context to form a group and normalizes rewards within each group to compute relative advantages, eliminating the need for value function and avoiding instability from inaccurate value estimation in methods like PPO (Schulman et al.] 2017). This simplicity makes GRPO highly effective for alignment tasks such as reasoning (Huang et al. 2025) and search (Jin et al. 2025). Moreover, DAPO (Yu et al. 2025) extends GRPO by introducing asymmetric clipping and dynamic sampling to improve optimization stability and effi- ciency. Dr.GRPO (Liu et al. (025)) removes the per-group standard deviation term, aiming to reduce problem-level difficulty bias and stabilize optimization in language model training. VD-GRPO is inspired by Dr.GRPO but addresses fundamentally different problem in different domain: in safety-critical trajectory planning. rewards are multi-objective and often conflicting, where rare but catastrophic events (c.g., collisions) must be prioritized. We demonstrate that group-wise normal- ization erases pross-group scale differences, causing rare, high-variance aafety-violation groups to have similar advantages to abundant low-variance safe groups, suppressing optimization for safety- critical objectives. VD-GRPO address it by replacing normalization with centering and fixed scaling, ensuring that safety-critical objectives remain dominant throughout training.
METHOD
We propose PIan-R1, two-stage framework that decouples planning principle alignment from be- havior learning. As illustrated in Figure 1] Plan-R1 1 consists of an autoregressive pre-training stage and a rule-based RL fine-tuning stage. The model is first pre-trained on expert demonstrations to capture diverse human-like motion behaviors, and then fine-tuned with rule-based rewards to align the ego motion with explicit planning principles. In ssubeectionn 3.1 we formulate trajectory plan- ning a principle-aligned sequence prediction task. |subsection 3.2 describes the autoregressive pre-trane trainiiinng process, while isubsection 3.3) presents the fine-tuning stage and introduces Variance- Decoupled GRPO (VD-GRPO) PO) to further address the safety-critical optimization challenge.
3

---

this for- Published as a conference paper at ICLR 2026
Agent-Agent Self-Attention
Sample
probability
probability
Agent- Map
Autoregressive Decoder
Temporal Self-Attention
(a) Pre-training
(b) Fine-tuning
MudelModel Rollout
Variance Deccoulled GRPO
Ha
1.0
-0.3
-1.0
Optimize
Autoregressive Decoder (Planning)
Centering, Scaling
Rule-based Reward
On-road Reward
Callision Reward
Speed Limit Reward
Comfort Reward
Autoregressive Decoder (World Model)
Rollout
Sample
Tunable
Frozen
Map embedding
Ego embedding
Agent embedding
Figure 1:
Illustration of our Plan-RI: Stage (a) pre-trains a motion predictor on expert data; Stage
(b) fine-tunes it with VD-GRPO using rule-based rewards to align with planning principles.
3.1 PROBLEM FORMULATION
Trajectory planning is inherently sequential, making autoregressive modeling natural choice. We extend this formulation into principle-aligned autoregressive prediction task by explicitly incor- porating high-level planning principles (e-g.. safety, comfort, traffic rule compliance). mulation, the ego vehicle's 's trajectory is generated jointly with the predicted motions of surrounding agents (Hagedorn et al. 2024 Zheng et al., 20244 Chen al. 2024), enabling the planner to model realistic multi aggnt interactions. Formally, given P. our goal is to generate the joint future motions = {Vt. /t=::;;; n=0:N for all agents, where is the planning horizon, N is the number of surrounding agents, Vt,a denotes the motion of agent time step I, and the ego vehicle is indexed by n = 0. Directly modeling p(Y | C, P) is intractable, but leveraging temporal causality (Seff al. 2023) and short-term conditional independence (Tang
context C' and set of planning principles
& Salakhutdinov 2019: Rhinehart et al. 2019) allows factorization as:
F
P(Y CYI.PP)
II p(u.ON BELONN, P)
p(u.o VCL.ONN C,P) II p(u., VCe.ONN C, P) n=1
(1)
me (uno VeLONN C, P)
II
Pa Ver,N, C):
ego planser
agent predictor
where the last approximation assumes that surrounding agents' future motions are independent of the ego vehicle's planning principles P. This yields two sub-problems: (i) predicting each sur- rounding agent's motion (V V<t,u:N- C), and (ii) generating the ego vehicle's trajectory ne (V Fer.O.N. C, P) with additional conditioning on planning principles. The agent predic- tor Pa be learned via next-motion prediction large-scale datasets (Wu et al. 2024), whereas
4

---

Published as a conference paper at ICLR 2026
modeling the ego planner is more challenging because it must not only produce human-likk mo- tions while satisfying P. Since Pa is trained on large-scale human driving data that implicitly reflects P. it provides strong prior for Te. However, this alignment is only implicit and imperfect: human demonstrations may still contain unsafe or suboptimal behaviors, and Pa itself cannot guarantee strict adherence to P. To address this, initialize ae with Pa and fine-tune it using reinforcement learning, where rulc-based rewards explicitly ensure compliance with P.
3.2 AUTOREGRESSIVE PRE-TRAINING
Tokenization. To enable autoregressive modeling, continuous trajectories are first discretized into motion tokens. Temporally, trajectories are segmented fixed intervals. Spatially, the K-disk clus- tering algorithm (Philion et al., 2023) is applied to the resulting motion segments based on their average corner distance, producing motion token vocabulary. Each token represents a prototypi- cal displacement and heading change over fixed time step. For each agent, the trajectory is thus transformed into sequence of motion tokens spanning the prediction horizon.
Model architecture. Our model employs transformer decoder with factorized atten- tion [Nayakanti et al.] 20033 Ngiam et al.] [021) to model multi-agent spatio-temporal interac- tions, as shown Figure (a). The architecture is designed to capture the temporal dynamics of each agent's 's trajectory and its interactions with the map and neighboring agents. Following prior work (Wu al. 2024; Seff et al., 2023), motion tokens are processed through stack of attention- based fusion blocks. Each block consists of three components: Temporal self-attention, modeling motion continuity and capturing temporal dependencies for each agent; Agent-map cross-attention, integrating spatial constraints by attending road elements and ensuring compliance with the road network; Agent-agent cross-attention, capturing local interactions with neighboring agents to reason about dynamic behaviors. To maintain translation and rotation invariance, relative spatio-tenn temporal position embeddings (Zhou et al. 2023] Tang et al.] 2024) are applied to all attention modules. Both ego and non-ego agents share the same encoder structure, and all attention computations implemented in a query-centric manner for efficient multi-agent rollout.
Training objective. The model is trained with next-motion-token prediction objective, minimiz- ing the negative logllikel of the next token for all agents:
Cpectrain
Σ log PogPa (y VeLeueN C.).
This objective enables the model to approximate the latent distribution of human driving behaviors, capturing diverse and realistic motion patterns from large-scale expert data. At this stage, the model simply imitates human-like behavior without explicit planning principles.
3.3 REINFORCEMENT LEARNING -INNITUNING
While autoregressive pre-training captures human-like motion patterns, does not explicitly en- force high-level planning principles. To address this gap, we fine-tune the ego motion predictor using reinforcement learning, as shown in Figure (b), optimizing it to better align with predefined planning principles such as safety, comfort, and rule compliance. The generation process is formu- lated as sequential decision-makin problem, where the ego agent iteratively selects motion tokens to maximize rewards that reflect desirable planning behaviors.
Dual-Model rollout. A key challenge in RL fine-tuning is to realistically simulate how surround- ing agents react to the ego vehicle's actions. naive solution is to replay ground-truth (GT) behav- iors of surrounding agents, which ignores ego interventions and thus yields unrealistic, non-reactive simulations. We address this issue with a dual-model design: a trainable ego planner He interacts with frozen copy of the pre-trained model Pa. which acts reactive world model for surrounding agents. During rollouts, ne explores alternative decisions while Pa predicts the responses other agents based the evolving joint history, enabling interaction-ant aware simulation without requiring additional training for P.. This separation allows ego-centric policy updates without destabilizing non-ego dynamics, resulting in stable and realistic multi-agent rollouts.
5

---

Published as a conference paper at ICLR 2026
Rule-based Rewards. Unlike pure RL-based planning, approach does not need to learn alistic human- behaviors from scratch. This is because the pre-trained model already generates plausible, human-like motions, which greatly simplifies reward design: it only needs to target spe- cific aspects such as safety, comfort, and rule compliance, without the burden of modeling basic driving realism. Here, design aspects such as collision avoidance, driving area compliance, comfort, speed limit compliance, and progress. Following (Caesar al. 2021] Hu al. a 2024), we compute the total reward as the product of two components: set of multipiicative safety indicators (i.e., collision avoidance, driving area compliance) and a weighted sum of soft cost terms (e.g., comfort, speed limit compliance, progress):
of interpretable, rule-based reward functions covering key
R(y)) = II 1
Σ JET
Wy Wy Ty(e)),
where 1 E {0, }} denotes whether safety constraint is satisfied at step t, and ry (B(e)) is the score of cost term with weight wy. This formulation ensures that violations of critical safety conditions will nullify the total reward, while soft planning objectives are optimized only when safety constraints are satisfied. Details of the reward implementation are provided in inAppendix
Mariance-Decoulle GRPO. We adopt Group Relative Policy Optimization (GRPO) (Shao et al. 2024) to fine-tune the ego policy. GRPO eliminates the need for an explicit value function by computing relative advantages within each sampled group, reducing implementation complexity and avoiding instability caused by inaccurate value estimation.
During fine-tuning, the pre-trained trajectory predictor serves fixed reference policy Tea while the ego policy тe is updated to better align with planning principles. For each scenario, a group of G future trajectories {Y ..... yo} is sampled from the old ego policy Tear and the loss is:
G
(v)
C. C.P. (yf
Lineeune
Ag 3 DKL med] C.P.
(4)
GF
Teas
t=1
where - -F R(yg) is the cumulative advantage of token t. Standard GRPO normalizes rewards within each group as R(v?) (R(v²) mn)/on, with pnz an being the group mean and standard deviation. The KI. divergence term regularizes the update, keeping the fine-tumed policy close to the reference policy and thereby retaining numan-like behaviors learned during pre-training.
However, directly applying GRPO to trajectory planning yields limited improvements. We attribute this to key limitation of GRPO in multi-objective planning: group-wise normalization erases cross- group scale differences (Figure 4). As a result, rare safety-violation groups are normalized to have similar advantages as the abundant safe groups. Since most safe groups only exhibit minor fluc- tuations in comfort or other secondary objectives, critical safety signals become indistinguishable during optimization. Over time, the optimizer gradually shifts its focus toward these abundant non- safety objectives, leading to severe under-optimization of rare but catastrophic safety cases.
To address this issue, we propose Variance- Variance-Deccupleed GRPO (VD-GRPO), which replaces per- group normalization with centering and a fixed global scaling constant e:
(R(v)) UR)
RVD (y))
(5)
By decoupling normalization from variance, VD-GRPO preserves absolute reward scales across groups, ensuring safety-critical objectives remain dominant over secondary goals. High-variance groups, typically corresponding to rare catastrophic cases, naturally produce larger gradients, am- plifying safety signals without manual reweighting. This enables continuous improvement on rare but important safety
in late-stage training.
4 EXPERIMENTS
4.1 EXPERIMENTAL. SETUP
Datasets. We evaluate our method on the nuPlan benchmark (Caesar et al., 2021). a large-scale platform for trajectory planning in autonomous driving. The dataset contains 1,300 hours of

---

Published as a conference paper at ICLR 2026
Table I: Comparison with SOTAs on nuplan benchmark. The best result is in bold and the second
best result is underlined. *: with rule- aale-baedd post-processing. NR/R: non-reactive/reactiv mode.
Val14
Test14-hard
Test 4-random
Type
Planner
NR
NR
NR
Expert
Log-Replay
93.53
80.32
85.96
68.80
94.03
75.86
IDM (Triiber et al.
2000)
75.60
77.33
56.15
62.26
70.39
72.42
Dauner et al.
2023
92.84
92.12
65.08
75.19
90.05
91.64
PDM-Hybrid Dauner el al.
202.5
92.77
92.11
65.99
76.07
90.10
91.28
Gameformer* Fuang el al. 20233
79.94
79.78
68.70
67.05
83.88
82.05
Rule-based
PLUTO* (Cheng et al.
2024a
92.88
89.84
80.08
76.88
92.23
90.29
Hybrid
PlanAgent" (Zhngg
2024b)
93.26
92.75
72.51
76.82
Diffusion Planner (Zenng
2025
94.26
92.90
78,87
82.00
94.80
91.75
Carplanner* (Zhang et al.
2023
94.07
91.10
Plan-R (Ours)
94.72
93.54
78.46
81.70
94.64
93.71
Driver (Scheel et al.
2022
68.57
64.11
50.40
49.95
51.83
67.15
PDM-Open (Dauner el al. 2023)
53.53
54.24
33.51
35.83
52.81
57.23
PlanTF Cheng ct al.
PLUTO Cheng 20240
84.27
76.95
69.70
61.61
85.62
79.58
Learning-baad
202451
88.89
78.11
70.03
59.74
89.90
78.62
Diffusion Planner (Zheng ct al
2025
89.87
82.80
75.99
69.22
89.19
82,93
Plan-R1 (Ours)
88.98
87.69
77.45
77.20
91.23
90.04
A 1 10T
LPU 4
Spece Speed 4.4 ta
ducrupt Succe ai A
Auctago Sprcoh Spece Line 4A4 mis
Speed 4. mm
(a) Expert
(b) Pluto
(c) Diffusion Planner
(d) Plan-R1 (Ours)
Figure 2: Comparison of closed-loop ego trajectories. Trajectories are color-coded: orange indicates speeding segments, while green represents compliant motion. The expert trajectory (a) shows a clear speeding segment. Both PLUTO (b) and Diffusion Planner (c) mimic this behavior, indicating that planners trained solely on expert data tend inherit undesirable patterns. In contrast, Plan-R1 (d) avoids speeding, demonstrating the effectiveness of rule-based reinforcement learning fine-tuning.
expert driving logs collected across four cities, covering diverse and challenging urban driving sce- narios. Simulation runs for seconds using a bicycle model and an LQR controller, while surrounding agents either follow replayed tra- jectories or react through the IDM (Treiber et al.) 2000) policy. Following PLUTO (Cheng et al.) 2024a), we sample 1M training instances for autoregressive pre-training. To reduce the computa- tional cost of closed-loop rollouts during reinforcement learning, we build a smaller fine-tum fine-tuning set of 100K scenarios. Evaluation is performed on the standard Val14, Test14-randon and Test1 4ardd splits (Dauner et al. 2023 Cheng et al. 2024b) under both non-reactive and reactive settings.
Hz: the ego vehicle executes its planned trajectory
Metrics. We evaluate planning performance in closed-loop simulation using two standard metries: Non-Reactive Closed-Loop Score (NR-CLS) and Reactive Closed-Loop Score (R-CLS). NR-CLS replays logged trajectories for surrounding agents, while R-CLS uses the IDM (Treiber et al. 2000) planner to generate reactive behaviors, providing more challenging and realistic evaluation. Both metrics assess 15-second rollouts across key driving objectives such as collision avoidance and speed limit compliance, with scores ranging from 0 100 (higher is better).
4.2 COMPARISON WITH SOTAS
Quantitative results. We compare method, Plan-R1. against wide range of existing plan- ners on the nuPlan benchmark, as shown in Table I] Plan-RI matches the strongest prior method, Diffusion Planner (Zheng al. 2025), under the non-reactive setting, showing that RI. fine-tuning

---

Published as a conference paper at ICLR 2026
Table 2: Ablation study on the importance of ruled-based RL fine-tuning and VD-GRPO.
Planner
NR-CLS
Collision
TTC
Drivable
Speed
Comfort
Progress
R-CLS
Pre-training only
85.61
94.83
90.04
94.64
96.57
99.62
91.64
82.81
GRPO
88.65
93.87
91.57
96.93
99.65
99.62
94.11
88.35
VD-GRPO (Ours)
91.23
97.32
95.02
97.32
99.45
99.62
91.94
90.04
A (Ours Pre-training)
+5.62
+2.49
+4.98
+2.64
+2.88
+0.00
+0.30
+7.23
Off road
Speeding
Collision
(a) Baseline (w/o fine-tuning)
(b) Plan-Rl (w/ fine-tuning)
Figure 3: Comparison of closed-loop ego trajectories generated by the pre-trained baseline (a) and Plan-RI (b). Orange indicates speeding segments, and cyan marks static obstacles. The baseline exhibits issues such as off-road driving (left), speeding (middle), and collision with static obstacles (right), while Plan-R1 avoids these failures, producing safe and feasible trajectories.
preserves expert-like behavior. In the more challenging reactive setting, where surrounding agents dynamically respond to the ego vehicle, Plan-R1 achieves statc-of-the-ar scores 87.69 (Val14), 77.20 (Test14-hard), and 90.04 (Testl4-random), surpassing Diffusion Planner by +4.89, +7.98 and +7.11 points, respectively. This demonstrates Plan-R1's 's superior ability to handle highly interactive scenarios, thanks the dual-model design and rule-based principle alignment. Following Diffusion Planner, we also apply the existing refinement module (Sun et al. 2024) for post-processing without any parameter tuning, where Plan-RI still achieves the highest NR-CLS and R-CLS of 94.72 and 93.54 on Val14, respectively. These results validate the effectiveness of our two-stage framework: pre-training establishes strong behavioral prior, while RL fine-tuning explicitly optimizes for safety and fcasibility, enabling Plan-R1 to generate trajectories that are both robust and compliant.
Qualitative Results. We conduct case study where the expert trajectory violates the local speed limit. As shown in Figure 2] both PLUTO (Cheng et al.] (024a) and Diffusson Planner (Zheng al. 2025), trained solely expert demonstrations, replicate this speeding behavior. In contrast, PIan-RI maintains a compliant velocity throughout the rollout, demonstrating its ability to correct undesirable behaviors inherited from expert data. This highlights the effectiveness of rule-based RL fine-tuning in overcoming the limitations of expert-only training, producing more reliable behaviors.
4.3 ABLATION STUDIES
The importance of rule-based RL fine-tuning. We first evaluate the effect of adding rule-based RL fine-tuning to the pre-trained model. As shown in Table GRPO fine-tuning consistently im- proves the overall score (e.g., +3.04 NR-CLS, +5.54 R-CLS) and most individual metrics (e.g..
8

---

Published as a conference paper at ICLR 2026
(a) GRPO
Uneada
13"
20r
Garo -O- VD-CRPO
Table 3: Ablation on model ca- pacity and world model (WM) choices.
Probabiity Densty
Unosfe Group Batio
1.
(b) VD DD-GRPO
106
6.75%
20-1
Capacity WM
R-CLS
30
10
20
30
Pre-train Pre-train (2x)
82.81
Absolute Advantage Value (AA
Egoch
84.94
GT replay
87.44
Figure 5: Proportion of unsafe Figure 4: Distributions of |A] for safe vs. unsafe groups.
Reactive WM (Ours)
90.04
groups during training.
+2.29 drivable area compliance), showing that rule-based RL explicitly enhances safety awareness while also correcting undesirable behaviors, such as speeding, that are inherited from human demon- strations. Qualitative results in Figure 3 further illustrate this improvement: the pre-trained baseline exhibits off-road driving, speeding, and collision with static obstacle, while the RL fine-tuned model produces safe and feasible trajectories.
The effect of VD-GRPO. While standard GRPO significantly improves soft objectives such as progress (+2.47) and speed compliance (+3.08), it causes drop of -0.96 in the critical collision avoidance metric, which is undesirable. To investigate this issue, we analyze the distributions of absolute advantage values (I.4) for safe and unsafe (violation) groups, as shown in Figure 4 (a). Even though our reward function (Equation 3) assigns absolute priority safety indicators such as collision avoidance via multiplicative terms, the two distributions almost completely overlap in the low-advantage region, while safe groups with small variance from soft-term fluctuations produce disproportionately large advantages, and unsafe groups with higher variance yield smaller advan- tages. This stems from GRPO's s group-wise normalization, which erases reward scale differences across groups and dilutes rare yet critical safety signals.
As shown in Figure (b), VD-GRPO addresses this issue by preserving absolute reward magnitudes, allowing safety-critical (unsafe) groups to naturally produce larger advantages and remain dominant throughout training. We further track the proportion of unsafe groups during training (Figure 5), where VD-GRPO reduces this ratio from 6.7% to 4.7% (29.8% reduction), indicating substantially safer planning behaviors. Results in Table further confirm these benefits: collision avoidance +3.45, drivable area compliance +0.39, NR-CLS +2.58, and R-CLS +1.69 compared GRPO. These findings demonstrate that VD-GRPO effectively amplifies rare yet critical safety signals, con- tinuously improving safety while maintaining balanced performance on secondary objectives.
Dual-model design. We conduct an ablation study to verify the effectiveness of our dual-model design (Figure 3). Simply replaying logged trajectories of surrounding agents (GT replay) yields an R-CLS score of 87.44, which is better than the pre-trained model but substantially below our full method. This indicates that non-reactive simulation provides limited benefit, as it fails to cap- ture realistic multi-agent interactions and thus cannot fully support effective reinforcement learning fine-tuning. Replacing GT replay with a learned world model substantially improves performance: reactive world model achieves R-CLS of 90.04, demonstrating the importance of interaction- aware simulation. To isolate this effect from model capacity, we also compare against a pre-trained baseline with doubled parameters (Pre-train (2x)). While the larger model provides a modest gain of +2.13, is far smaller than the +7.23 improvement achieved by introducing the reactive world model. This confirms that the performance gain primarily stems from our dual-model design rather than network size. These results highlight the critical role of modeling responsive surrounding- agent behaviors during RL fineituning, enabling stable optimization and superior closed-loop per- formance. Finally, more results and ablation studies can be found in Apppendix A|and (andnnppeddxx B
5 CONCLUSION
We presented Plan-R1, a two-stage framework that decouples planning principle alignment from behavior learning for safe and feasible trajectory planning. In the first stage, a motion predictor is

---

Published as a conference paper at ICLR 2026
pre-trained on expert demonstrations capture diverse, Aluman-like driving behaviors. In the second stage, the ego policy is fine-tuned with rule-based rewards to explicitly align planning with principles such safety, comfort, and traffic rule compliance. To address the limitation of standard GRPO, where group-wise normalization suppresses optimization for rare but critical safety violations, we proposed
Decoupled GRPO (VD--GRPO), which preserves absolute reward magnitudes so that safety-critical objectives remain dominant throughout training. Experiments on the nuPlan benchmark demonstrate that Plan-RI significantly enhances safety and feasibility, achieving state- of-the-an art performance, particularly in challenging reactive settings.
6 ACKNOWLEDGEMENT
This work is supported by the National Natural Science Foundation of China (Nos. (Nos.6249500 62495082, 62461, and 62495084).
USE OF LARGE LANGUAGE MODELS (LLMS)
We acknowledge that large language models (LLMs) were used solely for minor manuscript edit- ing and language polishing. No LLMs were involved in data collection, labeling, model training, or generating any experimental results. The scientific contributions, including algorithm design, implementation, and evaluation, were entirely completed by authors.
ETHICS STATEMENT
Our work focuses on improving the safety and reliability of autonomous vehicle planning. While improved planning algorithms have the potential to enhance road safety, premature deployment of this technology in real-world systems could lead to safety-critical failures. To mitigate these risks, emphasize that our method evaluated solely in simulation, and any real-world deployment should undergo rigorous validation and regulatory approval. We disclose all methodological details transparently foster responsible research and enable auditing by the community.
REPRODUCIBILITY STATEMENT
We have taken several steps to ensure the reproducibility of our work. Our Plan-RI is described in detail in section All hyperparameterss, model architectures, and training configurations fully reported in Appendix D We release our source code, model checkpoints, and experi- ment scripts required reproduce our results, which are available hhtpp:: /ttthtttthu/u/// com/ XiaolongTang23 Paan R1, The nuPlan dataset used in our experiments is publicly available, and our data preprocessing pipeline is provided within the released codebase. In addition, we clude comprehensive ablation studies in subsection 4.3 and Appendix B to verify the robustness of our approach.
REFERENCES
Mayank Bansal, Alex Krizhevsky, and Abhijit Ogale. Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst. arXiv preprint arXiv:1812.03079, 2018.
Holger Caesar, Juraj Kabzan, Kok Seang Tan, Whye Kit Fong, Eric Wolff, Alex Lang, Luke Fletcher, Oscar Beijbom, and Sammy Omari. nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles. arXiv preprint arXiv:2 106. 1810, 2021.
Zhili Chen, Maosheng Ye, Shuangjie Xu, Tongyi Cao, and Qifeng Chen. Ppad: Iterative interactions of prediction and planning for end-to-end autonomous driving. In Proceedings of the European Conference Computer Vision (ECCV), 2024.
Jie Cheng, Yingbing Chen, and Qifeng Chen. Pluto: Pushing the limit of imitation learning- baased planning for autonomous driving. arXiv preprint arXiv:2404. 14327, 2024a.
10

---

Published as a conference paper at ICLR 2026
Jie Cheng, Yingbing Chen, Xiaodong Mei, Bowen Yang, Bo Bo and Ming Liu. Rethinking imitation- based planners for autonomous driving. In IEEE International Conference on Robotics and Au- tomation (ICRA), 2024b.
Marco Cusumano- -osumano-Towwer, David Hafner, Alex Hertzberg, Brody Huval, Aleksei Petrenko, Eugene Vinitsky, Erik Wijmans, Taylor Killian, Stuart Bowers, Ozan Sener, et al. Robust autonomy emerges from self-play. arXiv preprint
2025.
Daniel Dauner, Marcel Hallgarten, Andreas Geiger, and Kashyap Chitta. Parting with misconcep- tions about bearning-basee vehicle motion planning. In Conference on Robot Learning (CoRL), 2023.
Steffen Hagedorn, Marcel Hallgarten, Martin Stoll, and Alexandru Paul Condurache. The integra- tion of prediction and planning in deep learning automated driving systems: A review. IEEE Transactions on Intelligent Vehicles, 2024.
Marcel Hallgarten, Julian Zapata, Martin Stoll, Katrin Renz, and Andreas Zell. Can vehicle motion planning generalize to realistic long-tail scenarios? 2024.
Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oriented autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), 202P
Yihan Hu, Siqi Chai, Zhening Yang. Jingyu Qian, Kun Li, Wenxin Shao, Haichao Zhang, Wei Xu, and Qiang Liu. Solving motion planning tasks with a scalable generative model. In Proceedings of the European Conference Computer Vision (ECCV), 2024.
Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.066 2025.
Zhiyu Huang, Haochen Liu, and Chen Lv. Gameformer: Game-theoretic modeling and learning of transformer-based interactive prediction and planning for autonomous driving, In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Zhiyu Huang, Xinshuo Weng, Maximilian Igl, Yuxiao Chen, Yulong Cao, Boris Ivanovic, Marco Pavone, and Chen Lv. Gen-drive: Enhancing diffusion generative driving policies with reward modeling and reinforcement learning fine-tuning. arXiv preprini arXiv22410.05582. 2024.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516 2025.
Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy Liang. Spoc: Search-based pseudocode to code. Advances in Neural Information Processing Systems (NIPS), 2019.
Derun Li, Jianwei Ren, Yue Wang, Xin Wen, Pengxiang Li, Leimeng Xu, Kun Zhan, Zhongpu Xia, Peng Jia, Xianpeng Lang, et al. Finetuning generative trajectory model with reinforcement learning from human feedback. arXiv preprint 2rXiv:2503.10444, 2025.
Qifeng Li, Xiaosong Jia, Shaobo Wang, and Junchi Yan. Think2drive: Efficient reinforcement learning by thinking with latent world model for autonomous driving (in carla-v2), In Proceedings of the European Conference on Computer Vision (ECCV), 2024.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: critical perspective. arXiv preprint 2503::2503..0783, 2025.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proceedings of the International Conference Learning Representations (ICIR), 2018.
11

---

Published as a conference paper at ICLR 2026
Yiren Lu, Justin Fu, George Tucker, Xinlei Pan, Eli Bronstein, Rebecca Roelofs, Benjamin Sapp, Brandyn White, Aleksandra Faust, Shimon Whiteson, et al. Imitation not enough: Robus- tifying imitation with reinforcement learning for challenging driving scenarios. In IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), 2023.
Nigamaa Nayakanti, Rami Al-Rfou, Aurick Zhou, Kratarth Goel, Khaled S Refaat, and Benjamin Sapp. Wayformer: Motion forecasting via simple & efficient attention networks. In IEEE Inter- national Conference on Robotics and Automation (ICRA), 2023.
Jiquan Ngiam, Vijay Vasudevan, Benjamin Caine, Zhengdong Zhang, Hao-Tier Lewis Chiang, Jef- frey Ling, Rebecca Roelofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, al. Scene trans- former: A unified architecture for predicting future trajectories of multiple agents. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems (NIPS), 2022.
Jonah Philion, Xue Bin Peng, and Sanja Fidler. Trajeglish: Learning the language of driving scenar- ios. arXiv preprint arXiv:2312.04555,, 2023.
Nicholas Rhinehart, Rowan McAllister, Kris Kitani, and Sergey Levine. Precog: Prediction con- ditioned on goals in visual multi-agent settings. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
Oliver Scheel, Luca Bergamini, Maciej Wolczyk, Btažej Osiński, and Peter Ondruska. Urban driver: Learning to drive from real-world demonstrations using policy gradients. In Conference on Robot Learning (CoRL), 2022.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
Ari Seff, Brian Cera, Dian Chen, Mason Ng. Aurick Zhou, Nigamaa Nayakanti, Khaled S Refaat, Rami Al-Rfou, and Benjamin Sapp. Motionlm: Multi-agent motion forecasting language mod- eling. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Qiao Sun, Huimin Wang, Jiahao Zhan, Fan Nie, Xin Wen, Leimeng Xu, Kun Zhan, Peng Jia, Xian- peng Lang, and Hang Zhao. Generalizing motion planners with mixture of experts for autonomous driving, arXiv preprint arXlv:2410.15774.000 2024.
Charlie Tang and Russ R Salakhutdinov. Multiple futures prediction. Advances in Neural Informa- fion Processing Systems (NIPS), 2019.
Xiaolong Tang, Meina Kan, Shiguang Shan, Zhilong Ji, Jinfeng Bai, and Xilin Chen. Hpnet: Dy- namic trajectory forecasting with historical prediction attention. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Xiaolong Tang, Meina Kan, Shiguang Shan, and Xilin Chen. Rap: Role-aware joint prediction and planning in autonomous driving. IEEE Robotics and Automation Letters (RA-I.), 2025.
Martin Treiber, Ansgar Hennecke, and Dirk Helbing. Congested traffic states in empirical observa- tions and microscopic simulations. Physical review E, 2000.
Wei Wu, Xiaoxin Feng, Ziyan Gao, and Yuheng Kan. Smart: scalable multi agent real-time motion generation via next-token prediction. Advances in Neural Information Processing Systems (NIPS), 2024.
12

---

Published as a conference paper at ICLR 2026
Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan-Yee K Wong, Zhenguo Li, and Hengshuang Zhao. Drivegpt4: Interpretable end-to-end autonomous driving via large language model. IEEE Robotics and Automation Letters (RA-L), 2024.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503.144777 2025.
Dongkun Zhang, Jiaming Liang, Ke Guo, Sha Lu, Qi Wang, Rong Xiong, Zhenwei Miao, and Yue Wang. Carplanner: Consistent auto-regressive trajectory planning for large-scale reinforcement learning in autonomous driving. arXiv preprint arXiv:2502. 19908, 2025.
Wenzhao Zheng, Ruiqi Song, Xianda Guo, Chenming Zhang, and Long Chen. Genad: Generative end-to-end autonomous driving. In Proceedings of the European Conference Computer Vision (ECCV), 2024a.
Yinan Zheng, Ruiming Liang, Kexin Zheng, Jinliang Zheng, Liyuan Mao, Jianxiong Li, Weihao Gu, Rui Ai, Shengbo Eben Li, Xianyuan Zhan, al. Diffusion- based planning for autonomous driving with flexible guidance. arXiv preprint arXiv:2501. 15564, 2025.
Yupeng Zheng. Zebin Xing, Qichao Zhang, Bu Jin, Pengfei Li, Yuhang Zheng, Zhongpu Xia, Kun Zhan, Xianpeng Lang. Yaran Chen, et al. Planagent: A multi-modal large language agent for closed-loop vehicle motion planning. arXiv preprint arXiv:2466.00 200.01187. 2024b.
Zikang Zhou, Jianping Wang, Yung-Hui Li, and Yu-Kai Huang. Query-centric trajectory predic- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
13

---

Published as a conference paper at ICLR 2026
A VISUALIZATION OF CLOSED-LOOP PLANNING RESULTS
Figure 6: Closed-loop planning results: each row represents scenario at 0, 5, 10, and 15 seconds intervals. Each frame includes the future planning of the ego vehicle, predictions for neighboring agents.
B ADDITIONAL RESULTS
Pass @ analysis. To further assess the reliability of our method under multi-sampling conditions, we conduct a @ (Kulal et al., 2019) analysis which measures the probability that at least one out of k generated samples is successful. In our context, a "successful" trajectory is defined one that satisties all core safety and feasibility criteria, including staying within drivable areas, avoiding collisions, respecting speed limits, and maintaining comfort. For both the pretrained baseline and our Plan-R1, we generate trajectories per scenario and compute the proportion of cases where at least one trajectory meets these criteria. As shown in Figure 7 pass @ improves with increasing for both models. However, Plan-RI consistently outperforms the bascline, indicating yields high-quality trajectories with greater likelihood. This suggests that fine-tuning not only improves
14

---

Published as a conference paper at ICLR 2026
0.75
0.70
0.65
Pass@k
0.60
Table 4: Ablation on Group Size G.
0.55
G
NR-CLS
R-CLS
GPU Memory (GB)
0.50
Baseline
90.60
89.81
12
0.45
Plan-R1
(used)
91.23
90.04
24
2
16
32
6
91.12
89.35
36
Number of Samples (k)
Figure 7: Comparison of pass k between the baseline and Plan-R1.
92
92
90
90
NR-CLS
VR-CLS
88
86
86
84
Baseline
Baseline
Plan-R1
82
Plan-RI
80
10
10-3
1
10-1
10
10-4
10-2
109
KL Regularization Weight (g)
Scaling Constant (c)
Figure 8: Effect of KL regularization weight.
Figure 9: Effect of fixed scaling constant.
the single-shot performance but also enhances the model's ability to generate feasible trajectories across multiple samples, which is essential for robust decision-making in safety-critical domains where uncertainty must be mitigated through diverse sampling.
Ablation study group size. We evaluate the impact of different group sizes (G), where G determines the number sampled trajectories per scenario for computing group-level advantages. As shown in Figure 4 increasing G from 2 to improves both NR-CLS and R-CLS. However, further increasing to 6 yields no performance gain and causes significant rise in GPU memory usage (from 24GB to 36GB). Thus, we set G 4 in all main experiments, achieving the best balance between accuracy and computational efficiency.
Effect of KL regularization and fixed scaling, We study two key hyperparameters in VD-GRPO: KL Regularization Weight (3) and fixed Scaling Constant (c). controls how closely the fine- tuned policy stays aligned with the pretrained reference policy. As shown in Fnguue 8, setting 3 too low causes the model to deviate excessively, harming generalization and stability. Conversely, an excessively high 3 prevents the policy from improving safety-critical behaviors. A moderate value (3 0.1) achieves the best trade-off between exploration and behavioral retention. The scaling constant in VD-GRPO (Equation 5) controls the magnitude of the unnormalized rewards. As shown in Figure 9 performance stable within a wide range (10-4 < c < 10-1). Setting c too low amplities gradient noise, while too high value diminishes the effect of fine-tuninig. In all experiments, we fix c = 10-1 for stable and efficient optimization.
Notably, these two hyperparameters interact in competitive manner: 3 governs the strength of the KL regularization loss, which encourages the policy to stay close to the pretrained reference, while determines the relative strength of the reinforcement learning signal that drives policy improve-
15

---

Ablation study on reward components. We further investigate the contribution different re- ward components by removing each term from the total reward function in Equation 3) As shown in Published as a conference paper at ICLR 2026
Table Ablation study on the reward components.
Reward setting
NR-CLS
Collision
TTC
Drivable
Speed
Comfort
Progress
w/o Collision
73.10
94.15
74.33
96.56
99.55
99.23
94.15
w/o Drivable
88.88
96.17
94.25
95.79
99.40
99.62
92.19
w/o Speed
90.04
94.83
92.29
98.47
92.29
99.62
98.02
w/o Comfort
90.83
96.93
93.87
97.32
99.47
99.23
93.16
w/o Progress
80.01
97.70
95.79
98.85
99.69
98.85
68.95
Full rewards (Plan-R1)
|
91.23
97.32
95.02
97.32
99.45
99.62
91.94
ment. A higher 3 implicitly suppresses the influence of the RL objective, making the optimization more conservative, whereas larger e amplifies the RL gradients and pushes the policy toward more aggressive exploration and deviation from the reference. Thus, tuning these two hyperparameters involves a delicate trade-off between stability and exploration, where ensures retention of safe, human-like behaviors, while enables sufficient optimization pressure to achieve safety-critical im- provements.
Table 5
each component plays a distinct role:
Collision term is the most critical: removing it causes NR-CLS to drop sharply to 73.10,
demonstrating that explicit collision avoidance is essential for safe planning.
Drivable area compliance has strong effect on feasibility, with removal leading to notice- able decrease in Drivable metric.
Speed limit compliance and Comfort terms provide secondary benefits, encouraging smoother and rule-compliant driving behaviors.
Progress term strongly affects trajectory efficiency: without it, the model tends to stay stationary or overly conservative, resulting in a low NR-CLS of 80.01.
These results confirm that our reward design balances safety and efficiency, with each component playing complementary role.
To further assess the robustness of our Plan-R1 under challenging and per- turbed scenarios, we additionally evaluate the model on the interPlan benchmark Hallgarten et al. (2024) under all 335 scenarios. InterPlan has recently been adopted by some planning works (Sun et al. (2024) as an auxiliary robustness testhed. It is derived from nuPlan but systematically per- turbs the original scenes by adding or removing agents, inserting obstacles, or modifying goals to construct long-tail, interaction-heavy situations that stress-test planner's generalization capability.
Results on interPlan.
As shown Table 6 without post-processing, Plan-RI achieves a score 56.64, slightly below PLUTO (57.74) but outperforming all other baselines. This is expected because interPlan perturbs scenes by adding or removing agents and obstacles an augmentation strategy that aligns closely with PLUTO's contrastive training pipeline, naturally favoring it. With post-pracessing. Plan-R R achieves the highest overall score (72.33), outperforming PDM-Closed by +2.69. These results demonstrate that Plan-RI exhibits strong robustness under perturbed and long-tai scenarios, further validating its effectiveness beyond the standard nuPlan benchmark.
Robustness of the Frozen World Model Under Ego-State Perturbations. A key requirement of our dual-model rollout is that the frozen world model must remain reliable even when the ego policy deviates from expert demonstrations. If the world model were highly sensitive to ego-distribution shift, inaccuracies in surrounding compounding errors, weakening the effectiveness of RL fine-tuning. To assess this potential issue, we conduct an additional robustness experiment that evaluates the stability of the frozen world model under controlled perturbations of the ego state. Specifically, inject zero-mean Gaussian noise
predictions could accumulate during rollouts and introduce
16

---

Published as a conference paper at ICLR 2026
Table Comparison with SOTAs on interPlan benchmark. The best result in bold and the second best result is underlined. *: with uue-based post- post-processing.
Type
Planner
interPlan
Expert
Log-Replay
14.76
IDM (Treiber et al. 2000)
47.07
PDM-Closed" (Dauner cf al. 2023
69.64
Rule-based & Hybrid
PLUTO* Cheng el al. 2024a
63,88
Plan-R1 (Ours)
72.33
UrbanDriver (Scheel et al. 2022
5,56
PDM-Open (Dauner et al. 2023
26.22
PlanTF (Cheng et al. 2024b)
47.72
Learning- based
PLUTO (Cheng et al.
20224a
57.74
Diffusion Planner (Zheng et al.
2025
50.07
Plan-R1 (Ours)
56.64
of increasing magnitude (a 0, 0.1, 0.3, 1,3, 10m) into the ego's observed historical and current states, and measure the resulting prediction errors for surrounding agents using ADE and FDE.
As shown in Table the frozen world model highly stable under ego-distribution shifts. For realistic ego deviations (a < 1m), degradation is minimal: ADE increases by only +0.11 m and FDE by +0.34 m. For larger perturbations (a 2 Imm, errors quickly saturate rather than diverge. Even under an extreme perturbation of 10m (far beyond any plausible deviation), the FDE increases by only +0.58 m, with no indication of exploding or compounding drift. This confirms that the world model provides reliable surrounding-agen -agent trajectories distorted. This robustness arises because the world model conditions on multiple complementary cues each agent's own motion history, interaction context, and map topology of which the ego state is only one component, As a result, moderate ego deviations do not dominate the multi-agent representation, and even extreme deviations are effectively anchored by the consistent behavior of surrounding agents and road structure, preventing instability during multi-step rollouts.
when the ego input is significantly
Table 7: Robustness of the frozen world model under ego-state perturbations. Gaussian noise with standard deviation a is injected into ego states.
Noise a (m)
ADE (m) 4
FDE (m) +
1.03
3,01
0.1
1.07
3.15
0.3
1.14
3.35
1.0
1.22
3.53
3.0
1,25
3.59
10.0
1.26
3.59
C RULE-BASED REWARD DESIGN
We design the total reward as product of two components: a set of binary safety indicators and weighted sum of soft cost terms. This formulation first ensures that no reward is given if any critical safety condition is violated, and then encourages the agent to optimize for smoother and more efficient trajectories when safety satisfied. Below, we describe the components in detail.
Safety indicators (Isate). The following constraints must all be satisfied receive a non-zero reward:
Driving area compliance: All predicted positions must lie within the drivable area polygon.
No collision with dynamic agents: At each prediction step, the ego vehicle must not collide with the predicted positions of any surrounding agent the corresponding time step.
17

---

Published as a conference paper at ICLR 2026
No collision with static obstacles: All predicted positions must avoid collisions with static ob- stacles.
To ensure accurate safety assessment, all collision checks are performed using the full bounding boxes of the ego vehicle and surrounding agents, rather than using center points.
Soft cost terms (Tema). When all safety indicators are satisfied, the agent receives a soft reward computed as a weighted sum of four normalized terms:
Comfort (w = 2): This term penalizes high lateral and longitudinal accelerations, angular veloc- ity, and angular acceleration. If any of these exceed predefined thresholds, the comfort score is set to 0; otherwise, it is 1.
Time-to-collision (TTC) (w 5): This term is set to 1 if the predicted TTC remains above a safety threshold; otherwise, it is set to O.
Speed limit compliance (w = 2): This term linearly penalizes the predicted speeding behavior by computing the ratio between the total over-speed distance and predefined maximum over-speed threshold. The score decreases linearly from to 0 as this ratio increases, following the same formulation in the nuPlan benchmark (Caesar al. 2021
Progress (w 1): This term measures the forward progress of the predicted trajectory along the reference route. It is normalized using the progress of the expert trajectory. The progress is computed by projecting each predicted position onto the centerline of the reference route and accumulating the forward distance along the projected path. Note that this is not an imitation loss: we do not minimize the distance between the predicted and expert trajectories, but only use the expert's final progress as a normalization reference.
Importantly, most reward components, such as collisions, comfort, and TTC, are naturally defined the token level since they depend on each predicted position or step. However, Speed limit com- pliance and Progress are inherently trajectory-level metrics because they require aggregated infor- mation over the entire predicted trajectory (e.g.. total distance traveled or cumulative overspeed distance). To integrate these trajectory-leve metrics into token-level GRPO optimization, we sim- ply normalize each trajectory-level reward to [0, and assign the same value to every token within the trajectory. This ensures that all tokens share identical signals for these terms, while maintaining consistent token-level advantage computation in GRPO. Empirically, this straightforward approach proved stable and effective, providing clear optimization signals without introducing additional vari- ance.
D IMPLEMENTATION DETAILS
Architecture. We discrete motion token vocabulary of size 1024 for each agent category (Vehicle, Pedestrian, Cyclist), where each token corresponds to 0.5-second movement segment. The ego agent shares the same token vocabulary as other vehicles in the Vehicle category. Our model adopts 6-layer Transformer decoder, each with 8 attention heads and hidden dimension of 128. At the end of each decoder layer, token prediction head applied, implemented as two-layer MLP. The parameters of an autoregressive predictor are approximately 5.05M, and total parameters of mudel are about M..
Training. During pretraining, we use teacher forcing to optimize next- coxt-motion-tokees prediction with cross-entropy loss. The model is trained for 32 epochs with a batch size of 64 using the AdamW optimizer (Loshchilov & Hutter 2018). with learning rate of 3 x 10-1 and a weight decay of x 10-4, dropout rate of 0. is applied throughout training. cosine annealing scheduler used decay the learning rate zero during pretraining. For fine-tuning, we apply GRPO with group size of = 4, KL divergence regularization weight 3 0.1, and a reduced learning rate of x 10-6 Fine-tuning is performed for 5 epochs, using the same dropout rate of 0.1 in pretraining. No data augmentation is applied during either training or fine-tuning. All experiments are conducted on 8 NVIDIA RTX 4090 GPUs.
Inference. During closed-loop simulation, both the pretrained and fine-tuned models generate tra- jectories by selecting the top-1 token at each step.
18

---

Published as a conference paper at ICLR 2026
E THEORETICAL ANALYSIS OF VD-GRPO
In this section, provide complete theoretical analysis of VD-GRPO. We focus two questions: (i) why GRPO's group-wise variance normalization is problematic in safety-critical multi oujti-obeecctve planning, and (ii) why introducing fixed scaling constant after removing variance is theoretically sound and practically necessary. We follow the notations in Section 3
E.1 PRELIMINARIES
For each scenario with context C' and planning principles P, sample group of G future trajec-
tories [Y',... yd} from the old ego policy Teaa-- Each trajectory y = {ií Vf has horizon
F and receives token-level rewards R(y(".. The fine-tuning objective (Equation is:
G GF
1
Te (vi CC, F21)
Liinetune
Ag - B DKL Tret) Teota (v) CC
(6)
t=1
where Ag is the cumulative advantage of rollout g from step t:
Ag Σ R(y2)-
Throughout the analysis, we treat the KL term as an independent regularizer: its gradient is additive and does not interact with reward normalization. Thus, to analyze the effect of normalization, focus on the RL term.
E.2 GRPO GRADIENT AND THE IMPLICIT VARIANCE WEIGHT
Group-wise normalization in GRPO. Standard GRPO normalizes rewards within each sampled group:
R(y!) UR
RGRPO
(8)
OR
where
F
F [R(y)) - 2R)2
PR
R(y
OR
(9)
F
Plugging Equation becomes:
(8) into Equation
and then into Equation 6 the RL-only gradient of GRPO
VoLORPO
GRPP (u?)
Vo log me
(10)
GF
F
(R(y)) - -1(2)
Vo log ne.
(11)
GF
[g) R
g.t
Key observation. Equation [T] shows that for each group g. the entire advantage is scaled by
1/0R. Hence GRPO implicitly applies group-level importance weight
1
WCRPO OR
(12)
In single-objective tasks (c.g., mathematical reasoning), this weighting helps equalize gradient mag- nitudes across groups. However, multi-objective planning where reward scales encode priorities, such variance-based reweighting distort optimization.
19

---

Published as a conference paper at ICLR 2026
E.3 REWARD STRUCTURE INDUCES LARGER VARIANCE FOR UNSAFE GROUPS
Our total reward (Equation 3) is:
R(
II
1x.09
Σ Euyrs(hi)
(13)
jEIcas
where 1g..g compliance), and Σj (yrr(1)) is a weighted sum of soft cost terms (e.g., comfort, speed compli- ance, progress).
10 is multiplicative safety indicator (e.g., collision avoidance, drivable
Define
ST
II kEIente
1E,0 E 10,
C7
Σ
wors(vi) E 10,
(14)
JEIcaa
that R(vE) S C₁¹.
Safe vs. unsafe groups. A safe group is one where Sy = 1 for all and all rollouts so rewards purely soft-cost:
R(y!) C7 (safe group).
(15)
unsafe group contains at least one step where S so some rewards are nullified to exactly zero:
Rly") = for some 1,
R(v?) = C for others (unsafe group).
(16)
Lemma E.1 (Variance bound for distributions the unit interval). For any random variable X [0, 1), its variance satisfies
Var () S 1
(17)
Proof. Let μ E E [0. 1]. Since S X 1. we have X2 S X, hence E[X²] S [[[] n. Therefore, Var(X) maximized at μ = 2 giving Var(X) Equality holds iff E {0, 1] a.s. and Pr(X 1)
E[ 1²² S p n²² u(1 - n). The quadratic function μ(1 is
Pr(X = 0) = 2-
Lemma E.2 (Variance of unsafe reward mixture). Consider non-trivial unsafe group where
fraction a E (0, 1) of rewards are zero and the remaining fraction (1 ai) are positive soft-cost
rewards Cg. Then the unsafe-group reward satisfies
Var(R.) = (1 1l-)))ar((( Var(C)) + a
(18)
Proof. Let
0, w.p. a
R
C2 w.p. a.
Then E = (1 - [C and (1 = (1 a) (3) Using Var(X) = [[ - [
and B[C = Var(C) + (B(CC)) yields Eq. (E.1).
Corollary E. (Unsafe groups have larger variance under task statistics). Assume: (i) C E [0.
with (C)) S 1/4 (Lemma E.1y: (ii) unsafe groups satisfy a < 0.3: and (iii) E[C 0.8 (empirically measured under the pretrained model). Then unsafe groups have strictly larger reward variance:
Var > Var(C.).
(19)
Proof. From Lemma
Var Var(C?) = ( Var(C
Using
0.3 and E C
0.8 gives (1 (E 0.7 x 0.64 0.448. Meanwhile
Lemma E. gives Var C < 0.25. Thus the bracketed term in (E.2) exceeds 0.198, implying
Var R
Var (C)) 0.
20

---

Published as a conference paper at ICLR 2026
Consequence for GRPO. By Corollary anl the GRPO weight in Equation 12 unsafe groups
satisfy oR (unsafe) > o (safe).. hence
WGRPO (unsafe) (unsa(
(GRPo(safe).
(20)
Therefore GRPO systematically down-weights safety-critical trajectories, contrary to the intended priority structure in planning.
E.4
VD-GRPO REMOVES VARIANCE- BIAS
VD-GRPO replaces Equation ( with centering and fixed scaling:
RvD R(v!) PR
(21)
where с > 0 is a global constant shared by all groups.
Plugging Equation 21 into Equation 66 the RL-only gradient becomes
F -we)
Σ 9.t
Σ (R(AA))
V ToLYD-GRO
Vo log Te.
(22)
cGF
Key property. Equation 22 shows that VD-GRPO applies the same scaling to all groups, thus removing the implicit variance weight 1/oR- Gradient magnitudes therefore reflect true reward differences: safe groups (only soft costs, small fluctuations) contribute smaller gradients, while unsafe groups retain larger gradients consistent with higher safety priority.
E.5 ROLE OF THE SCALING CONSTANT
E.5.1
THEORETICAL SOUNDNESS: C DOES NOT CHANGE OPTIMALITY
From Equation 22 introducing multiplies all advantages by the same positive scalar 1/c. Hence the update direction, stationary points, and optimal policies are unchanged.
E.5.2
PRACTICAL NECESSITY: RESTORING RL.KLL BALANCE
Because R(y() E [0, 1]. the within-grout deviation is bounded.
Lemma E.3 (Upper bound on R). For any group whose rewards lie in [o, 1), we have
< oR S 2
Proof. By Lemma E.1
Var R < 1/4 for any bounded reward, so S 1/2.
Thus GRPO implicitly amplifies RL advantages by
1
2.
(23)
ØR
The KL coefficient 3 in Equation (6) was tuned under this implicit amplification. If we remove without compensation, the RL gradient shrinks relative to 3 KL(-), causing KL to dominate and suppress policy improvement. VD-GRPO introduces < restore the RL seale to the regime implicitly expected by GRPO, while still avoiding variance-induced bias. This explains the stable performance across a wide range of e observed in Figure
E.6 SUMMARY
GRPO's variance normalization induces implicit group weight 1/0R. Under our reward de- sign, unsafe groups exhibit larger reward variance than safe groups, so GRPO down-weights safety- critical trajectories. VD-GRPO removes this bias by decoupling normalization from variance. The fixed scaling constant e is theoretically benign (does not alter optimality) and practically required to preserve the RL-KI. balance without retuning 3.
21


Reviews:
[{"review_id": "R1ii7rg3UU", "reviewer": "Reviewer_nc4d", "summary": "The paper introduces a two-stage framework called Plan-R1 for safe, feasible trajectory planning in autonomous driving. In the first stage, the model learns diverse, human-like driving behaviors via pretraining on expert data. In the second stage, the model is fine-tuned with rule-based reinforcement learning (RL) to align trajectory planning with explicit principles such as safety, comfort, and traffic rules. To address the limitation of standard GRPO (Group Relative Policy Optimization)—which can dilute safety-critical signals in multi-objective planning—Plan-R1 proposes Variance-Decoupled GRPO (VD-GRPO), which preserves absolute reward magnitudes to ensure the dominance of safety objectives. Experiments show that Plan-R1 significantly improves planning safety and feasibility on the nuPlan benchmark, especially in challenging reactive settings.", "questions": "1) In VD-GRPO, the choice of the fixed scaling constant (c) is empirical. Is there a more theoretical way to determine an optimal c, or is it highly task- and reward-design-dependent? (See p.15, lines 799–807.)  \n2) The paper notes that VD-GRPO “replaces in-group normalization with centering and fixed scaling.” Could you provide more concrete implementation details of this “fixed scaling,” and how it differs mathematically from traditional normalization methods? (See p.6, lines 302–303.)  \n3) In the ablation study of §4.3, Table 2 shows that VD-GRPO substantially improves the collision metric (+3.45) but has no effect on the comfort metric (+0.00). Could you explain in detail why VD-GRPO impacts some metrics more than others, and whether this relates to the high priority assigned to safety objectives in the reward function? (See p.8, Table 2; p.9, lines 453–458.)", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "• Clear two-stage framework: Plan-R1 decouples behavior learning from principle alignment, retaining human-like behavior while enhancing safety awareness and removing undesirable patterns present in expert data (p.1, lines 054–058).  \n• Novel VD-GRPO: In response to standard GRPO’s limitations, VD-GRPO replaces in-group normalization with centering and fixed scaling, effectively preventing rare but critical safety-violation signals from being washed out and ensuring safety-critical objectives dominate training (p.2, lines 075–083; p.6, lines 295–311).  \n• Rule-based rewards: Instead of relying on human preference data, Plan-R1 uses rule rewards that offer consistent, unbiased supervision, avoiding bias and improving scalability and reliability (p.3, lines 122–127; p.5, lines 266–269).  \n• Dual-model design: During RL fine-tuning, a trainable ego planner explores alternative decisions, while a frozen copy of the pretrained model acts as a reactive world model to predict other agents’ responses, enabling stable, interaction-aware joint prediction (p.2, lines 066–069; p.5, lines 256–265).  \n• Significant performance gains: On the nuPlan benchmark, Plan-R1 achieves state-of-the-art performance in both non-reactive and reactive settings, notably surpassing Diffusion Planner by +4.89, +7.98, and +7.11 points in the reactive setting, substantially improving safety and feasibility (p.7, Table 1, lines 370–377).  \n• Clear qualitative results: Figures 2 and 3 clearly show how Plan-R1 avoids undesirable behaviors common in pretrained or expert-data-only models, such as speeding, off-road driving, and collisions.", "weaknesses": "• Definition of pivots: The paper states that trajectories are discretized into motion tokens but does not delve into how these “pivot” points are chosen or defined, nor whether such discretization might miss key kinematic or geometric features (p.5, lines 217–223).  \n• Detailed analysis of VD-GRPO: Although VD-GRPO is proposed, there is limited theoretical analysis of how its parameters (e.g., the fixed scaling constant c) affect training dynamics; the discussion is mostly empirical (p.9, lines 453–460; p.15, lines 799–807).  \n• Missing user-uploaded images: The text mentions “Image generation: enabled,” but no actual images are provided, making it impossible to assess the quality or completeness of the figures referenced in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ueWB0bG6tg", "reviewer": "Reviewer_eJWp", "summary": "This paper presents Plan-R1, a two-stage framework that formulates trajectory planning as language modeling. The idea of decoupling behavior learning from principle alignment, inspired by LLM training paradigms, is both novel and elegant. The proposed Variance-Decoupled GRPO (VD-GRPO) effectively addresses the limitation of standard GRPO by preserving safety-critical gradients, leading to substantial performance gains on the nuPlan benchmark. The experimental section is comprehensive, including detailed ablations and clear visualizations, which convincingly demonstrate the method’s safety and feasibility improvements.\n\nHowever, the theoretical justification of VD-GRPO (especially the fixed scaling constant) could be further strengthened, and an additional evaluation on another dataset (e.g., CARLA) would help validate generalization. Overall, this is a well-written and technically sound paper with strong empirical results and a creative conceptual contribution.", "questions": "- Have the authors analyzed the learned motion tokens to see if they correspond to interpretable motion primitives or semantic driving actions?\n\n- How would Plan-R1 perform under partial observability or sensor noise conditions compared to diffusion-based planners?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper introduces a novel and well-motivated idea of framing trajectory planning as language modeling through the two-stage Plan-R1 framework. The decoupling of behavior learning and principle alignment is conceptually elegant and practically effective. The proposed VD-GRPO clearly addresses a key limitation in standard GRPO, preserving safety-critical gradients and improving rare-event optimization. Experiments on the nuPlan benchmark are extensive and convincing, with strong gains in both non-reactive and reactive settings. The writing is clear, the figures are informative, and the work demonstrates a high level of technical maturity.", "weaknesses": "While the empirical results are strong, the theoretical justification of VD-GRPO remains limited. The fixed scaling constant is treated as a hyperparameter without principled analysis of its effect on convergence or stability. The dual-model setting, with a frozen world model, may introduce distribution drift in long-term interactions. Moreover, evaluation is restricted to nuPlan; results on other benchmarks such as CARLA or Waymo would strengthen claims of generalization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rvrM7I32Aq", "reviewer": "Reviewer_5bV5", "summary": "This paper proposes Plan-R1, a two-stage trajectory planning framework for autonomous driving. A motion predictor is first pre-trained to capture diverse human-like driving behaviors, and then fine-tuned with rule-based reinforcement learning to align with safety and planning principles. The authors further identify a critical limitation in GRPO's normalization for safety-critical domains and introduce Variance-Decoupled GRPO (VD-GRPO) to preserve absolute reward magnitudes. Combined with a dual-model interactive rollout design, Plan-R1 achieves state-of-the-art performance on the nuPlan benchmark, especially under reactive closed-loop evaluation.", "questions": "Same as weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The authors clearly diagnose how per-group variance normalization in GRPO suppresses rare but safety-critical violations, and propose a principled solution that consistently enhances safety optimization without sacrificing secondary objectives.\n\n2. Using a frozen world model for surrounding-agent responses ensures interaction-aware rollouts while preventing instability in non-ego behaviors. This design proves crucial for strong performance in reactive scenarios.\n\n3. Plan-R1 achieves new state-of-the-art performance in both non-reactive and reactive nuPlan evaluations, with significant gains in safety metrics such as collisions and drivable area compliance. The ablations convincingly support the contributions of both VD-GRPO and the dual-model rollout.", "weaknesses": "1. Limited analysis of world model reliability when ego deviates from expert behavior. The frozen world model is assumed to remain accurate when the ego policy explores beyond regions well-covered by expert data. However, there is no case study or quantitative analysis showing how the world model behaves under large ego deviations or unusual interaction patterns, which could introduce compounding errors during RL fine-tuning.\n\n2. Tokenization design lacks ablation on discretization choices. The trajectory discretization process (e.g.spatial quantization granularity, or temporal segmentation interval) may significantly influence expressiveness and planner performance. The paper does not provide ablations or analysis on these design factors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1fsazLLmPo", "reviewer": "Reviewer_fE6E", "summary": "This paper studies how to use RL with rule-based rewards like GRPO to align ego planning with principles for trajectory planning models. Overall, the authors utilize a two-stage paradigm: pretraining and post-training to align with rules. They also modify GRPO into VD-GRPO, so as to remain safety-critical objectives remain dominant. Experiments are conducted on nuPlan dataset, and advantage against other previous methods are shown.", "questions": "Have you tried to use your fine-tuned model as 'world model' for other driving agents? Though I indeed recognize that this might not give too much performance uplift.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors study the interesting and important problem on whether recent advances in post-training language model, or basically GRPO, can help improve trajectory planning models. Their observations and modifications on GRPO provide clear and inspiring answer to this question. Especially, they conduct ablation study, analyzing the dominating term of advantages, and show that during GRPO training the unsafe samples are not assigned enough importance.\n2. Experiments on nuPlan prove the validity of their proposed method.\n3. The dual-model rollout, which utilizes frozen pre-trained model as 'world model' for other driving agents, enables the trained model to interact with other driving agents. This is also an interesting design choice.\nOverall, the discovery about GRPO is interesting, and related improvement method, i.e. VD-GRPO, is reasonable and shown to be effective.", "weaknesses": "1. Experiments are mainly conducted on nuPlan, not including other datasets. Given that said, experiments are thorough on nuPlan.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)