RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS-based Reinforcement Learning
Authors: Hao Gao
Shaoyu Chen
Bo Jiang
Bencheng Liao
Yiang Shi
Xiaoyang Guo
Yuechuan Pu
haoran yin
Xiangyu Li
xinbang zhang
ying zhang
Wenyu Liu
Qian Zhang
Xinggang Wang
Venue: NeurIPS 2025 poster ()
Keywords: End-to-end
Autonomous driving
Reinforcement learning
3DGS  environment
Closed-loop
Primary area: applications
TL;DR: A 3DGS-based Reinforcement Learning training paradigm for end-to-end autonomous driving

Abstract: Existing end-to-end autonomous driving (AD) algorithms typically follow the Imitation Learning (IL) paradigm, which faces challenges such as causal confusion and an open-loop gap. In this work, we propose RAD, a 3DGS-based closed-loop Reinforcement Learning (RL) framework for end-to-end Autonomous Driving. By leveraging 3DGS techniques, we construct a photorealistic digital replica of the real physical world, enabling the AD policy to extensively explore the state space and learn to handle out-of-distribution scenarios through large-scale trial and error. To enhance safety, we design specialized rewards to guide the policy in effectively responding to safety-critical events and understanding real-world causal relationships. To better align with human driving behavior, we incorporate IL into RL training as a regularization term. We introduce a closed-loop evaluation benchmark consisting of diverse, previously unseen 3DGS environments. Compared to IL-based methods, RAD achieves stronger performance in most closed-loop metrics, particularly exhibiting a 3× lower collision rate. Abundant closed-loop results are presented in the supplementary material. Code is available at https://github.com/hustvl/RAD for facilitating future research.

RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS-based Reinforcement Learning
Hao Gao 1:0
Shaoyu Chen 122.1
Bo Jiang',
Bencheng Liao
Yiang Shi!
Xiaoyang Guo
Yuechuan Pu²
Haoran Yin
Xiangyu Li
Xinbang Zhang"
Ying Zhang²
Wenyu Liu'
Qian Zhang²
Xinggang Wang"
Huazhong University of Science & Technology
2 Hoorizn Robotics
Abstract
Existing end-to-end autonomous driving (AD) algorithms typically follow the Imitation Learning (IL) paradigm, which faces challenges such causal confusion and an open-loop gap. In this work, we propose RAD, a 3DGS-based closed-loop Reinforcement Learning (RL) framework for end-to-end Autonomous Driving. By leveraging 3DGS techniques, we construct a photorcalistic digital replica of the real physical world, enabling the AD policy extensively explore the state space and learn to handle out-of-distribution scenarios through large-scale trial and error. To enhance safety, we design specialized rewards to guide the policy in effectively responding to safety-critiche events and understanding real- world causal relationships. To better align with human driving behavior, we incorporate IL into RL training as regularization term. We introduce a closed-loop evaluation benchmark consisting of diverse, previously unseen 3DGS environments. Compared to I[--based methods, RAD achieves stronger performance in most closed-loop metrics, particularly exhibiting 3x lower collision rate. Abundant closed-loop results are presented in the supplementary material. Code is available at hhttps://githuu: http://itthuub.co//uuueeeseeeee cww.hustwl/RAa.. for facilitating future research.
Introduction
End-to-end autonomous driving (AD) is currently a trending topic in both academia and industry. replaces modularized pipeline with holistic one by directly mapping sensory inputs to driving actions, offering advantages in system simplicity and generalization ability. Most existing end-to-end AD algorithms I
7 8 follow the Imitation Learning (IL) paradigm, which trains a neural network to mimic human driving behavior. However, despite their simplicity, ILsed methods face significant challenges in real-world deployment.
3
4 5
One key issue causal confusion. IL trains network to replicate human driving policies by learning from demonstrations. However, this paradigm primarily captures correlations rather than causal relationships between observations (states) and actions. As a result, II.-trained policies may struggle to identify the true causal factors behind planning decisions, leading to shortcut learning 9 e.g., merely extrapolating future trajectories from historical ones [10 [T] Furthermore, since IL training data predominantly consists of common driving behaviors and does not adequately cover long-tailed distributions, IL-trained policies tend to converge to trivial solutions, lacking sufficient sensitivity to safety-criticn events such as collisions.
Another major challenge is the gap between open-loop training and closed-loop deployment. IL policies are trained in an open-loop manner using dell-distributed driving demonstrations. However,
The work was done when Hao Gao (g hww.hust..eeeccc. cn) was an intern of Horizon Robotics. Project lead (shaoyu. wwwn@horizon.cooo auto). & Corresponding author (xgwang@hust.c edu.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) Imitation Learning for End-to-End
Imitation Learning
Limitations:
AD Policy
Human Driving Demonstrations
Open-loop gap
Real World
Supervision
Causal confusion
Sensor Data
(b) Simulator-based Reinforcement Learnin for End-to-End AD
AD Policy
Action
Limitations:
Sim2real gap
CARLA
Simulated
Reward
Simulated
Naive actor behavior
Simulator
Sensor Data
World
(c) Ours: 3DGS-based Reinforcement Learning with Imitation Learning for End-to-End AD
Reinforcement
Imitation Learning
Advantages:
Large-Scale 3DGS
Learning
AD Policy
Human Driving
Realistic digital world
Environments
Rollout
Keep similarity
Demonstrations
Model the causations
to humans
Narrow open-loop gap
Complementary
Figure Different training paradigms of end-to-end autonomous driving (AD).
real-world driving is a closed-loop process where minor trajectory errors at each step accumulate time, leading compounding errors and out-of-distribttion scenarios. IL-trained policies often struggle in these unseen situations, raising concerns about their robustness.
A straightforward solution to these problems is to perform closed-loop Reinforcement Learning (RL) training, which requires a driving environment that can interact with the AD policy. However, using real-world driving environments for closed-loop training poses prohibitive safety risks and operational costs. Simulated driving environments with sensor data simulation capabilities [12 [3] (which required for end-to-end AD) are typically built on game engines [14 [5] but fail to provide realistic sensor simulation results.
In this work, we establish a 3DGS- based [16 closed-loop RL training paradigm. Leveraging 3DGS techniques, we construct a photorealistic digital replica of the real world, where the AD policy can extensively explore the state space and learn to handle -uitt--distribution situations through large- scale trial and error. To ensure effective responses to safety-critica events and a better understanding of real-world causations, we design specialized safety-related rewards. However, RL training presents several critical challenges, which this paper addresses.
One significant challenge is the Human Alignment Problem. The exploration process in RL can lead policies that deviate from human-like behavior, disrupting the smoothness of the action sequence. To address this, we incorporate imitation learning as regularization term during RL training, helping maintain similarity human driving behavior. illustrated Fig. [ RL and IL work together to optimize the AD policy: RL enhances IL. by addressing causation and the open-loop gap, while IL improves RL by ensuring better human alignment.
Another major challenge is the Sparse Reward Problem. RL often suffers from sparse rewards and slow convergence. To alleviate this issue, we introduce dense auxiliary objectives related collisions and deviations. which help constrain the full action distribution. Additionally, we streamline and decouple the action space reduce the exploration cost associated with RL.
To validate the effectiveness our approach, we construct closed-loop evaluation benchmark com- prising diverse, unseen 3DGS environments. Our method, RAD, outperforms IL-eb approaches across most closed-loop metrics, notably achieving a collision rate that 3x lower.
The contributions of this work are summarized as follows:
We propose the first SDGS-based RL framework for training end-to-end AD policy. The reward, action space, optimization objective, and interaction mechanism are specially designed to enhance training efficiency and effectiveness.
We propose to combine RL and II synergistically optimize the end-to-end AD policy. RL com- plements II. by modeling the causations and narrowing the open-loop gap, while II. complements RL in terms of human alignment.

---

We validate the effectiveness of RAD on a closed-loop evaluation benchmark consisting of diverse, unseen 3DGS environments. RAD achieves stronger performance in closed-loop evaluation. particularly 3 lower collision rate, compared to II--based methods.
2
Related Work
Dynamic Scene Reconstruction. Implicit neural representations have been widely used in novel view synthesis and dynamic scene reconstruction, as in UniSim [17]. MARS [18] and NeuRAD [19] which leverage neural scene graphs for structured decomposition. However, their slow rendering speeds hinder real-time applications. Several recent works (20] 21 22 have demonstrated the
effectiveness of 3D Gaussian Splatting (3DGS) [16] for dynamic urban scene reconstruction. While prior works [1 22 primarily utilize reconstructed scenes for closed-loop evaluation, we go
step further by incorporating scenes reconstructed via 3DGS into the RL training loop to enhance
policy learning.
End-to-End Imitation Learning for Autonomous Driving. Recent advances in learning-based planning have shown strong potential, driven by large-scale data. UniAD [[] integrates multiple perception tasks to boost planning, while VAD
2 improves efficiency with compact vectorized
7 enhance the single-trajecton shift towards multi-modal planning with improved scoring. DiffusionDrivve g proposes truncated diffusion policy that denoises an anchored Gaussian distribution to a multi-mode driving action distribution. While existing methods largely follow an IL paradigm, we enhance it by incorporating closed-loop RI to further optimize the policy.
representations. Follow-up works [23 4
3
T0
24
25
26
27
paradigm, whereas VADv2 [6] and Hydra-MDP
Reinforcement Learning. Reinforcement Learning is promising technique that has not been fully explored. AlphaGo 22 and AlphaGo Zero (29] have demonstrated the power of Reinforcement Learning in the game of Go. Recently, OpenAl O1 30 and Deepseek-R1 31 have leveraged
Reinforcement Learning to develop reasoning abilities. RL has also been applied in autonomous
driving 332 33 35 36), though often using conppoooorealistcc simulators (e.g., CARLA [12]
or requiring perfect perception inputs. To the best of our knowledge, RAD is the first to train an end-to-end autonomous driving agent using RL a photorealistic 3DGS environment.
Combining Imitation and Reinforcement Learning. To combine the sample efficiency of IL with the robustness of RL, CADRE (37] and CIRL (38) adopt two-stage pipeline, first imitating expert demonstrations and then fine-tuning policies with RL to improve success rates in basic and complex CARLA scenarios. Lu er al. 335 and Huang er al. [39] instead perform joint and RL optimization, where IL constrains RL exploration to maintain human-like behaviors while improving robustness and rare-case success, Although these approaches demonstrate the benefits of combining IL and RL, they are still limited to non non-photorealistie simulators or rely on structured BEV representations and virtual sensor data. In contrast, RAD the first to jointly integrate IL and RL within 3DGS based photorealistic digital-twin environment, enabling fully end-to-end policy learning from real-world sensor inputs.
3 RAD
3.1
End-to-End Driving Policy
The overall framework of RAD is depicted in Fig- input, transforms the sensor data into scene token embeddings, outputs the probabilistic distribution of actions, and samples an action to control the vehicle.
2 RAD takes multi-view image sequences as
BEV Encoder. We first employ BEV encoder [40] transform multi-view image features from the perspective view to the Bird's Eye View (BEV), obtaining a feature map in the BEV space. This feature map then used to learn instance-level map features and agent features.
Map Head. Then we utilize a group of map tokens of the driving scene from the BEV feature map, including lane centerlines, lane dividers, road boundaries, arrows, traffic signals, etc.
to learn the vectorized map elements
Agent Head. Besides, a group of agent tokens 44 is adopted to predict the motion information of other traffic participants, including location, orientation, size, speed, and multi-mode future trajectories.
3

---

BEV Encoder
Perception Head
Planning Head
Image Encoder
Lateral/ Longitudinal
Data
Action Distribution
Training Stage 1
Training Stage 2
Training Stage 3
Perception Pre- Training
Planning Pre- Pre-Training
Reinforced Post- Bost-Training
Sensor Data O
Sensor Data
O
Selected Human Driving Demonstrations
BEV Encoder
BEV Encoder
Image Encoder
Initation Searning
Perception Head
Perception Head
Planning Head
State
AD Policy
Action
Releborrmmnt
Rollout
Iniiation Louering
Agent Ground Truth
Ground Truth
Humnn Driving Demonstrations
Reward
3DGS
Figure 2: Overall framework of RAD. RAD takes three-stage training paradigm. In the perception pre-training, ground-truths of map and agent are used to guide instance-level tokens to encode corresponding information. In the planning pre-training stage, large-scale driving demonstrations used to initialize the action distribution. In the reinforced post-training stage, RL and IL synergistically fine-tune the AD policy.
Image Encoder. Apart from the above instance-level map and agent tokens, we use an individual image encoder 45, 46 to transform the original images into image tokens. These image tokens provide dense and rich scene information for planning, complementary to the instance-level tokens.
Action Space. We propose a decoupled discrete action representation by separating lateral and longitudinal actions. Each action is defined over short 0.5-second horizon, assuming constant linear and angular velocities. This modeling assumption allows direct computation of control signals from the current action predictions. The resulting formulation reduces the dimensionality of the action space, facilitating more efficient policy optimization.
Planning Head. We use Excene denote the scene representation, which consists of map tokens, agent tokens, and image tokens. We initialize a planning embedding denoted as Empaan cascaded Transformer decoder takes the planning embedding Fiplan the query and the scene representation Escene as both key and value.
The output of the decoder is then combined with navigation information Enavi and ego state Estate to output the probabilistic distributions the lateral action a" and the longitudinal action a#:
π (a s) =sofmax (MLP (o( Eptan Excene)
+ Enavi
Estate)),
π (aa =ootmax (MLP( Eplan: Excene)
+ Enavi + Estane))
where Eplams Fnavis Fatate and the output of MI.P are all of the same dimension (1 x D).
The planning head also outputs the value functions (s(s and Vs)s),, which estimate the expected cumulative rewards for the lateral and longitudinal actions, respectively:
V MLP( (ppaa Escene) Ensvi + Estane);
(2)
(s(s) MLP( Eplan: Excen)) Enavi + Estute).
The value functions are used in RI training (Sec. 3.5
3.2 Training Paradigm
We adopt three-stage training paradigm: perception pre-training, planning pre-training, and rein- forced post-training, as shown in Fig. 2
Perception Pre-Training. Information in the image is sparse and low-level. In the first stage, the map head and the agent head explicitly output map elements and agent motion information, which are supervised with ground-trun truth labels. Consequently, map tokens and agent tokens implicitly encode

---

3DGS environments parallel rollout
W Worker-N
Selected Human Driving Demonstrations
IL- IL-training
W ookker 2
steps
orker-1
AD Policy
AD Policy
Rollout
3DGS
Record
3DGS-based Rollout Data (5 "c-11-S+11--
RL-training steps
Update
Figure 3: Post-training. N workers parallelly run. The generated rollout data (st, at, Tt+-1 st......)) are recorded in a rollout buffer. Rollout data and human driving demonstrations are used in RL- and II-training steps to fine-tune the AD policy synergistically.
the corresponding high-level information. In this stage, we only update the parameters of the BEV encoder, the map head, and the agent head.
Planning Pra-Trainnng. the second stage, to prevent the unstable cold start of RL training, IL. is first performed to initialize the probabilistic distribution of actions based on large-scale real-world driving demonstrations from expert drivers. this stage, we only update the parameters of the image encoder and the planning head, while the parameters of the BEV encoder, map head, and agent head are frozen. The optimization objectives of perception tasks and planning tasks may conflict with each other. However, with the training stage and parameters decoupled, such conflicts are mostly avoided.
Reinforced Tosinnrain In the reinforced post-training, RL and IL synergistically fine-tune the distribution. RL aims to guide the policy to be sensitive to critical risky events and adaptive to aut-of-distribution situations. II. serves as the regularization term to keep the policy's behavior similar to that of humans.
We select large number of risky, dense-traffic clips from collected driving demonstrations, and for each clip, train an independent 3DGS model to reconstruct it as digital driving environment. As illustrated in Fig. 3 deploy N parallel workers. Each worker randomly samples 3DGS environment, performs a rollout where the AD policy controls the ego vehicle to interact with the environment, and stores the resulting trajectories (st, at, ft+1, S(+1, ...) in a shared buffer.
For policy optimization, we alternate between RL and IL steps. In RL steps, the Proximal Policy Optimization (PPO) [47] is used to update the policy from the rollout buffer. In IL steps, expert demonstrations are used for supervised updates. After a fixed number of steps, the updated policy is synchronized across all workers mitigate distributional shift. During training, only the image encoder and planning head are updated, while the BEV encoder, map head, and agent head are kept frozen. The detailed RL design introduced below.
3.3 Interaction Mechanism between AD Policy and 3DGS Environment
In the 3DGS environment, the ego vehicle acts according to the AD policy. Other traffic participants can be controlled in many manners: 1) using a smart agent model: 2) using Intelligent Driver Model (IDM) for route tracking; 3) log-replay with real-world data. In order to fully recover the real-world traffic flow, we use log-replay to control other traffic participants in experiments. conventional kinematic bicycle model is employed to iteratively update the ego vehicle's pose at every At seconds as follows:
Ve tan (6.) At,
(3) where xt and yt" denote the position of the ego vehicle relative to the world coordinate; viz" is the heading angle that defines the vehicle's 's orientation with respect the world æ-coordinate; 8 the linear velocity of the ego vehicle; ó, is the steering angle of the front wheels; and L is the wheelbase, i... the distance between the front and rear axles.
21t = x."h + 21 cos (1b²)) At, Vit1 = VE" + Us sin (V.T) At, W = 1/
+
L
During the rollout process, the AD policy outputs actions (a7 a²) for a 0.5-second time horizon at time step t. We derive the linear velocity it and steering angle & based on (af a.). Based on the kinematic model in Eq. 3 the pose of the ego vehicle in the world coordinate system is updated from Pt (xx" yt 1/2" to Pt++ (x vi+1: 1/ 1))
Based on the updated Pt+1; the 3DGS environment computes the new ego vehicle's 's state 8 The updated pose Pt+1 and state st+1 serve the input for the next iteration of the inference process.
5

---

Table 1: Comparison of training strategies in Stage
Long.
Lat.
Stage
3
CR.
DCR1
SCR,
DR.
PDR!
HDR
ADD
Jerk.
Jerk.
IL
0.229
0.211
0.018
0.066
0.039
0.027
0.238
3,928
0.103
RL
0.143
0.128
0.015
0.080
0.065
0.015
0.345
4.204
0.085
RL+IL
0.089
0.080
0.009
0.063
0.042
0.021
0.257
4.495
0.082
(1) Dynamic Collision
(2) Static Collision
(3) Positional Deviation
(4) Heading Deviation
Figure 4: Example diagram of four types of reward sources. (1) Collision with dynamic obstacle
ahead triggers rde- (2) Hitting a static roadside obstacle incurs Use- (3) Moving onto the curb triggers Tpa- (4) Drifting toward the adjacent lane triggers Thd-
The 3DGS environment also generates rewards (Sec. 3.4) according to multi-soure source information (including trajectories of other agents, map information, the expert trajectory of the ego vehicle, and the parameters of Gaussians), which are used to optimize the AD policy (Sec. 3.5).
3.4 Reward Modeling
The reward is the source of the training signal, which determines the optimization direction of RL. The reward function is designed to guide the ego vehicle's behavior by penalizing unsafe actions and encouraging alignment with the expert trajectory. It is composed of four reward components: (1) collision with dynamic obstacles, (2) collision with static obstacles, (3) positional deviation from the expert trajectory, and (4) heading deviation from the expert trajectory:
R fraes Uses Tpd Tbd
As illustrated in Fig. 4 these reward components are triggered under specific conditions. In the 3DGS environment, dynamic collision is detected if the ego vehicle's bounding box overlaps with the annotated bounding boxes of dynamic obstacles, triggering a negative reward Ta Similarly, static collision is identified when the ego vehicle's 's bounding box overlaps with the Gaussians of static obstacles, resulting negative reward rsc- Positional deviation measured as the Euclidean distance between the ego vehicle's current position and the closest point on the expert trajectory. deviation beyond predefined threshold dmax incurs a negative reward rpa- Heading deviation calculated as the angular difference between the ego vehicle's ' current heading angle thy and the expert trajectory's 's matched heading angle Weexpent deviation beyond a threshold M'max results a negative reward Tnd-
Any of these events, including dynamic collision, static collision, excessive positional deviation, or excessive heading deviation, triggers immediate episode termination. Because after such events occur, the 3DGS environment typically generates noisy sensor data, which detrimental to RL training.
3.5 Policy Optimization
In the closed-loop environment, the error in each single step accumulates over time. The aforemen- tioned rewards are not only caused by the current action but also by the actions of the preceding steps. The rewards propagated forward with Generalized Advantage Estimation (GAE) (88 to optimize the action distribution of the preceding steps.
Specifically, for each time step I, we store the current state 8t, action dt, reward rt. and the estimate of the value V(*.). Based on the decoupled action space, and considering that different rewards have different corrllations to lateral and longitudinal actions, the reward r; is divided into lateral reward rī and longitudinal reward r
r r
hd r
r ddc
(5)

---

Similarly, the value function ((s)) is decoupled into two components: (st) for the lateral di- mension and (s.) for the longitudinal dimension. These value functions estimate the expected cumulative rewards for the lateral and longitudinal actions, respectively. The advantage estimates then decomposed into components
Ar and A are computed using GAE (detailed in Appendix aligned with reward types:
A.3
A A + Apd A.
A Age.
(6)
The policy my is optimized via a modified PPO objective:
CPPO _PP0 (0) + 4 -P (0),
(7)
with independent clipping mechanisms for each dimension (full equations in Appendix A.3). The
clipped objective prevents excessively large updates to the policy parameters 0.
3.6 Auxiliary Objective
To address the sparse reward challenge in RL, we design directional auxiliary objectives that provide dense supervision for both longitudinal and lateral controls. These objectives adaptively adjust action probabilities based on real-time collision risk and trajectory deviation, offering more informative gradients during training.
Specifically, a dynamic collision auxiliary loss La penalizes acceleration and encourages deceleration when past collisions with preceding vehicles are detected. A static collision auxiliary loss L. suppresses steering toward static obstacles and promotes avoidance behavior. For trajectory alignment, the positional deviation loss Lpd encourages lateral corrections toward the reference path, while the heading deviation loss Lhd promotes angular corrections to minimize orientation error. Additional implementation details are provided in Appendix A.4
The composite optimization objective integrates the PPO policy gradient with auxiliary guidance: L()) _ (0) + > La (0) + (0) + pa + Cna
(8)
where Å1. X2, A3. and \ are weighting coefficients that balance the contributions of each auxiliary objective.
Table 2: Impact of different reward components on performance.
Dyn. Col.
Sta.
Pos.
Head. Dev.
Long.
Lat.
ID
CR
DCR,
SCR,
DR.
PDR
HDR
ADD]
Col.
Dev.
Jerk,
Jerk
0.172
0.154
0.018
0.092
0.033
0.059
0.259
4.211
0.095
V
0.238
0.217
0.021
0.090
0.045
0.045
0.241
3.937
0.098
3
/
/
0.146
0.128
0.018
0.060
0.030
0.030
0.263
3.729
0.083
4
/
/
0.151
0.142
0.009
0.069
0.042
0.027
0.303
3.938
0.079
5
0.166
0.157
0.009
0.048
0.036
0.012
0.243
3.334
0.067
6
V
v
0.089
0.080
0.009
0.063
0.042
0.021
0.257
4.495
0.082
Table 3: Impact of different auxiliary objectives performance.
ID
PPO
Obj.
Dyn. Col.
Aux. Obj.
Sta. CoL. Aux. Obj.
Pos. Dev. Obj.
Head. Dev.
Aux.
CR.
DCRJ
SCR.J
DR.
PDRJ
HDRJ
ADD]
Long.
Jork]
Jerk.
Lat.
0.047
0.178
0.163
0.015
0.151
0.101
0.050
0.501
3.906
0.085
0.137
0.125
0.012
0.157
0.145
0012
0.296
3.419
0,071
0.018
0.098
1
0.149
0.134
0.015
0.063
0.057
0.006
0.524
3,980
0.086
0.128
0.119
0.009
0.06
0.030
D.136
0.254
4.102
0.092
0.112
V
0.089
0
0.009
0.063
0.042
0.021
0.257
4.495
0.082
4 Experiments
4.1 Experimental Settings
Dataset and Benchmark. We collect 2000 hours of expert driving demonstrations in real-world conditions and generate map and agent annotations via an automated pipeline for perception pre- training. Ego-vchicle odometry is employed for planning pre-training. For reinforcement learning,
7

---

Table 4: Closed-loop quantitative comparisons with other IL--based methods the 3DGS evaluation benchmark.
Method
CR.
DCR
SCR4
DR.
PDRL
HDR
ADD
Long. Jerk.
Lat.
Jerk.
TransFuser 27
0.320
0.273
0.047
0.235
0.188
0.047
0.263
4.538
0.142
VAD 2
0.335
0.273
0.062
0.314
0.255
0.059
0.304
5.284
0.550
GenAD 3
0.341
0.299
0.042
0.291
0.160
0.131
0.265
11.37
0.320
VADv2
n
0.270
0.240
0.030
0.243
0.139
0.104
0.273
7.782
0.171
RAD
0.089
0.080
0.009
0.063
0.042
0.021
0.257
4.495
0.082
select 4305 real- world driving scenes covering diverse road types, traffic densities, and agent behaviors to ensure environmental diversity. These scenes are first reconstructed into 3DGS environments, from each of which fixed-length 8-second clip is extracted. Among these clips, 3968 are used for RL training, and the other 337 are used as closed-loop evaluation benchmarks. More training details are in the Appendix A.5
Metric. We evaluate the performance of the AD policy using nine key metries. Dynamic Collision Ratio (DCR) and Static Collision Ratio (SCR) quantify the frequency of collisions with dynamic and static obstacles, respectively, with their sum represented as the Collision Ratio (CR). Positional Deviation Ratio (PDR) measures the ego vehicle's s deviation from the expert trajectory with respect to position, while Heading Deviation Ratio (HDR) evaluates the ego vehicle's ' consistency to the expert trajectory with respect to the forward direction. The overall deviation is quantified by the Deviation Ratio (DR), defined as the sum of PDR and HDR. Average Deviation Distance (ADD) quantifies the mean closest distance between the ego vehicle and the expert trajectory before any collisions or deviations occur. Additionally, Longitudinal Jerk (Long. Jerk) and Lateral Jerk LLat. Jerk) assess driving smoothness by measuring acceleration changes in the longitudinal and lateral directions. CR, DCR, and SCR mainly reflect the policy's 's safety, and ADD reflects the trajectory consistency between the AD policy and human drivers. More details are provided in Appendix A.6
4.2 Ablation Study
To evaluate the impact of different design choices in RAD, we conduct three ablation studies. These studies highlight the importance of combining RL and IL, the role of different reward sources, and the effect of auxiliary objectives.
Training Strategy Comparison. We compare pure II., pure RI., and our mixed IL+RL strategy in reinforced post-training (Tab. [ Pure IL achieves low deviation (ADD 0.238) but suffers from high collision risk (CR 0.229). Pure RL improves safety (CR O. 0.143) but deviates more from expert behavior (ADD 0.345). Our IL+RL strategy achieves the best balance (CR 0.089, ADD 0.257), demonstrating improved safety without sacrificing behavioral fidelity.
Reward Source Analysis. We analyze the influence of different reward components (Tab. 2p. Policies trained with only partial reward terms (e.g., ID 1, 2, 3, 4, 5) exhibit higher collision rates (CR) compared the full reward setup (ID 6), which achieves the lowest CR (0.089) while maintaining a stable ADD (0.257). This demonstrates that a well-balanced reward function, incorporating all reward terms, effectively enhances both safety and trajectory consistency. Among the partial reward configurations, ID 2, which omits the dynamic collision reward term, exhibits the highest CR (0.238), indicating that the absence of this term significantly impairs the model's ability to avoid dynamic obstacles, resulting in a higher collision rate.
Auxiliary Objective Analysis. We examine the impact of auxiliary objectives (Tab. By. Compared to the full auxiliary setup (ID 8), omitting any auxiliary objective increases CR, with a significant rise observed when all auxiliary objectives safety, Notably, ID 1, which retains all auxiliary objectives but excludes the PPO objective, results in CR of 0.187. This value is higher than that of IDD 8, indicating that while auxiliary objectives help reduce collisions, they are most effective when combined with the PPO objective.
removed. This highlights their collective role in enhancing
4.3 Comparisons with Existing Methods
As presented in Tab. 4 we compare RAD with other end-to-end autonomous driving methods in the proposed 3DGS-based closed-loop evaluation. For fair comparisons, all the methods are trained with the same amount of human driving demonstrations. The 3DGS environments for the RL training in
8

---

15
10
X-100m
(1) First Frame (Real Worle)
(2) Lost Prame (Real World)
(5) Relative Position over Time
2003
Reel Word
25
105
(3) First Frame (3DGS Env.)
(4) Last Frame (3DGS Erw.)
Relative Y Position over Time
Figure 5: Consistency analysis between 3DGS environment and real-world environment. We compare the driving behaviors of the same driving policy in both environments. Subfigure (1)-(4) present the first and last frames during closed-loop evaluation in both environments. Subfigure (5)-(6) depict the temporal evolution of the ego vehicle's position.
RAD are also based on these data. RAD achieves better performance compared to IL--bsed methods most metrics. Especially in terms of CR, RAD achieves 3x lower collision rate, demonstrating that RL helps the AD policy learn general collision avoidance ability.
4.4 Consistency Analysis
To demonstrate the consistency between 3DGS environment and real-world environment, we compare the driving behaviors of the same driving policy in both environments. As illustrated in Fig. 5 subfigure (1)-(4) present the first and last frames during closed-loop evaluation in both environments. Subfigure (6) depict the temporal evolution of the ego vehicle's position. Both qualitative and quantitative results indicate a high degree behavioral consistency between the 3DGS environment and the real world environment. It shows 3DGS-basee closed-loop evaluation reflect the real- world driving performance.
TOCINI
PLAN FICANT
CAN ERONT
CAM
IL-only poicc
FRONT
CAN FRNNT
CAM PINNT
CAM SROA
RAD
Figure 6: Closed-loop qualitative comparisons between the IL-only policy and RAD in a Yield to Pedestrians scenario. The IL-only policy fails yield (Row 1), while RAD successfully yields to pedestrians (Row 2).
4.5 Qualitative Comparisons
We provide qualitative comparisons between the IL-only AD policy (without reinforced post-training) and RAD, as shown in Fig. 66 The IL-only method struggles in dynamic environments, frequently failing to avoid collisions with moving obstacles or manage complex traffic situations. contrast, RAD consistently performs well, effectively avoiding dynamic obstacles and handling challenging tasks. These results highlight the benefits of closed-loop training in the hybrid method, which enables better handling of dynamic environments. Additional visualizations are included in Fig. 7

---

Conclusion
In this work, we propose RAD, the first 3DGS- closed-loop reinforcement learning framework for end-to-end autonomous driving. We combine RL and IL, with RL complementing IL to model the causations and narrow the open-loop gap, and IL complementing RL in terms of human alignment. Complemented by a targeted reward system and auxiliary objectives, RAD achieves 3x lower collision rates than state-of-the-an IL methods, with strong performance in challenging scenarios like unprotected left-turns and dense traffic.
Limitations and future work. The effect of 3DGS still has room for improvement, particularly in rendering non-rigid pedestrians. unobserved views, and low-light scenarios. Future works will focus on addressing these issues and scaling up RL the next level.
Acknowledgement: This work was partially supported by National Natural Science Foundation of China (No. 62376102).
References
[1] Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oriented autonomous driving. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 17853-17862, 2023.
[2] Bo Jiang. Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vectorized scene representation for efficient autonomous driving. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 8340-8350, 2023.
[3] Wenzhao Zheng, Ruiqi Song, Xianda Guo, Chenming Zhang, and Long Chen. Genad: Genera- tive end-to-end autonomous driving. In ECCV, 2024.
[4] Xinshuo Weng, Boris Ivanovic, Yan Wang, Yue Wang, and Marco Pavone. Para-drive: Par- allelized architecture for real-time autonomous driving. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 15449-15458, 2024.
[5] Zhenxin Li, Kailin Li, Shihao Wang. Shiyi Lan, Zhiding Yu, Yishen Ji, Zhiqi Li, Ziyue Zhu, Jan Kautz, Zuxuan Wu, et al. Hydra-mdp: End end multimodal planning with multi-targe hydra-distillation arXiv preprint arXiv:2406. 06978, 2024.
[6] Shaoyu Chen, Bo Jiang, Hao Gao, Bencheng Liao, Qing Xu, Qian Zhang, Chang Huang, Wenyu Liu, and Xinggang Wang, Vadv2: End-to-end vectorized autonomous driving via probabilistic planning. arXiv preprint urXiv:2402. 13243, 2024.
[7] Wenchao Sun, Xuewu Lin, Yining Shi, Chuang Zhang, Haoran Wu, and Sifa Zheng. Sparsedrive: End-to-end autonomous driving via sparse scene representation. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 8795-8801. IEEE, 2025.
[8] Bencheng Liao, Shaoyu Chen, Haoran Yin, Bo Jiang, Cheng Wang, Sixu Yan, Xinbang Zhang, Xiangyu Li, Ying Zhang, Qian Zhang, al. Diffusiondrive: Truncated diffusion model for end-to-end autonomous driving. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 2037-120047, 2025.
[9] Robert Geirhos, Jörn- Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2020.
[10] Zhiqi Li, Zhiding Yu, Shiyi Lan, Jiahan Li, Jan Kautz, Tong Lu, and Jose M Alvarez. Is ego status all you need for open-loop end-to-end autonomous driving? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14864-14873, 2024.
[11] Jiang-Tian Zhai, Ze Feng, Jihao Du Yongqiang Mao, Jiang-Jiang Liu, Zichang Tan, Yifu Zhang, Xiaoqing Ye, and Jingdong Wang. Rethinking the open-loop evaluation of end-to-end autonomous driving in nuscenes. arXiv preprint arXiv:2305.1042 2023.
10

---

[12] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. Conference on robot learning, pages 1-16. PMLR, 2017.
[13] Applied Intuition. Carsim. atttps:/www.carssine com/ 2023.
[14] Epic Games. Unreal engine.
1998.
[15] Unity Technologies. Unity. hatpss//hatts::/uulttyyoo 2005.
[16] Bernhard Kerbl, Georgios Kopanas, Thomas Leimküüler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139-11 2023.
[17] Ze Yang, Yun Chen, Jingkang Wang. Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyee Yang, and Raquel Urtasun. Unisim: A neural closed-loop sensor simulator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1389-1399, 2023,
[18] Zirui Wu, Tianyu Liu, Liyi Luo. Zhide Zhong, Jianteng Chen, Hongmin Xiao, Chao Hou, Haozhe Lou, Yuantao Chen, Runyi Yang, et al. Mars: An instance-aware, modular and realistic simulator for autonomous driving. In CAAI International Conference on Artificial Intelligence. pages 3-15. Springer, 2023.
[19] Adam Tonderski, Carl Lindström, Georg Hess, William Ljunghergh, Lennart Svensson, and Christoffer Petersson. Neurad: Neural rendering for autonomous driving. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14895-14904, 2024.
[20] Yunzhi Yan, Haotong Lin, Chenxu Zhou, Weijie Wang, Haiyang Sun, Kun Zhan, Xianpeng Lang, Xiaowei Zhou, and Sida Peng. Street gaussians: Modeling dynamic urban scenes with gaussian splatting. In European Conference on Computer Vision, pages 156-173. Springer, 2024.
[21] Xiaoyu Zhou, Zhiwei Lin, Xiaojun Shan, Yongtao Wang. Deqing Sun, and Ming-Hsuan Yang. Drivinggaussian: Composite gaussian splatting for surrounding dynamic autonomous driving scenes. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 21634-21643, 2024.
[22] Hongyu Zhou, Longzhong Lin, Jiabao Wang, Yichong Lu, Dongfeng Bai, Bingbing Liu, Yue Wang, Andreas Geiger, and Yiyi Liao. Hugsim: A real-time, reoto-realistic and closed-loop simulator for autonomous driving, arXiv preprint arXiv:2412.017118 2024.
[23] Yingyan Li, Lue Fan, Jiawei He, Yuqi Wang, Yuntao Chen, Zhaoxiang Zhang, and Tieniu Tan. Enhancing end-to-end autonomous driving with latent world model. arXiv preprint arXiv:2406.. 08481, 2024.
[24] Yuqi Wang. Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14759, 2024.
[25] Xunjiang Gu, Guanyu Song, Igor Gilitschenski, Marco Pavone, and Boris Ivanovic. Producing and leveraging online map uncertainty in trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14521-14530, 2024.
[26] Zhili Chen, Maosheng Ye, Shuangjic Xu, Tongyi Cao, and Qifeng Chen. Ppad: Iterative interactions of prediction and planning for end-to-end autonomous driving. In European Conference on Computer Vision, pages 239-256. Springer, 2024.
[27] Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-modal fusion transformer for end-to-end autonomous driving. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 7077-7087, 2021.
[28] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanetot, et al. Mastering the game of go with deep neural networks and tree search. nature, 2016.
11

---

[29] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy P. Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of go without human knowledge. Nat., 2017.
[30] OpenAl. Openai ol. https::/openaii.com comoo/ 2024.
[31] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
[32] Marin Toromanoff, Emilie Wirbel, and Fabien Moutarde. End-to-end model-free reinforce- ment learning for urban driving using implicit affordances. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 7153-7162, 2020.
[33] Dian Chen, Vladlen Koltun, and Philipp Krähenbühh. Learning to drive from world on rails. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15590-15599, 2021.
[34] Zhejun Zhang, Alexander Liniger, Dengxin Dai, Fisher Yu, and Lue Van Gool. End-to-end urban driving by imitating reinforcement learning coach. In Proceedings of the IEEF/CVF international conference on computer vision, pages 15222-15232, 2021.
[35] Yiren Lu, Justin Fu, George Tucker, Xinlei Pan, Eli Bronstein, Rebecca Roelofs, Benjamin Sapp, Brandyn White, Aleksandra Faust, Shimon Whiteson, et al. Imitation is not enough: Robustifying imitation with reinforcement learning for challenging driving scenarios. In 2023 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7553-7560. IEEE, 2023.
[36] Yihan Hu, Siqi Chai, Zhening Yang, Jingyu Qian, Kun Li, Wenxin Shao, Haichao Zhang, Wei Xu, and Qiang Liu. Solving motion planning tasks with scalable generative model. In European Conference on Computer Vision, pages 386 404. Springer, 2024.
[37] Yinuo Zhao, Kun Wu, Zhiyuan Xu, Zhengping Che, Qi Lu, Jian Tang, and Chi Harold Liu. Cadre: A cascade deep reinforcement learning framework for dision-based autonomous urban driving. In Proceedings of the AAAI conference artificial intelligence, volume 36, pages 3481-3489, 2022.
[38] Xiaodan Liang, Tairui Wang, Luona Yang, and Eric Xing. Cirl: Controllable imitative rein- forcement learning for vision-based self-driving. In Proceedings of the European conference on computer vision (ECCV), pages 584 599, 2018.
[39] Zhiyu Huang, Jingda Wu, and Chen Lv. Efficient deep reinforcement learning with imitative expert priors for autonomous driving. IEEE Transactions on Neural Networks and Learning Systems,
2022.
[40] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Qiao Yu, and Jifeng Dai. Bevformer: learning bird's-eye s-eee-view representation from lidar-camera via spatiotemporal transformers. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024.
[41] Bencheng Liao, Shaoyu Chen, Yunchi Zhang, Bo Jiang, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Maptrv2: An end-to-end framework for online vectorized hd map construction. International Journal of Computer Vision, 133(3):1352-1374 2025.
[42] Bencheng Liao, Shaoyu Chen, Xinggang Wang, Tianheng Cheng, Qian Zhang, Wenyu Liu, and Chang Huang. Maptr: Structured modeling and learning for online vectorized hd map construction. arXiv preprint arXiv: /4437, 2022.
[43] Bencheng Liao, Shaoyu Chen, Bo Jiang, Tianheng Cheng, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Lane graph as path: Continuity-preservin path-wise modeling for online lane graph construction. In European Conference on Computer Vision, pages 334-351. Springer, 2024.
12

---

[44] Bo Jiang. Shaoyu Chen, Xinggang Wang, Bencheng Liao, Tianheng Cheng, Jiajie Chen, Helong Zhou, Qian Zhang. Wenyu Liu, and Chang Huang. Perceive, interact, predict: Learning dynamic and static clues for end-to-end motion prediction. arXiv preprini arXiv:2212.02/88,, 2022.
[45] Alexey Dosovitskiy. An image is worth 16x 6 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1192. 2020.
[46] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, 2016.
[47] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
[48] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint /506: 02438, 2015.
[49] Tsung Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
[50] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[51] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 11X1v:1711.05101, 2017.
13

---

A Technical Appendices
A.1 3DGS Reconstruction and Rendering Optimization
To support closed-loop training, we extend the StreetGaussian (00 framework, focusing on enhancing rendering realism and geometric accuracy, particularly for high-fidelity rendering in off-trajectory views.
First, employ mesh modeling to constrain the geometry of the road surface, constraining Gaussian spheres to the mesh surface to ensure precise road geometry from any viewpoint. Additionally, we model the sky separately to prevent confusion with foreground objects, improving rendering realism under complex lighting conditions.
For foreground objects (e.g., vehicles, pedestrians), optimize their poses during training and incorporate depth and normal consistency as supervision signals to further enhance geometric recon- struction accuracy and surface detail fidelity. These optimizations significantly improve rendering quality in novel viewpoints, especially in dynamic scenes, where object motion trajectories and surface details remain consistent with real-world observations.
These improvements allow the 3DGS environment to more effectively support closed-loop training, providing a foundation of high realism and geometric accuracy for large-scale rial-and-error learning in autonomous driving strategies.
A.2 Action Space Details
Here, provide more comprehensive explanation of the action space design. To ensure stable control and efficient learning, we define the action space over a short time horizon of 0.5 seconds. The ego vehicle's movement is modeled using discrete displacements in both the lateral and longitudinal directions.
Lateral Displacement. The lateral displacement, denoted as a", represents the vehicle's movement in the lateral direction over the 0.5-second horizon. We discretize this dimension into N₂ options, symmetrically distributed around zero allow leftward and rightward movements, with an additional option to maintain the current trajectory. The set of possible lateral displacements is:
a E {dmins
O, O,..., d
(9)
In our implementation, we use N₂ = 61, with d -0.75 m, d 0.75 m, and intermediate values sampled uniformly.
Longitudinal Displacement. The longitudinal displacement, denoted as a", represents the vehicle's movement in the forward direction over the 0.5-second horizon. Similar the lateral component, we discretize this dimension into N, options, covering range of forward displacements, including an option to maintain the current position:
a {0,
d 3
(10)
In our setup, N, 61, with di 15m, and intermediate values sampled uniformly.
A.3 Policy Optimization Details
GAE Computation Details. The advantage estimates Aü and A/ are then computed as follows:
5/- r (V((se+1) Vz(s,).
6 r [Vy(3e+11 - Vs)(s,,,
À Σ (72))6911
(11)
1=0
AV Σ (7
1=0
14

---

where o7 and of are the temporal difference errors for the lateral and longitudinal dimensions, is the discount factor, and À the GAE parameter that controls the trade-off between bias and variance.
Complete PPO Objectives. The full clipping objectives are:
cx. C E, min (pt A clip(pf. 1 1x (1.1+c.0 (2,,) c2)
(12)
F (0) E. [in (p" clip(p), - cy, ev) A
where Pf
T #) is the importance sampling ratio for the lateral dimension, p New (o [s.] the importance sampling ratio for the longitudinal dimension, cæ and cy small constants that control the clipping range for the lateral and longitudinal dimensions, ensuring stable policy updates.
(a7) s+)
x ee))
A.4 Auxiliary Objectives Details
Action Probability Decomposition. The auxiliary objectives are designed to penalize undesirable behaviors by incorporating specific reward sources, including dynamic collisions, static collisions, positional deviations, and heading deviations. These objectives are computed based on the actions s.oodd and at goold selected by the old AD policy Teas at time step t. To facilitate the evaluation of these actions, we separate the probability distribution of the action into four parts:
a,
Amye
Σ of <apiou
To(a,"
st);
A 800 y
Σ
To (a st);
(13)
Am left
Σ
[a(a ss),,
of <aa ad
Artight
Σ
To(a sc).
Here, ATTe represents the total probability of deceleration actions, Any represents the total prohability of acceleration actions, Amt represents the total probability of leftward steering actions, represents the total probability of rightward steering actions.
and A right nm
Dynamic Collision Auxiliary Objective. The dynamic collision auxiliary objective adjusts the longitudinal control action of based on the location of potential collisions relative to the ego vehicle. If a collision detected ahead, the policy prioritizes deceleration actions (a"' a. collision detected behind, encourages acceleration actions (a >> at y.old. formalize this behavior, we define directional factor fac:
if the collision is ahead,
fae {-1 if the collision is behind.
(14)
The auxiliary objective for dynamic collision avoidance is defined as:
La (0) = IE, A fae (Amga AmTeo) where Ade the advantage estimate for dynamic collision avoidance.
(15)
Static Collision Auxiliary Objective. The static collision auxiliary objective adjusts the steering control action of based on the proximity static obstacles. If the static obstacle is detected on the
left side, the policy promotes rightward steering actions (aş a, 2.oold if the static obstacle is detected on the right side, it promotes leftward steering actions (af < a, 2ldd, To formalize this behavior, we define directional factor fxc:
1
if the static obstacle is the left, the static obstacle is the right.
JNc
(16)
15

---

The auxiliary objective for static collision avoidance is defined as: Es A fsc (Am Amp
(
(17)
where Are is the advantage estimate for static collision avoidance.
Positional Deviation Auxiliary Objective. The positional deviation auxiliary objective adjusts the steering control action of based on the ego vehicle's lateral deviation from the expert trajectory. If the ego vehicle deviates leftward, the policy promotes rightward corrections (af > ai ag.old); if it deviates rightward, promotes leftward corrections (aī d; ) We formalize this with a directional factor fpd:
if ego vehicle deviates leftward, if ego vehicle deviates rightward.
fN -1
(18)
The auxiliary objective for positional deviation correction is: (p) = E, A Spa (Arriihe right Am!
(19)
where Ald estimates the advantage of trajectory alignment.
Heading Deviation Auxiliary Objective. The heading deviation auxiliary objective adjusts the steering control action of based on the angular difference between the ego vehicle's current heading and the expert's reference heading. If the ego vehicle deviates counterclockwise, the policy promotes clockwise corrections (af > at 2.oodd ); it deviates clockwise, promotes counterclockwise corrections (af < at ) formalize this behavior, we define a directional factor fua
if ego vehicle deviates clockwise, if ego vehicle deviates counterclockwise.
fna = (-1
(20)
The auxiliary objective for heading deviation correction is then defined as: AmT left
(f) = Eg [A fna (Arri
(21)
where Abd is the advantage estimate for heading alignment.
A.5 Implementation Details
In this section, summarize the training settings, configurations, and hyperparameters used in approach.
Planning Pre- Tre-Training. {aa a.) N... Each anchor corresponds to specific dieering-speed combination within the 0.5-second planning horizon. Given the ground truth vehicle position at t 0.3 s denoted as Py = (p pg), we implement normalized nearest-neighbor matching predefined anchor posi-
The action space is discretized using predefined anchors A
tions:
a² d
Pgt
i = arg min
dsr
de
dl
dz
max
(22)
a
j arg min
d
di
Based on the matched anchor indices (i. we formulate the imitation learning objective as dual focal loss 49
Lu. Lrecal (T (a s)) Lrocal (TT (a²) s))
(23)
where Liocal is focal loss for discrete action classification.
Reinforced Post- most-Training. During the training process, we use cycle where reinforcement learning (RI. and imitation learning (II.) alternate. In each full cycle, we run four rounds of RI. training, followed by one round of IL training.
16

---

Each RL training round consists of 320 iterations. Our clips are 8 seconds long with frame rate of 10Hz, meaning that 320 iterations cover four clips of data. We employ a sliding window mechanism that holds four clips of data, updated in a first-in, first-out manner. In each RI. training round, data from one new clip collected and the sliding window is updated accordingly. We find that an RL-to-II. ratio of (320 x 4) 320 = 4 yields the best results.
Training configurations. We provide detailed hyperparameters for the two main stages, Planning Pre- re--irannng and Reinforced Post- Dest-Training. Tab. [ and Tab. respectively.
Table 5: Hyperparameters used in RAD Planning Pre-Training Training stage.
contig
Planning Pre- Training
learning rate learning rate schedule
le-4
cosine decay
optimizer
AdamW (50) ST]
optimizer syper-parametrr. weight decay
81. B2. 0.9, 0,999, 1e-8
le-4
batch size training steps
512
30k
planning head dim
256
Traning GPU
128 RTX4090
Table 6: Hyperparameters used in RAD Reinforced Post- fost-Training stage.
config
Reinforced Post-Trainie
learning rate learning rate schedule optimizer
5c-6
cosine decay
Adam W 50 ST
optimizer hyper-parameters weight decay
B1, 82, 0.9, 0.999, 1c-8
le-4
RL worker number
32
batch size
II. batch size
128
GAE parameter
0.9, 0.95
clipping thresholds deviation threshold planning head dim value function dim
0.1, cy = 0.2
dmax
2.0m, 4/max 40
256
256
Traning GPU
RTX4090
A.6 Metric Details
We evaluate the performance of the autonomous driving policy using nine key metrics.
Dynamic Collision Ratio (DCR). DCR quantifies the frequency of collisions with dynamic obstacles. It is defined as:
Nac
DCR
(24)
Ni
where Nac is the number of clips in which collisions with dynamic obstacles occur, and Niotal is the total number of clips.
Static Collision Ratio (SCR). SCR measures the frequency of collisions with static obstacles and is defined as:
Noc
SCR
(25)
Niotal
where Noe is the number of clips with static obstacle collisions.
17

---

Collision Ratio (CR). CR represents the total collision frequency, given by:
=R= = DCR + SCR.
(26)
Positional Deviation Ratio (PDR). PDR evaluates the ego vehicle's adherence to the expert trajectory in terms of position. is defined as:
N
PDR
(27)
Niotal
where Nut the number of clips in which the positional deviation exceeds predefined threshold.
Heading Deviation Ratio (HDR). HDR assesses orientation accuracy by computing the proportion of clips where heading deviations surpass predefined threshold:
Nid
HDR
(28)
Niotal
where Nid the number of clips where the heading deviation exceeds the threshold.
Deviation Ratio (DR), captures the overall deviation from the expert trajectory, given by:
DR PDR HDR.
(29)
Average Deviation Distance (ADD). ADD quantifies the mean closest distance between the ego vehicle and the expert trajectory during time steps when no collisions or deviations occur. It is defined as:
1 Taaf
ADD
E
dman (t),
(30)
Taafe
where safe represents the total number of time steps in which the ego vehicle operates without collisions or deviations, and (t) denotes the minimum distance between the ego vehicle and the expert trajectory time step t.
Finally, Longitudinal Jerk (Long, Jerk) and Lateral Jerk (Lat. Jerk) quantify the smoothness of vehicle motion by measuring acceleration changes. Longitudinal jerk is defined as:
d² Ulong
Jtong
(31)
di2
where Monn represents the longitudinal velocity. Similarly, lateral jerk is defined as:
d'viat
Jaat
(32)
dta
where Vhat is the lateral velocity. These metrics collectively capture abrupt changes in acceleration and steering, providing a comprehensive measure of passenger comfort and driving stability.
A.7 More Qualitative Results
Fig.
presents additional qualitative comparisons across various driving scenarios, including de- tours, crawling in dense traffic, traffic congestion, and U-turn maneuvers. The results highlight the effectiveness of our approach in generating smoother trajectories, enhancing collision avoidance, and improving adaptability in complex environments.
18

---

IL-only
policy
RAD
IL-only
policy
PRINT
RAD
IL-only
policy
RAD
IL-only policy
RAD
Figure 7: More Qualitative Results. Comparison between the II-only policy and RAD in various driving scenarios: Detour (Rows 1-2), Crawl Dense Traffic (Rows 3-4), Traffic Congestion (Rows 5-6), and U-turn(Rows 7-8).
19

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the
paper's contributions and scope?
Answer: [Yes]
Yes, the main claims in the abstract and introduction accurately reflect the paper's contributions and scope, providing clear and consistent overview of the methods and findings presented.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, the paper includes a discussion of limitations in Sec outlining potential constraints and directions for future improvement.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
20

---

Answer: [Yes]
Justification: Yes, the paper provides a complete and correct proof for each theoretical result, along with the full set of underlying assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Yes, the paper fully discloses all necessary details to reproduce the experimental results, ensuring the validity of its claims and conclusions.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
21

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Due to company confidentiality and data privacy policies, we are currently unable to release the dataset. However, we are committed to promoting reproducibility and plan to open-source the key components our model implementation after the review process.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://niis. cc/
Sublic/guides httpei/ggiiide/eeeuuubiiiseeee
CoodeSubmissionoool1
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
cilips.cc/publle http://uwlccgguudeeeeeeeeeeeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, the paper clearly specifies all experimental settings necessary to under- stand and interpret the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Yes, the paper includes appropriate information on statistical significance to ensure the reliability of the experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτor rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, the paper provides detailed information on the computational resources used to support reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
/ http://paarra..ccoppaaalcaacttaa
cc
Answer: [Yes]
Justification: Yes, the research fully adheres the NeurIPS Code of Ethics, ensuring ethical standards in data use, experimentation, and reporting.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, the paper discusses the potential societal impacts of the work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
23

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not involve data or models that pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, all creators and original owners of assets used in the paper properly credited, and their licenses and terms of use are explicitly acknowledged and respected.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, all new assets introduced in the paper well documented, with accom- panying documentation provided to facilitate their use and understanding.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
26


Reviews:
[{"review_id": "pFo6nFbMM0", "reviewer": "Reviewer_uDLu", "summary": "This work uses 3DGS techniques to create a digital replica of the real world for autonomous driving. It develops a reinforcement learning (RL) algorithm with imitation learning (IL) loss to learn driving policies through trial-and-error in the simulator. Experimental results across nine key metrics, including collision and deviation ratios, demonstrate the effectiveness of the proposed method.", "questions": "1) The training pipeline, algorithm, and action space in RAD overlap with those in [1] mentioned in the Weaknesses. A critical comparison between RAD and the works mentioned in Weaknesses, as well as [37] in the manuscript, is needed to clearly position this work.\n\n2) What does the abbreviation RAD stand for?\n\n3) Is it possible to modify visual elements like weather and car appearance in 3DGS? If so, could the authors report the results of generalization performance with these visual changes?\n\n4) Can 3DGS’s environment reconstruction offer more realistic physical interactions, such as collisions, compared to CARLA’s built-in physics engine? How can physical realism be ensured during online training?", "limitations": "YES", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 2, "strengths_and_weaknesses": "**Strengths**:\n1) This paper proposes the first 3DGS-based RL framework for training an end-to-end AD policy.\n\n2) Based on RL implementation, the proposed method achieves lower collision ratios compared with IL methods.\n\n3) This paper is well written and easy to follow.\n\n**Weakness**:\n1) This paper focuses on 3DGS simulations, which lack sufficient evaluation in more challenging scenarios like varying weather and road conditions. \n\n2) The human experience is incorporated through a vanilla imitation loss, without accounting for the complex distribution of expert data or its suboptimal performance. \n\n3) Relevant works on combining RL with IL in autonomous driving [1-4] are missing.\n\n[1] Zhao Y, Wu K, Xu Z, et al. Cadre: A cascade deep reinforcement learning framework for vision-based autonomous urban driving[C]//Proceedings of the AAAI conference on artificial intelligence. 2022, 36(3): 3481-3489.\n\n[2] Liang X, Wang T, Yang L, et al. Cirl: Controllable imitative reinforcement learning for vision-based self-driving[C]//Proceedings of the European conference on computer vision (ECCV). 2018: 584-599.\n\n[3] Lu Y, Fu J, Tucker G, et al. Imitation is not enough: Robustifying imitation with reinforcement learning for challenging driving scenarios[C]//2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2023: 7553-7560.\n\n[4] Huang Z, Wu J, Lv C. Efficient deep reinforcement learning with imitative expert priors for autonomous driving[J]. IEEE Transactions on Neural Networks and Learning Systems, 2022, 34(10): 7391-7403.", "was_revised": true, "final_justification": "Thanks to the authors for their reply. The rebuttal has addressed some of my concerns regarding the generalization performance and completeness of this work. Although the authors emphasize that the 3DGS-based framework focuses on visual reconstruction rather than low-level physical simulation, the action space for RAD consists of low-level discretized actions. This makes it unclear whether low-level metrics, such as collision rate, can reliably assess performance. Nonetheless, I acknowledge the contribution of this work in improving visual fidelity in autonomous driving simulation and will maintain my initial score.", "rebuttal": "We sincerely thank Reviewer uDLu for the valuable comments and suggestions. We especially appreciate your recognition of the key strengths of our paper, including the novelty of using a 3DGS-based reinforcement learning framework, the superior collision performance over imitation learning methods, and the overall clarity of the writing. Below we provide detailed responses to each of your concerns.\n\n\\\\(\\textcolor{red}{Question \\hspace{0.3em} 1:} \\\\)  This paper focuses on 3DGS simulations, which lack sufficient evaluation in more challenging scenarios like varying weather and road conditions.\n\n\\\\( \\textcolor{blue}{Response \\hspace{0.3em} 1:} \\\\) We thank the reviewer for the constructive comment. In response, we partition our benchmark scenarios more finely along two dimensions—weather conditions (clear vs. challenging) and road conditions (unprotected turns, narrow roads, and traffic congestion). The detailed  evaluation results are summarized below:\n\n| Weather Condition|Clear| Challenging (Rain, Night) |\n|-|:-:|:-:|\n| IL Baseline Collision Rate ↓ |24.6%|17.6%|\n| RAD Collision Rate ↓|**10.7%**|**3.5%**|\nTable R1: Collision rates under different weather conditions.\n\n| Road Conditions| Unprotected Turns | Narrow Roads | Traffic Congestion |\n|-|:--:|:-:|:-:|\n| IL Baseline Collision Rate ↓ |26.2%|47.8%|52.4%|\n| RAD Collision Rate ↓|**7.7%**|**8.7%**|**14.3%**|\nTable R2: Collision rates under different road conditions.\n\nThis validates our method's robustness under diverse environmental constraints.\n\n\\\\(\\textcolor{red}{Question \\hspace{0.3em} 2:} \\\\)  The human experience is incorporated through a vanilla imitation loss, without accounting for the complex distribution of expert data or its suboptimal performance.\n\n\\\\( \\textcolor{blue}{Response \\hspace{0.3em} 2:} \\\\) We thank the reviewer for the insightful comment. We adopt a standard imitation loss by design, and complement it with targeted data curation and policy modeling choices to capture complex and multi-modal expert behaviors effectively.\n\nTo improve data quality, we filter out suboptimal behaviors from the expert data, including instances with collision risks, illegal lane changes, or unjustified harsh braking. We also reduce the proportion of simple lane-following behaviors to ensure a more diverse and challenging training distribution.\n\nMoreover, although the imitation objective itself is unimodal, our policy outputs a distribution over discrete actions, which allows it to represent multiple plausible behaviors in ambiguous scenarios. This effectively captures the multi-modal nature of expert demonstrations.\n\nNotably, as shown in **Table 1 and Table 4 in the manuscript**, our policy—trained solely with imitation learning—achieves superior performance compared to the end-to-end baselines included in our experiments on key metrics.\n\n\\\\(\\textcolor{red}{Question \\hspace{0.3em} 3:} \\\\)  Relevant works on combining RL with IL in autonomous driving [1-4] are missing.\n\n\\\\( \\textcolor{blue}{Response \\hspace{0.3em} 3:} \\\\) We will include a discussion of [1–4] in the related work section and clarify how our approach differs from these prior methods in the revised version. We thank the reviewer for pointing out these relevant works and apologize for the omission.\n\n\\\\(\\textcolor{red}{Question \\hspace{0.3em} 4:} \\\\)  The training pipeline, algorithm, and action space in RAD overlap with those in [1] mentioned in the Weaknesses. A critical comparison between RAD and the works mentioned in Weaknesses, as well as [37] in the manuscript, is needed to clearly position this work.\n\n\\\\( \\textcolor{blue}{Response \\hspace{0.3em} 4:} \\\\) We thank the reviewer for highlighting the need for a more critical comparison between RAD and prior Imitation Learning + Reinforcement Learning (IL+RL)  frameworks. To clearly position RAD, we summarize the key distinctions below:\n\n| Work| IL Strategy|RL Strategy|IL+RL Strategy|Auxiliary Objective in RL|Action Space|Platform| End-to-End |Model Input|\n|-|-|-|-|-|-|-|-|-|\n| **References in Weakness** |||||||||\n| CADRE [1]| 1. Initialize policies| 1. Improve success rate in basic scenarios|Two-stage| ✗|Continuous| CARLA [1*] | ✓| Virtual Sensor Data from Game Engine Rendering|\n| CIRL [2]| 1. Initialize policies| 1. Improve success rate in complex scenarios| Two-stage| ✗|Continuous| CARLA [1*]| ✓| Virtual Sensor Data from Game Engine Rendering |\n| Imitation is not Enough [3]| 1. Initialize policies  2. Constrain RL to maintain human-like behaviors | 1. Improve success rate in rare scenarios  2. Enhance robustness| Joint| ✗|Continuous| Replay environment based on logged perception outputs| ✗| Structured scene information           |\n| Huang et al. [4]| 1. Initialize policies  2. Constrain RL to maintain human-like behaviors | 1. Improve success rate in complex scenarios  2. Enhance robustness| Joint|✗| Continuous| SMARTS [2*]| ✗| Structured Bird's-Eye View (BEV) Representation |\n| **References in the manuscript** |||||||||\n| Roach [37]| 1. Imitate expert behaviors and learn expert driving policies | 1. Learn policies via exploration 2. The acquired policies provide supervision for imitation learning| Two-stage|✗|Continuous| CARLA [1*]| ✓| Virtual sensor image stack|\n| **Ours (RAD)**| 1. Initialize policies  2. Constrain RL to maintain human-like behaviors | 1. Improve safety in complex real-world scenarios 2. Enhance robustness| Joint|✓| 1. Discrete  2. Horizontal and vertical movements are decoupled | Realistic 3DGS digital twin environment | ✓| Real-world Sensor Data                 |\n\nTable R3: Comparison of RAD with prior IL+RL methods.\n\nAs shown, RAD is the first to integrate IL and RL in a 3DGS-based real-world visual simulation environment with fully end-to-end policy learning from raw sensor inputs. In contrast, prior works often rely on game-engine rendering and handcrafted abstractions to simplify perception and decouple learning stages.\n\n[1*]Dosovitskiy, Alexey, et al. \"CARLA: An open urban driving simulator.\" Conference on robot learning. PMLR, 2017.\n\n[2*]Zhou, Ming, et al. \"Smarts: Scalable multi-agent reinforcement learning training school for autonomous driving.\" arXiv preprint arXiv:2010.09776 (2020).\n\n\\\\( \\textcolor{red}{Question \\hspace{0.3em} 5:} \\\\) What does the abbreviation RAD stand for?\n\n\\\\( \\textcolor{blue}{Response \\hspace{0.3em} 5:} \\\\) **RAD** stands for \"3DGS-based Closed-loop **R**einforcement Learning for End-to-End **A**utonomous **D**riving\", which serves as the name of our training framework. We sincerely apologize for the oversight in not explicitly defining the abbreviation RAD in the initial submission and will clarify this definition in the revised manuscript to avoid any ambiguity.\n\n\\\\( \\textcolor{red}{Question \\hspace{0.3em} 6:} \\\\) Is it possible to modify visual elements like weather and car appearance in 3DGS? If so, could the authors report the results of generalization performance with these visual changes?\n\n\\\\( \\textcolor{blue}{Response \\hspace{0.3em} 6:} \\\\) We thank the reviewer for this valuable question.\nIn our current 3DGS-based setup, visual elements such as weather and vehicle appearance are reconstructed directly from real-world imagery, making physically consistent modification challenging without additional generative models or rendering techniques.\n\nWhile we do not currently support explicit visual editing, our dataset naturally includes a wide range of real-world visual conditions—such as clear weather, rain, nighttime scenes, and diverse vehicle types. To evaluate the robustness of our method under these variations, we conduct performance comparisons between RAD and imitation learning (IL) baselines across different weather and road conditions. The results, summarized in **Table R1 and Table R2**, show that RAD consistently achieves lower collision rates, indicating strong generalization to naturally occurring appearance shifts.\n\nWe recognize the importance of controllable appearance and weather changes and will continue to monitor related research developments in this area.\n\n\\\\( \\textcolor{red}{Question \\hspace{0.3em} 7:} \\\\) Can 3DGS’s environment reconstruction offer more realistic physical interactions, such as collisions, compared to CARLA’s built-in physics engine? How can physical realism be ensured during online training?\n\n\\\\( \\textcolor{blue}{Response \\hspace{0.3em} 7:} \\\\) We thank the reviewer for the thoughtful question.\nOur 3DGS-based framework emphasizes high-fidelity visual reconstruction rather than low-level physical simulation. Unlike CARLA’s rigid-body physics engine, our approach focuses on trajectory-level decision-making, where fine-grained force modeling is unnecessary. Additionally, to evaluate physical interactions such as collisions, we perform precise geometric checks in the reconstructed 3D scene using proxy geometry, which provides effective and efficient safety assessment without requiring a full dynamics engine.\n\nWhile our current method does not require fine-grained physical modeling, improving physical realism in 3DGS environments could help simulate vehicle-environment interactions more accurately, especially for future low-level control tasks. Recent work such as PhysGaussian[3*] has begun exploring this direction, which we will continue to monitor relevant developments.\n\n[3*] Xie, Tianyi, et al. \"Physgaussian: Physics-integrated 3d gaussians for generative dynamics.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024."}, {"review_id": "I74H22H6DW", "reviewer": "Reviewer_BysM", "summary": "The source introduces RAD, a framework for end-to-end autonomous driving (AD) that combines Reinforcement Learning (RL) and Imitation Learning (IL). It addresses the limitations of traditional IL-based methods, such as causal confusion and the open-loop gap, by establishing a 3DGS-based closed-loop RL training paradigm. This approach leverages photorealistic digital replicas of the real world for large-scale trial-and-error learning, incorporating specialized reward designs for safety-critical events and IL as a regularization term to maintain human-like driving behavior. The paper demonstrates that RAD significantly improves collision avoidance and overall performance compared to existing methods, validated on a diverse 3DGS evaluation benchmark.", "questions": "* What is RAD abbreviated for? It is used in the abstract without any explanation of whether it’s an abbreviation.\n* If the author disagrees with the novelty assessment. Please elaborate how the techniques used in this work differ from previous work that also combine IL + RL.\n* If the authors can provide more specific plans on open sourcing part of the work to the research community. I am happy to raise my score if the plan can allow future research utilizing some of artifacts built in this work.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "_Strengths_\n\n* One of the paper's main strengths is the introduction of a large-scale photorealistic benchmark for end-to-end autonomous driving (AD).\nThis benchmark is built by leveraging 3DGS (3D Gaussian Splatting) techniques to construct a photorealistic digital replica of the real physical world. This enables the AD policy to explore the state space and learn to handle out-of-distribution (OOD) scenarios through large-scale trial and error.\n\n_Weaknesses_\n\n* One of the main limitations of the paper is related to the availability of its resources. The authors state that they are currently unable to release the dataset due to company confidentiality and data privacy policies. Although they are committed to promoting reproducibility and plan to open-source the key components of their model implementation after the review process. Without knowing the specific plans, it’s hard to assess how they can be used by the community for future research.\n\n* In terms of the use of RL + IL as regularization. This has been a fairly common technique used by various related online/offline RL work (with some variations, see [1,2] to name just a few) so I would consider this as a novel contribution. Having said that, the authors demonstrate the successful applications of RL+IL at scale which is useful, but again the utility is specifically limited with limited open-source plans.\n\n\n* [1] http://arxiv.org/abs/1707.08817\n* [2] https://arxiv.org/pdf/2106.06860", "was_revised": true, "final_justification": "The authors have addressed my points well and I decide to raise the score.", "rebuttal": "We sincerely thank the reviewer BysM for the thoughtful and constructive feedback.  We appreciate the recognition of our effort to improve out-of-distribution (OOD) generalization in end-to-end autonomous driving by leveraging photorealistic 3DGS-based simulation.\n\nIn addition, we appreciate the reviewer’s thoughtful questions about the novelty of our IL+RL formulation, result reproducibility, and open-source plans. We provide detailed responses below.\n\n\\\\( \\textcolor{red}{Question \\hspace{0.3em} 1:} \\\\) What is RAD abbreviated for? It is used in the abstract without any explanation of whether it’s an abbreviation.\n\n\\\\( \\textcolor{blue}{Response \\hspace{0.3em} 1:} \\\\) **RAD** stands for \"3DGS-based Closed-loop **R**einforcement Learning for End-to-End **A**utonomous **D**riving\", which serves as the name of our training framework. We sincerely apologize for the oversight in not explicitly defining the abbreviation RAD in the initial submission and will clarify this definition in the revised manuscript to avoid any ambiguity.\n\n\\\\( \\textcolor{red}{Question \\hspace{0.3em} 2:} \\\\) In terms of the use of RL + IL as regularization. This has been a fairly common technique used by various related online/offline RL work  so I would consider this as a novel contribution. If the author disagrees with the novelty assessment. Please elaborate how the techniques used in this work differ from previous work that also combine IL + RL.\n\n\\\\( \\textcolor{blue}{Response \\hspace{0.3em} 2:} \\\\) We appreciate the reviewer’s concern regarding the novelty of combining imitation learning (IL) and reinforcement learning (RL). We would like to emphasize that RAD is the first framework to integrate IL and RL within a photorealistic 3DGS-based digital twin environment reconstructed from real-world sensor data. Unlike prior works that often rely on non-photorealistic simulators or structured perception inputs, RAD enables truly end-to-end, closed-loop policy learning directly from raw sensor inputs in a realistic and diverse environment. This design better supports generalization to complex real-world scenarios.\n\nTo further clarify RAD’s distinctions and novelty, we provide a detailed comparison between RAD and prior IL+RL methods in **Table R1**, highlighting differences in learning objectives, training strategies, platforms, and sensory inputs. We hope this explanation addresses the reviewer’s concerns regarding the novelty and distinctiveness of our IL+RL integration compared to prior works.\n\n| Work| IL Strategy|RL Strategy|IL+RL Strategy|Auxiliary Objective in RL|Action Space|Platform| End-to-End |Model Input|\n|-|-|-|-|-|-|-|-|-|\n| CADRE [1*]| 1. Initialize policies| 1. Improve success rate in basic scenarios|Two-stage| ✗|Continuous| CARLA [6*] | ✓| Virtual Sensor Data from Game Engine Rendering|\n| CIRL [2*]| 1. Initialize policies| 1. Improve success rate in complex scenarios| Two-stage| ✗|Continuous| CARLA [6*]| ✓| Virtual Sensor Data from Game Engine Rendering |\n| Imitation is not Enough [3*]| 1. Initialize policies  2. Constrain RL to maintain human-like behaviors | 1. Improve success rate in rare scenarios  2. Enhance robustness| Joint| ✗|Continuous| Replay environment based on logged perception outputs| ✗| Structured scene information           |\n| Huang et al. [4*]| 1. Initialize policies  2. Constrain RL to maintain human-like behaviors | 1. Improve success rate in complex scenarios  2. Enhance robustness| Joint|✗| Continuous| SMARTS [7*]| ✗| Structured Bird's-Eye View (BEV) Representation |\n| Roach [5*]| 1. Imitate expert behaviors and learn expert driving policies | 1. Learn policies via exploration 2. The acquired policies provide supervision for imitation learning| Two-stage|✗|Continuous| CARLA [6*]| ✓| Virtual sensor image stack|\n| **Ours (RAD)**| 1. Initialize policies  2. Constrain RL to maintain human-like behaviors | 1. Improve safety in complex real-world scenarios 2. Enhance robustness| Joint|✓| 1. Discrete  2. Horizontal and vertical movements are decoupled | Realistic 3DGS digital twin environment | ✓| Real-world Sensor Data                 |\n\nTable R1: Comparison of RAD with prior IL+RL methods.\n\n[1*] Zhao Y, Wu K, Xu Z, et al. Cadre: A cascade deep reinforcement learning framework for vision-based autonomous urban driving[C]//Proceedings of the AAAI conference on artificial intelligence. 2022, 36(3): 3481-3489.\n\n[2*] Liang X, Wang T, Yang L, et al. Cirl: Controllable imitative reinforcement learning for vision-based self-driving[C]//Proceedings of the European conference on computer vision (ECCV). 2018: 584-599.\n\n[3*] Lu Y, Fu J, Tucker G, et al. Imitation is not enough: Robustifying imitation with reinforcement learning for challenging driving scenarios[C]//2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2023: 7553-7560.\n\n[4*] Huang Z, Wu J, Lv C. Efficient deep reinforcement learning with imitative expert priors for autonomous driving[J]. IEEE Transactions on Neural Networks and Learning Systems, 2022, 34(10): 7391-7403.\n\n[5*] Zhejun Zhang, Alexander Liniger, Dengxin Dai, Fisher Yu, and Luc Van Gool. End-to-end urban driving by imitating a reinforcement learning coach. 2021.\n\n[6*]Dosovitskiy, Alexey, et al. \"CARLA: An open urban driving simulator.\" Conference on robot learning. PMLR, 2017.\n\n[7*]Zhou, Ming, et al. \"Smarts: Scalable multi-agent reinforcement learning training school for autonomous driving.\" arXiv preprint arXiv:2010.09776 (2020).\n\n\\\\( \\textcolor{red}{Question \\hspace{0.3em} 3:} \\\\) If the authors can provide more specific plans on open sourcing part of the work to the research community. I am happy to raise my score if the plan can allow future research utilizing some of artifacts built in this work.\n\n\\\\( \\textcolor{blue}{Response \\hspace{0.3em} 3:} \\\\) We appreciate the reviewer’s concern and fully agree that openness is essential for enabling future research.\nWe have already identified the components we currently plan to open-source, which cover the core techniques introduced in our paper. These include, but are not limited to:\n1. IL+RL training framework, including action space design, directionally decoupled action head (x/y-axis), policy optimization objectives, and auxiliary losses.\n2. Interaction mechanisms between the policy and 3DGS environment, including action sampling, pose transformation and updates, and reward computation.\n3. Scripts to compute the evaluation metrics introduced  in our manuscript.\n\nThese components will be **released within one week after the review process concludes**. We hope this concrete open-source plan addresses the reviewer’s concerns and helps support further research in this area."}, {"review_id": "DyRSIk6wn2", "reviewer": "Reviewer_kB6h", "summary": "The paper proposes a 3DGS-based closed-loop RL paradigm for autonomous driving, constructing photorealistic environments to enable safe policy exploration and causal reasoning. The method is tested in the same environment and outperform other IL-based methods.", "questions": "What is the purpose of using separate x- and y-axis PPO optimization?\n\nReward hacking is an important issue in RL. How does RAD address this problem?", "limitations": "The paper introduces a lot of auxiliary loss, but lacks a certain ablation of the coefficient value.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths​​\n\nRL is an important technique for autonomous driving systems to surpass data limitations. This paper provides a good starting point for using RL to solve these problems.\n\nWeakness\n\nThe paper primarily employs existing techniques, including the combination of 3DGS with closed-loop rollout and PPO optimization, which offers limited novelty. Furthermore, the absence of evaluation on public benchmarks makes it challenging to properly assess the proposed method's effectiveness compared to state-of-the-art approaches.", "was_revised": true, "final_justification": "I appreciate the authors for their thorough reply. My major concerns have been addressed.\n\nAs a paper on autonomous driving applications, RAD integrates existing technologies. Although its framework and algorithms lack a certain degree of innovation, it provides a preliminary attempt at utilizing 3DGS + RL. The remaining issue may be that the paper employs a self-created benchmark, making it difficult for the community to conduct comparisons and development based on it.\n\nThus, I keep my positive rating.", "rebuttal": "We thank the reviewer kB6h for acknowledging the potential of reinforcement learning (RL) in addressing data limitations in autonomous driving. We also appreciate the reviewer’s thoughtful feedback on our method’s novelty, evaluation setting, and training design. Below, we address each of the reviewer’s concerns in detail.\n\n\\\\( \\textcolor{red}{Question \\hspace{0.3em} 1:} \\\\) The paper primarily employs existing techniques, including the combination of 3DGS with closed-loop rollout and PPO optimization, which offers limited novelty.\n\n\\\\( \\textcolor{blue}{Response \\hspace{0.3em} 1:} \\\\) We appreciate the reviewer’s comment and acknowledge that our approach builds upon established algorithms such as PPO. However, we emphasize that RAD introduces several key innovations that are critical for enabling stable and robust end-to-end training in photorealistic 3DGS environments—capabilities not supported by prior work.\n\nIn particular, we design an efficient and well-justified interaction mechanism between the policy and the 3DGS environment to enable stable closed-loop control. To further improve PPO convergence by reducing the complexity of the action space, we adopt a decoupled action space along the x- and y-axis, allowing independent optimization of longitudinal and lateral control policies.\n\nMoreover, we jointly optimize imitation and reinforcement objectives during  the reinforced post-training phase. Imitation learning guides policy exploration to maintain human-aligned behavior, while reinforcement learning models causality and reduces the open-loop gap, enhancing robustness in long-horizon and out-of-distribution scenarios. Additionally, we incorporate auxiliary objectives beyond the standard reinforcement objectives to enhance policy convergence stability and safety performance.\n\nThese design choices collectively enable scalable and robust end-to-end policy learning in complex and realistic driving environments. They allow our framework to maintain stable closed-loop control, improve training efficiency, and generalize to long-horizon and out-of-distribution scenarios.\n\n\\\\( \\textcolor{red}{Question \\hspace{0.3em} 2:} \\\\) The absence of evaluation on public benchmarks makes it challenging to properly assess the proposed method's effectiveness compared to state-of-the-art approaches.\n\n\\\\( \\textcolor{blue}{Response \\hspace{0.3em} 2:} \\\\) We appreciate the reviewer’s concern regarding the absence of evaluation on public benchmarks. To the best of our knowledge, there currently exists no publicly available benchmark that supports full closed-loop training and evaluation under photorealistic, 3DGS-reconstructed environments for end-to-end autonomous driving.\n\nWhile datasets such as nuScenes and Navsim offer useful foundations, they have limitations in terms of scenario diversity and control complexity—particularly in supporting closed-loop policy learning under realistic and challenging conditions. To address this gap, we construct a new benchmark using real-world 3DGS reconstructions, specifically selecting scenes with higher complexity (e.g., frequent turns, dense traffic). \n\n**Table R1** is a comparison of the proportion of straight and turning scenarios in our benchmark and these existing datasets:\n| Dataset       | Straight Ratio (%) | Turning Ratio (%) |\n|--------------|:--------------------:|:-------------------:|\n| nuScenes     | 92.80%             | 7.20%             |\n| Navsim       | 66.40%             | 33.60%            |\n| Our Benchmark | 52.50%               | 47.50%               |\n\nTable R1: Distribution of straight vs. turning scenarios in different datasets.\n\nFurthermore, to rigorously validate the effectiveness of our approach, we implement and evaluate several representative baseline methods on our benchmark. The comparative results, presented **in Table 4 of the manuscript**, demonstrate that our method consistently outperforms these baselines across various challenging scenarios.\n\n\\\\( \\textcolor{red}{Question \\hspace{0.3em} 3:} \\\\) What is the purpose of using separate x- and y-axis PPO optimization?\n\n\\\\( \\textcolor{blue}{Response \\hspace{0.3em} 3:} \\\\) We appreciate the reviewer’s interest in the motivation behind our use of separate PPO optimization for the x- and y-axis actions. To address the challenges of reinforcement learning in high-dimensional discrete action spaces, we adopt a decoupled action representation where longitudinal (x-axis) and lateral (y-axis) decisions are optimized independently. This formulation significantly reduces the total number of actions from a combinatorial 2D space (61 × 61 = 3721 joint actions) to two independent 1D distributions of 61 bins each, totaling only 122 actions. This design is summarized below:\n﻿\n| Action Space Design     | Number of Actions to Explore | PPO Optimization Scheme         |\n|-------------------------|:------------------------------:|--------------------------------|\n| Joint          | 61 × 61 = 3721               | Single PPO head over 2D space   |\n| Decoupled   | 61 + 61 = 122                | Separate PPO heads for x and y  |\n\nTable R2: Comparison of joint and decoupled action space designs. \n\nLarge discrete action spaces are known to increase the difficulty of RL optimization due to sparse exploration and slower convergence. By reducing the dimensionality of the action space, our approach facilitates more efficient and stable policy learning. This also enables more targeted control over longitudinal and lateral dynamics, which is particularly beneficial for learning interpretable and robust driving behaviors in complex closed-loop scenarios.\n\n\\\\( \\textcolor{red}{Question \\hspace{0.3em} 4:} \\\\) Reward hacking is an important issue in RL. How does RAD address this problem?\n\n\\\\( \\textcolor{blue}{Response \\hspace{0.3em} 4:} \\\\) We thank the reviewer for highlighting the issue of reward hacking, a well-recognized challenge in reinforcement learning (RL). It refers to situations where an agent exploits unintended shortcuts or loopholes in the reward function—maximizing numerical rewards without achieving the desired behaviors, and sometimes even inducing unsafe outcomes. This typically arises from overly simplified or misaligned reward definitions.\n\nIn RAD, we mitigate the reward hacking issue by designing a well-aligned and task-relevant reward function. Our reward signals are grounded in physically meaningful objectives that directly reflect safe and expert-aligned driving behavior, including collision avoidance and minimizing deviations in both position and heading from expert trajectories. This principled design ensures that the agent is incentivized to follow intended behaviors and reduces the likelihood of exploiting unintended shortcuts.\n\nFurthermore, RAD incorporates imitation learning (IL) as a regularization mechanism alongside reinforcement learning. The imitation component guides the policy towards expert-like behaviors, effectively constraining exploration within reasonable and human-aligned trajectories. This joint IL+RL framework also helps mitigate reward hacking by constraining the policy to stay close to expert-like behavior, reducing the chance of exploiting unintended reward signals.\n\nOverall, the combination of an objective reward design and imitation-guided exploration helps RAD maintain robust and safe driving policies without falling into reward hacking pitfalls.\n\n\\\\( \\textcolor{red}{Question \\hspace{0.3em} 5:} \\\\) The paper introduces a lot of auxiliary loss, but lacks a certain ablation of the coefficient value.\n\n\\\\( \\textcolor{blue}{Response \\hspace{0.3em} 5:} \\\\) We thank the reviewer for pointing out the absence of ablation on the auxiliary loss coefficients.  We include an ablation study in **Table 3 in the manuscript**, which demonstrates the effectiveness of the proposed auxiliary losses in enhancing policy robustness and safety-related behavior.\n\nRegarding the specific coefficients of these auxiliary losses, we would like to clarify that all weights were set to 1 uniformly across all experiments, and we did not conduct careful tuning of these coefficients. Our goal is to demonstrate the utility of these auxiliary objectives in a general setting without introducing additional hyperparameter complexity. We agree that further performance improvements may be achieved by carefully adjusting these coefficients, and we plan to explore this in future work."}, {"review_id": "3XQhflxM3c", "reviewer": "Reviewer_3GxG", "summary": "This paper introduces RAD, a 3D Gaussian Splatting (3DGS)-based reinforcement learning (RL) framework for end-to-end autonomous driving (AD). RAD addresses key limitations of imitation learning (IL)-based approaches, such as causal confusion and the open-loop training-deployment gap, by integrating RL with IL in a closed-loop training paradigm.", "questions": "1.Dynamic Obstacle Generalization: The policy is trained with log-replayed traffic participants. How does it handle unseen adversarial agents (e.g., sudden lane invasions)? Provide cross-validation on CARLA’s adversarial scenarios.\n2.Action Space Sensitivity: Why was the 0.5s horizon chosen? Ablate performance with varying horizons (0.3s, 1.0s) and anchor counts (Nx, Ny).\n3.Reward Design Robustness: The collision reward (Eq. 5) relies on annotated bounding boxes. How does RAD handle false positives/negatives in perception? Test under noisy detection inputs.\n4.Multi-Modal Planning: The action space is deterministic. Can RAD support probabilistic multi-modal planning (e.g., lane changes vs. braking)? Compare with diffusion-based policies (e.g., DiffusionDrive).", "limitations": "Generalization Beyond Training Data: Evaluation is limited to NuScenes-derived environments. Test on cross-domain datasets (e.g., Waymo, off-road).", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1.Comprehensive experiments across 3,305 scenarios with rigorous ablation studies (Tables 1-3). The hybrid RL-IL approach balances safety (CR reduction) and human alignment (ADD: 0.257).\n2.Clear pipeline description (Figs. 2-3), modular reward design (Eq. 4-6), and detailed training stages (pre-training, RL-IL synergy).\n\nWeaknesses:\n1.Action space decoupling (lateral/longitudinal) shares similarities with prior RL navigation works (e.g., NaviRL 2023).\n2.Ambiguities in action space discretization (e.g., rationale for 0.5s horizon, 61 anchors) and GAE/PPO hyperparameters (λ, γ).\n3.High computational costs (128 RTX 4090 GPUs for pre-training) limit accessibility. No discussion of sim-to-real gaps in dynamic lighting/weather.", "was_revised": true, "final_justification": "The author's experimental supplement is very sufficient and addresses my concerns, so I increase the score.", "rebuttal": "We sincerely thank the reviewer 3GxG for recognizing the strengths of our work, including the comprehensive experiments, clear pipeline design, and detailed training stages. We appreciate the constructive feedback provided and address each of the raised concerns below.\n\n\\\\( \\textcolor{red}{Question \\hspace{0.3em} 1:} \\\\) Action space decoupling (lateral/longitudinal) shares similarities with prior RL navigation works (e.g., NaviRL 2023).\n\n\\\\( \\textcolor{blue}{Response \\hspace{0.3em} 1:} \\\\) We thank the reviewer for highlighting this potentially related work. However, we could not find a publicly available publication named **NaviRL**. It may be a shorthand or typographical error. We would appreciate it if the reviewer could provide the full reference or link for proper comparison.\n\nWe would like to clarify the motivation and unique aspects of our action space decoupling strategy. We adopt a decoupled action representation where longitudinal (x-axis) and lateral (y-axis) controls are optimized independently via separate PPO heads. This decoupling reduces the total number of discrete actions from a 2D joint space of 3721 to two 1D action spaces with 122 actions, which substantially lowers the policy search space. This reduction reduces the difficulty of RL exploration by limiting the size of the action space. Furthermore, our design supports more robust control in closed-loop, photorealistic environments.\n\n\\\\( \\textcolor{red}{Question \\hspace{0.3em} 2:} \\\\) Why was a 0.5s action horizon and 61 anchors chosen for action discretization? How sensitive is the performance to these parameters?\n\n\\\\( \\textcolor{blue}{Response \\hspace{0.3em} 2:} \\\\) We thank the reviewer for raising this important point regarding the design trade-off in action space discretization. Below, we elaborate in detail on how the action horizon and anchor resolution affect policy performance.\n\nA short action horizon leads to closely clustered anchors, reducing spatial diversity and limiting the ability to represent complex maneuvers such as turns. In contrast, a longer horizon increases trajectory approximation errors due to the constant-motion assumption, potentially degrading control accuracy. A similar trade-off exists in the spatial resolution of anchors: a smaller number limits the agent’s ability to perform fine-grained control, while an overly fine resolution increases the exploration burden for RL and slows convergence.\n\nTo assess the impact of these choices, we conducted an ablation by varying the action horizon and anchor resolution. For each setting, we performed closed-loop control using the corresponding discrete action space and measured the average position error relative to the expert trajectory.\n\nAs shown in **Table R1 and Table R2**, the 0.5s horizon and a (Nx = 61, Ny = 61) resolution provide the best balance between sufficient control precision and practical learning efficiency. We therefore adopt this configuration in our main experiments.\n\n| Horizon | 0.3s | 0.5s | 1.0s |\n|-|:-:|:-:|:-:|\n| Average Position Error ↓ | 0.44m | 0.47m| 0.84m |\n\nTable R1: Average position error at different action horizons.\n\n| Count per Axis | 31 | 61 | 121 |\n|-|:-:|:-:|:-:|\n| Average Position Error ↓ |1.65m | 0.47m | 0.29m |\n\nTable R2: Average position error at different action resolutions.\n\n\\\\( \\textcolor{red}{Question \\hspace{0.3em} 3:} \\\\) Dynamic Obstacle Generalization: The policy is trained with log-replayed traffic participants. How does it handle unseen adversarial agents (e.g., sudden lane invasions)? Provide cross-validation on CARLA’s adversarial scenarios.\n\n\\\\( \\textcolor{blue}{Response \\hspace{0.3em} 3:} \\\\) We thank the reviewer for the insightful question regarding dynamic obstacle generalization.\n\nFirst, we acknowledge that the policy is trained using log-replayed traffic participants, which represent realistic behaviors captured from real-world driving data. Importantly, our dataset itself includes a substantial number of adversarial scenarios, such as sudden lane invasions and strong interactive maneuvers, ensuring the policy is exposed to such behaviors during training.\n\nOur benchmark dataset contains complex traffic flows with frequent risky interactions. Furthermore, the training and validation sets are strictly separated, guaranteeing that the policy is evaluated on unseen and challenging scenarios.\n\nWe note that CARLA’s agents are largely rule-based, whereas our 3DGS-based benchmark reflects authentic real-world traffic flows with higher visual and behavioral fidelity. **Table R3** summarizes the performance improvements of our method over baselines across several challenging interaction types:\n\n| Metric| Sudden Lane Invasion | Pedestrian Crossing |\n|-|:-:|:-:|\n| IL Baseline Collision Rate |57.6% | 58.1% |\n| RAD Collision Rate|**6.1%**|**12.9%**|\n\nTable R3: Collision rates for sudden lane invasion and pedestrian crossing scenarios. \n\n\\\\( \\textcolor{red}{Question \\hspace{0.3em} 4:} \\\\) Reward Design Robustness: The collision reward (Eq. 5) relies on annotated bounding boxes. How does RAD handle false positives/negatives in perception? Test under noisy detection inputs.\n\n\\\\( \\textcolor{blue}{Response \\hspace{0.3em} 4:} \\\\) We appreciate the reviewer’s concern regarding the impact of perception errors on our collision-based reward.\n\nTo begin with, collision-related objects typically appear near the ego vehicle, where detection is more accurate due to proximity and minimal occlusion. Thus, the bounding boxes used are generally reliable with low noise.\n\nFurthermore, our system incorporates multi-frame context in the closed-loop process, enabling temporal consistency checks that filter out transient detection errors, including both false positives and negatives.\n\nTogether, near-field reliability and temporal fusion ensure that the reward signal remains robust, even with moderate perception noise. We will clarify this in the revised version.\n\n\\\\( \\textcolor{red}{Question \\hspace{0.3em} 5:} \\\\) Multi-Modal Planning: The action space is deterministic. Can RAD support probabilistic multi-modal planning (e.g., lane changes vs. braking)? Compare with diffusion-based policies (e.g., DiffusionDrive).\n\n\\\\( \\textcolor{blue}{Response \\hspace{0.3em} 5:} \\\\) We appreciate the reviewer’s question regarding multi-modal planning and the nature of our action space. Although RAD uses a fixed set of discrete action anchors, it produces a probability distribution over these anchors at each decision step, similar to how language models generate probabilistic outputs. Therefore, the policy inherently supports probabilistic multi-modal planning, enabling diverse behavior patterns such as lane changes versus braking to emerge naturally from the learned distribution.\n\n\\\\( \\textcolor{red}{Question \\hspace{0.3em} 6:} \\\\) Explain the choices of GAE/PPO hyperparameters (λ, γ).\n\n\\\\( \\textcolor{blue}{Response \\hspace{0.3em} 6:} \\\\) We note the reviewer’s interest in the choice of GAE/PPO hyperparameters. Our implementation adopts commonly used RL hyperparameter values. To ensure clarity and reproducibility, we have documented all related hyperparameters in **the Technical Appendices (Section A.5) of the manuscript**.\n\nWhile we did not perform targeted ablations on these specific values, our empirical results demonstrate stable and consistent training across a wide range of scenarios. We agree that further sensitivity analysis of these parameters could provide additional insight, and we plan to explore this in future work.\n\n\\\\( \\textcolor{red}{Question \\hspace{0.3em} 7:} \\\\) High computational costs (128 RTX 4090 GPUs for pre-training) limit accessibility.\n\n\\\\( \\textcolor{blue}{Response \\hspace{0.3em} 7:} \\\\)We thank the reviewer for highlighting the concern about computational costs. While our pre-training was conducted using 128 RTX 4090 GPUs to accelerate the experimental iteration and development process, this setup is not a strict requirement. The training can be performed on fewer GPUs with proportionally longer training times. We emphasize that the computational resources we used were chosen primarily to shorten the overall experimentation cycle, and the core methodology itself remains accessible to researchers with fewer hardware resources.\n\n\\\\( \\textcolor{red}{Question \\hspace{0.3em} 8:} \\\\) No discussion of sim-to-real gaps in dynamic lighting/weather.\n\n\\\\( \\textcolor{blue}{Response \\hspace{0.3em} 8:} \\\\) We analyze the sim-to-real gaps in **Section 4.4 of the manuscript**, where our results demonstrate high consistency between the 3DGS environment and real-world driving behavior. To further assess reconstruction quality under varying lighting and weather conditions, we measure the Peak Signal-to-Noise Ratio (PSNR), obtaining 29.5 for clear weather, 28.8 for rain, and 28.2 for nighttime scenes. These close PSNR values indicate consistently high visual fidelity across different lighting and weather scenarios, suggesting that such factors have minimal impact on the realism of our benchmark.\n\n\\\\( \\textcolor{red}{Question \\hspace{0.3em} 9:} \\\\) Generalization Beyond Training Data: Evaluation is limited to NuScenes-derived environments. Test on cross-domain datasets (e.g., Waymo, off-road).\n\n\\\\( \\textcolor{blue}{Response \\hspace{0.3em} 9:} \\\\)  Thank you for the valuable comment. We would like to reiterate that our benchmark scenes are significantly more complex and challenging than nuScenes as shown in **Table R4**. Furthermore, our training and testing sets are strictly disjoint, ensuring that the evaluation on the test set reliably reflects the model’s generalization ability. \n\n| Dataset       | Straight Ratio (%) | Turning Ratio (%) |\n|-|:-:|:-:|\n| nuScenes| 92.80%| 7.20% |\n| Navsim| 66.40%| 33.60% |\n| Our Benchmark | 52.50%| 47.50%|\n\nTable R4: Distribution of straight vs. turning scenarios in different datasets."}]

Author rebuttal:

Meta-review: The paper proposes a reinforcement learning framework for end-to-end autonomous driving. It integrates imitation learning as regularization, uses a photorealistic digital twin for closed-loop training, and designs safety-aware rewards. Results show significant improvements, particularly lower collision rates, over IL baselines.

Strengths: first integration of IL+RL in 3DGS photorealistic environments; extensive closed-loop evaluation across thousands of scenarios; strong safety performance (3x lower collision rate); clear pipeline and ablations; code planned for release.

Weaknesses: limited novelty beyond combining existing components; reliance on a self-created benchmark limits external comparability; high computational cost; missing discussion on sim-to-real transfer; reproducibility dependent on future code release.

During rebuttal, authors added evaluations under diverse weather/road conditions, clarified IL+RL distinctions vs prior work, explained decoupled action space, addressed reward hacking, and confirmed open-sourcing plans. Reviewers acknowledged clarifications, with two raising scores and all maintaining positive recommendations. Concerns on novelty and benchmark remain but were judged minor compared to contributions.
Decision: Accept (poster)