Prioritizing Perception-Guided Self-Supervision: A New Paradigm for Causal Modeling in End-to-End Autonomous Driving
Authors: Yi Huang
zhan qu
Lihui Jiang
Bingbing Liu
Hongbo Zhang
Venue: NeurIPS 2025 poster ()
Keywords: End to end autonomous driving
Primary area: applications
TL;DR: We prioritize Perception-Guided Self-Supervision (PGS) as a new paradigm for causal modeling in end-to-end autonomous driving, achieving SOTA closed-loop results.

Abstract: End-to-end autonomous driving systems, predominantly trained through imitation learning, have demonstrated considerable effectiveness in leveraging large-scale expert driving data. Despite their success in open-loop evaluations, these systems often exhibit significant performance degradation in closed-loop scenarios due to causal confusion. This confusion is fundamentally exacerbated by the overreliance of the imitation learning paradigm on expert trajectories, which often contain unattributable noise and interfere with the modeling of causal relationships between environmental contexts and appropriate driving actions.

To address this fundamental limitation, we propose Perception-Guided Self-Supervision (PGS)—a simple yet effective training paradigm that leverages perception outputs as the primary supervisory signals, explicitly modeling causal relationships in decision-making. The proposed framework aligns both the inputs and outputs of the decision-making module with perception results—such as lane centerlines and the predicted motions of surrounding agents—by introducing positive and negative self-supervision for the ego trajectory. This alignment is specifically designed to mitigate causal confusion arising from the inherent noise in expert trajectories.

Equipped with perception-driven supervision, our method—built on a standard end-to-end architecture—achieves a Driving Score of 78.08 and a mean success rate of 48.64\% on the challenging closed-loop Bench2Drive benchmark, significantly outperforming existing state-of-the-art methods, including those employing more complex network architectures and inference pipelines. These results underscore the effectiveness and robustness of the proposed PGS framework, and point to a promising direction for addressing causal confusion and enhancing real-world generalization in autonomous driving.

Prioritizing Perception-Guided Seff-Supervision A New Paradigm for Causal Modeling in End-to-End Autonomous Driving
Yi Huang"?; Zhan Qu?; Lihui Jiang" Bingbing Liu?, Hongbo Zhang?
'The Chinese University of Hong Kong, Shenzhen
HHuawei Noah's Lab
yihuang11@ling cuhk edu. cn
(quzhan, jianglihuii, 1iu. liu. bingbing, bhanghougboo88B/huuaree.
Abstract
End-to-end autonomous driving systems, predominantly trained through imitation learning, have demonstrated considerable effectiveness in leveraging large-scale expert driving data. Despite their success in open-loop evaluations, these systems often exhibit significant performance degradation in closed-loop scenarios due to causal confusion. This confusion is fundamentally exacerbated by the overre- liance of the imitation learning paradigm expert trajectories, which often contain unattributable noise and interfere with the modeling of causal relationships between environmental contexts and appropriate driving actions. To address this fundamen- tal limitation, we propose Perception- Guidedd Self-Supervision (PGS)-a - simple yet effective training paradigm that leverages perception outputs the primary supervisory signals, explicitly modeling causal relationships decision- pecision-making. The proposed framework aligns both the inputs and outputs of the decision -aaiing module with perception results such as lane centerlines and the predicted mo- tions of surrounding agents-by by introducing positive and negative self-supervision for the ego trajectory. This alignment is specifically designed to mitigate causal confusion arising from the inherent noise in expert trajectories. Equipped with perception-driven supervision, our method-buil on standard end-to-end archi- achieves Driving Score of 78.08 and mean success rate of 48.6 89.64% on the challenging closed-loop Bench2Drive benchmark, significantly outperform- ing existing ffate-o-tthe-e methods, including those employing more complex network architectures and inference pipelines. These results underscore the effec- tiveness and robustness of the proposed PGS framework, and point promising direction for addressing causal confusion and enhancing real-world generalization in autonomous driving.
Introduction
Autonomous driving, significant application of AI, has made impressive advancements in recent years. End-to-end neural networks, which allow vehicles to make decisions directly from raw sensor signals, are considered capable of overcoming the cumulative error issues inherent in traditional modular approaches and offer the potential to scale with vast amounts of data. mainstream systems, perception, prediction, and planning tasks are integrated into single network. The planning module uses explicit or implicit representations of the environment provided by perception to plan the future
"Equal contribution. CCoresponning author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

behavior. with human or expert trajectories being used as the target of training. Researchers have made significant efforts to leverage large-scale human driving data to enable models learn the relationship between environmental context and vehicle behavior.
As classic paradigm for end-to-end systems, imitation learning became prominent alongside early benchmarks such as nuPlan [3), Argoverse 4 Oxford RobotCar [2]. These benchmarks typically provide open-loop metrics, with L2 error between predicted and ground-truth trajectories as the key indicator. Consequently, researchers focus on designing complex network architectures, incorporating multi-modal sensor information, and using imitation learning objective functions aligned with these metrics, to enhance the model's is ability to fit expert trajectories. However, recent studies have shown that trajectory fitting in open-loope evaluation cannot accurately reflect system performance in real- world scenarios [15 19 28). In closed-loop simulation tests, pure imitation learning models often show significant degradation in safety, comfort, and feasibility in complex scenarios. This inability to generalize in real-world environments has become a major challenge for end-to-end systems.
Among the factors affecting the closed-loop performance, the most significant is causal confusion. Causal confusion refers to the model's 's inability to associate driving behavior with the primary causal factors in the environment, instead linking it other noise factors. Although recent end- to-end approaches have reduced input noise by using sparse instance-level representation [4] of the environment, these methods still fail to fully address this problem. In this paper, We identify causal confusion as an unavoidable byproduct of the imitation learning framework, stemming from its reliance on suboptimal expert data. Expert or human trajectories often contain noise from factors like driving style, time of day, or control errors, making them suboptimal supervision targets. Learning from such noisy signals weakens the model's ability to capture true causal relationships. We argue that causal confusion stems not just from imperfect inputs, but more critically from noise in the supervision itself.
Unlike prior approaches that treat perception and prediction modules merely as feature extractors, we propose framework leverages their outputs as primary supervision signals for decision-making. By aligning both the inputs and outputs of the decision maaing module with perception outputs, our perception- -ugidedd self supervvperv paradigm exhibits stronger causal modeling capabilities in closed- loop evaluations than pure imitation learning. Specifically, we introduce three novel self-supervision mechanisms: Multi-Modal Trajectory Planning Self-SSuervision (MTPS), Spatial Trajectory Planning Self- Self-Supervision (STPS), and Negative Trajectory Planning Self supe-uupevisoon (NTPS). MTPS and STPS utilize lane centerlines to enforce topological constraints and support multimodal decision- making across available lanes. NTPS incorporates the predicted future trajectories of dynamic agents as negative supervision to guide the ego vehicle away from potential collisions. In this framework, human expert trajectories are used to filter or regularize cuff-supervision targets when perception-based guidance is unavailable.
In summary, we propose an innovative training paradigm for end-to-end autonomous driving systems, which does not rely on specialized network designs but emphasizes the use of perception- guided self-supervision as the main learning objective. Our contributions include the following:
1. Multi-Modal Trajectory Planning Self-Supervision as Target Lane Selection: We reformulate multi-modal ego decision-making as a target lane selection problem based on lane perception. This approach enhances the system's 's ability to associate surrounding obstacles and available lanes with appropriate driving decisions, thereby improving the performance of lane-change planning.
2. Spatial Trajecotry Planning Self-Supervision based on lane centerline: We take the lane centerline outputted from perception module as spatial trajectory without temporal dependeney, and use them as the primary learning target for planning ego trajectory. This design effectively reduces lane departures and mitigating causal confusion induced by inconsistent and noisy expert demonstrations.
3. Negative Trajectory Planning Self-Supervision from Dynamic Objects' Future bound- ing box: Our framework selects and utilizes the predicted future trajectories of surrounding agents as negative supervision signals for ego trajectory learning, enforcing non-overlapping constraints between future bounding boxes. This facilitates the learning of interactions with dynamic agents and reduces collision risk.

---

4. We made minimal modifications to a simple end-to-end network architecture to adapt and validate our proposed self-supervision training paradigm. In experiments on the challenging closed-loop benchmark, Bench2Drive, the self-supervised model outperformed the pure imitation learning version of the same architecture and recent works using more complex network structures and pipelines by large margin.
2 Related Work
End-to-end autonomous driving aims to generate planning trajectories directly from raw sensors. In the field, advancements have been categorized based on their evaluation methods: open-loop and closed-loop systems. We reviewed representative works based on these two evaluation schemes in the first and second subsections, and summarized existing techniques and improvements addressing the causal confusion in the third subsection.
2.1 Open-Loop End-to-End Driving Methods
In open-loop systems, UniAD 8 proposes a unified framework that integrates full-stack driving tasks with query-unified interfaces, enhancing task interaction. VAD (14) improves planning safety and efficiency, as demonstrated by its performance on the nuScenes dataset. SparseDrive [23] uses sparse representations mitigate information loss and error propagation in modular systems. ParaDrive (26) organizes perception, motion prediction, and planning tasks in parallelized architecture during train- ing, retaining only the planning module in inference. This approach improves planning performance and significantly reduces runtime latency
2.2 Closed-Loop End-to-End Driving Methods
Existing works (e.g., BEVPlanner 19) have found that metries like 1.2 error and collision rate used in open-loop evaluations do not comprehensively reflect model performance in real-world scenarios. As result, more approaches are being proposed for closed-loop evaluation. VADv2 (5] advances vectorized autonomous driving by generating action distributions for vehicle control, achieving outstanding performance on the CARLA Town05 benchmark. Transfuser 6 uses transformer modules at multiple resolutions to fuse perspective and bird's s-eye view feature maps, outperforming prior work on the CARLA leaderboard. Hydra-MDP [17] employs knowledge distillation from both human and rule-based teachers to train the student model, enabling the selection of the trajectory with optimal overall performance and securing first place in the Navsim challenge. Drive Transforrrr [I3 delves into task parallelism and sparse representation in architecture design, significantly improving driving scores and success rates on the Bench2Drive benchmark.
2.3 Techniques Addressing Causal Confusion in Autonomous Driving
Causal confusion has been a persistent challenge in imitation learning. In end-to-end driving, ChauffeurNet (1] addresses this issue by using past ego-motion intermediate BEV abstractions, and randomly dropping them during training. PrimeNet (25) improves performance by incorporating predictions from a single-frame model as additional input to a multi-frame model. DriveAdapter [II] mitigates the influence of noisy perception outputs by training a strong planner with privileged perception information, and aligning perception model output with the planner's input through an adapter. RAD ( proposes a 3DGS-bard based closed- reinforcement learning framework, which uses specialized rewards guide the policy understanding real-worle causal relationships more effectively. These approaches primarily aim to mitigate causal confusion by suppressing noise in inputs to the planning module. this paper, we propose novel perspective and an innovative training paradigm, where perception- pereptiionguidedd guided self-supervision plays a key role in addressing causal confusion by aligning the input and output of the planning module.
3 Method
In this section, we introduce a Prioritizing Perception-Guidee Guided Self-Supervision training paradigm built upon typical end end architecture. On one hand, sparse instance-level features extracted from perception are used as inputs to the unified decision and prediction module, helping minimize

---

input noise. On the other hand, the perception output is directly employed for self-supervision of the planning process. This alignment plays an important role in helping the planner learn causal relationships.
3.1
End-to-End Network Baseline
Topalogy Phorning Heed
STPg MTPS. MP5
Mep Query
Mep
PGS Panddgm
Ago Parning
Bareets
Agent Mation
Inltatiinn
Ego Trijectry
dineniiaaa PV Portores
fusure
Object MRL
Objact Trijecory
Coal point
Oiject Mode VILP
Object Query
Object Docader
300
Advanced commiadd
Sge Query
Mode Clasdiiation Score
Istwetion Quary
Figure 1: Overall Model Architecture. (1) Perception network provides map elements and future motions of dynamic agents. (2) The dashed box presents the proposed PGS, which generates three
self-supervised signals from perception outputs to enhance causal reasoning in ego planning.
Our baseline architecture is simple and similar to VAD 14. As illustrated in Figure I multi-view images are encoded into high-dimensional features in perspective view using a Resnet-based encoder. BEVFormer [88 then transform the features into BEV (Bird's Eye View) space as FBEV.
For task-speeific applications, learnable embeddings are used query the BEV features, extracting sparse representations of the environment. The map query interacts with the BEV features to decode static road topology such as lane markings, lane centerlines, sidewalks, and other structures. The object query decodes information about dynamic objects (pedestrians, cyclists, and vehicles), including their positions, sizes, orientations, and velocities at the current timestep. The perception process can be formally expressed follows:
q PerceptionDeecoder/ qi kv Foev), 9i Qmap U Uokj:
map y/
MLP map (g. = (tp)) ( Npmn ( J==1
Pj ( ER
(2)
MLP (g. (g:"0))= = (xte Ut, w. 1, cc, u D1
where of is the enhanced instance-level query, 0.muP and y are decoded information of map element and dynamic object. The former are represented as polyline, where D3 is the 2D coordinates j-th point of map element, and e denotes its class score. The latter includes the 2D coordinates of the object's center, its width and length, the heading angle, and the velocity components along the x and y axes at time
Next, instead of adopting the cascaded prediction decision architecture, unified decoder for both ego planning and object motion prediction is employed. Specifically. a learnable ego query Jego is augmented with a goal point embedding and an intent embedding corresponding to the high-level command c. For other agents, each motion query is formed by combining the updated object query sobj
deciisooon-deisioonn
qi with one of six implicit intent embeddings The two sets of queries concatenated to form the agent-level motion query for the current scenario:
Q Quotioon Concat MMLPeooAeggo ① + c), MLP (19
(()) + ek. = 1. K, 10:=1.... Novy)) (4)
In the unified decoder, agent-level motion queries attend to each other via self-attention and to map queries via cross-attention, enabling the model to capture interactions between agents constrained by the current road topology. The refined motion queries are again split into those for the ego and dynamic objects, which passes through seperated MLP for predicting the future trajectories. The planning and motion prediction process can be formally expressed as follows:
Q'ootoon MotionDecoder((g Qmotion: kv Q Qmmp)
(5)

---

mn = *MLL aood (
Trajahj
ML. Punio
(mL) where k I,...,K, I,...
Nobj, Lo @
Trajaoo = MLP ol9m fm Q mmotion_eeo
(7)
where a denotes the sigmoid function; f represents the score the k-th predicted modality for the .il j-th object, and Trajeby denotes the corresponding trajectory over the prediction horizon T; Trajego is
the planned trajectory of the ego vehicle.
In training phase, the loss terms of perception are same as in VAD 14] with imitation loss of L1 norm:
Tuocal Widet_ma mup /eteemmppppp map + Wodet_obj oby + cls modd_ls cls + Wmotion_ohy /mmotion obj
Wimi Timi
The outputs of the perception and motion prediction tasks- the implicit high- high-dimensional features and the structured trajectories and paradigm.
the foundation our self-supervised
3.2 Multi-Modal Trajectory Planning Self-Supervision (MTPS) as Target Lane Selection
Roud Tupobogy Guided Plonning Hend
Ege
git Rine
Map
Relevee Fhwiing
Embeellingg
Faw
Lana Select MLB
Score
Slow Down
Change Lane Left
Change Lane Right
Conterliee
Lanes
Condidane lanes
typ Trags MIP
Trajs
(a) Ego Multi- Modal Decision-Makingg Cruising Scenarios (Lane Follow)
(b) Road Topology Guided Planning Head
Figure 2: Environment-Aware Lane Command and Road Topology Guided Planning.
Lane status in the surrounding environment plays crucial role in guiding the ego's driving behavior. defines the drivable area and constrains the range of feasible trajectories. As shown Figure 2a when the command is "LaneFollow" but obstacle appears ahead, the ego either decelerate in the current lane or change lanes to overtake. This turns lateral multi-modal planning into a lane selection problem shaped by the environment. This insight forms the basis of our Multi-Modal Trajectory Planning Seff-Supervissio((MPP)))
MTPS leverages the surrounding lane structure to guide the selection of the ego vehicle's planning modality. As illustrated Figure 2bb this module includes a Road Topology Guided Planning Head, which generates both multi-modal decisions and corresponding ego trajectories, alongside topology- aligned self-supervision mechanism. Firstly, geometry-based lane filter leveraged to select ego- Nomop relevant lane centerlines from the perception output. Given the set of lane centerline P = {P)) j-
with each P, represented as sequence of centerline points, we compute the minimum Euclidean distance dj and the relative angle 92 between the ego vehicle's current position Tegos heading Oego
and each centerline. The relevant lane set FP = {(Fiett Pien ), (Feumeet: Peurret ), (Fright Pright constructed according to the following criteria, where denotes the implicit feature of each lane:
(Fe, Pa) = Pz) (z) = (F,,)),
id S 0.5W
dj min /ego - pll2 peP;
if (0.5W, 1.5] and ( <
coo Ozgo sin 0 Dego
9, (pj )
- (FF;P))
if id E (0.5W, 1.55] and y > 0
(F,P)=(0.0),
otherwise
(9) where p is the nearest point in P, to the ego vehicle, W is the standard lane width, and the subscripts c, I, and denote the current, left, and right. no centerline satisfies the above criteria, the corresponding feature F and point P are set zero.
This geometry-based filter simple yet effective, allowing robust efficient identification of the ego vehicle's current lane and adjacent lanes, thereby capturing all feasible lateral motion options.
5

---

Next, the implicit features of relevant centerlines are fused with the ego motion query. Two additional MLPs are utilized to predict the lane selection score and the corresponding trajectory. These scores are normalized by softmax operator, transforming the ego's multi-modal trajectory planning into lane-level classification task, as described below:
H, q B.DF Fs, AF. E FP, S = -softmaa score (H)), Trajego = MLP traj (Hrggaax (S)
(10)
During training, the index of target centerline is provided by measuring the average spatial distance between the terminal portion of the ground-trutt truth ego trajectory and each candidate lane centerline. The index of the centerline with the minimal distance is designated the target lane index 1", with the corresponding feature and polyline as F and P The loss function of selecting target lane is defined as:
LMPPS = LCE(S,1"
(11)
3.3 Spatial Trajectory Planning Self-Supervision (STPS) based on Lane Centerline
Lane centerlines, compared other road topology cues like markings and boundaries, play more critical role in learning robust driving behaviors. Human trajectories often deviate slightly from the centerline due to factors like driving style, weather, or control noise- difficlll to attribute and thus regarded by the model as learning noise. This noisy supervision can negatively impact the model's s causal understanding of driving behaviors, particularly in scenarios involving intersection turning. Lane centerlines naturally connect incoming and outgoing lanes, and training on trajectories that deviate from them increases the risk of drifting into the wrong lane due to cumulative errors.
Building on this insight, we propose a Spatial Trajectory Planning Superviperiiionn (STPS) mecha- nism, in which the expert trajectory Trajgt is replaced by centerline-aligned version the primary supervision signal. Specifically, each expert trajectory point is checked against nearby target center- line points (from the previous stage); if matched centerline point is found, it replaces the original point. Original expert point is retained only when no point is matched, which serves as a regular- ization term preserve the smoothness of the target trajectory. The resulting trajectory Traj' tgt supervises the Road topology guided trajectory regression head, working as a spatial ground-truth path free of temporal bias but more causally aligned. Formally, for expert trajectory point T rajj" 't tgt are given by:
at time step f, the resulting updated trajectory points Traj't,
p1 = arg min [Traj' - 11y122 DEEEP
(12)
phs (Taajat
if i| - pilla S w
Traj'k tgt
(13)
otherwise
This new trajectory Traj' tot
is then used to supervise the trajectory regression head as:
ITrasego Traj't tgt ll
LSTES
(14)
N
Last but not least, since the regression head also takes the target centerline features F* as input, aligning the trajectory target with the centerline further strengthens causal reasoning in trajectory prediction by jointly leveraging topological cues and supervision consistency.
3.4 Negative Planning Self-supervision (NPS) from Dynamic Objects' Future Bounding Boxes
An autonomous systems must dynamically respond to surrounding agents. While MTPS and STPS support positive causal modeling for general planning, safe interaction requires negative causal modeling to proactively avoid risky outcomes- e.g.. adjusting the ego trajectory to prevent overlap with the predicted motion of encroaching parked vehicle as shown in the Figure Ba
Motivated by this insight, we propose the Negative Trajectory Planning Self-Supervision (NTPS) mechanism, which imposes safety constraints on ego planning using the predicted future bounding boxes of surrounding agents. As illustrated in Figure (3b, we construct future bounding box sequences for both ego and dynamic objects using predicted trajectories and perceived object dimensions. The

---

9 Objjct Trajectory
Ego Trajs MLP
Trajs MLP
Construning bounding boxes
Ege Trajectory
Negative Loss
(a) Example of Risky Interaction Scenario
(b) Safety Constraint Mechanism of NTPS
Figure 3: Negative Trajectory Planning Self-Supervision (NTPS) for Safery-Constrained Ego.
orientation at each timestep estimated via trajectory offsets, and overlap detection is performed using the Separating Axis Theorem (SAT) ( Upon detecting overlaps, we introduce negative
supervision loss that penalizes ego trajectories encroaching into occupied space by encouraging divergence from the overlapping region. follows:
LNTPS Σ
max(0,,8 B i T obillll)
(15)
tET.
where Toll denotes each timestep the set of detected collision timesteps Teoll and Traj objjoojo col represents the trajectory point of the surrounding object that is predicted collide with the ego vehicle at timestep t.
In this process, SAT-based overlap detection identifies risk-inducing points along the trajectories of surrounding agents. These are treated as negative supervision signals, guiding the ego trajectory to diverge from potential collision zones and thereby enhancing safety in dynamic interactions.
3.5 Perception-Guider Self-supervision in Optimization
During training, the proposed PGS paradigm introduces guidance from perception by integrating the three distinct self-supervision losses described above total loss introduced in Section 3.1 as:
Lavial + (UMISSMMIPS uosresSsre INTPSINPS
(16)
where represent the relative importance of each loss component.
4 Experiment
4.1
Dataset & Metrics
Dataset: To evaluate the real-world effectiveness of our self-training paradigm, we evaluate on the challenging closed-loop benchmark Bench2Drive [10], built on CARLA v2. The dataset includes 1,000 short clips across 44 complex scenarios (950 for training, 50 for open-loop validation) and 220 predefined routes for standardized closed-loop evaluation nd fair performance comparison.
Metrics: We adopt Bench2Drive's official metrics: Driving Score, Success Rate, Efficiency, and Comfortness for closed-looy evaluation, and L2 Displacement Error (L2) for open-loop evaluation.
4.2
Implementation Details
The training process of PGS is divided into two stages, each with distinct learning objectives.
Stage focuses perception learning. We enhance the online map detection task by introducing lane centerlines as new class of map elements. While the task of motion prediction of dynamic objects is trained in this phase well. In addition, traffic light detection from front-view images is incorporated to capture critical causal dependencies for safely navigating signalized intersections. Training in this stage lasts for 6 epochs.

---

Stage 2 builds upon the perception capabilities from Stage and introduces joint optimization of the perception module and self-supervised objectives. Perception losses are retained to maintain accurate environmental understanding, while three self-supervised losses Massss-MPSS, STPS, and NTPS are introduced to supervise ego planning tasks. Stage also trained for epochs.
Training is conducted on 16 NVIDIA RTX V100 GPUs using the AdamW [21] optimizer, with weight decay of 0.01 and an initial learning rate of 4c-4. The loss weights set to WMTPS 1.0, 1.0.
WSTPS 0.3, and WNPS
4.3 Comparison with State-of-the-Art Methods
Table 1: Open-loop and Closed-loop results of planning in Bench2Drive. Avg. L2 averaged over the predictions in seconds under 2Hz. denotes expert feature distillation.
Method
Avg. 1.2
Driving Score f
Success Rate (%)
Efficiency
Comfortness
AD-MLP
28
3,64
18.05
0.00
48.45
22.63
UniAD-Base
8
0.73
45.81
16,36
129.21
43.58
UniAD-Tiny
8
0,80
40.73
13.18
123.92
47.04
VAD-Buse 14
0,91
42.35
15.00
157.94
46.01
VAD-Tiny 14
1.15
34.28
10.45
70.04
66.86
SparseDrive 23
0.87
44.54
16.71
170.21
48.63
GenAD 29
44.81
15.90
DiFSD 20
0.70
$2.02
21.00
178.30
DriveTrans
13
0.62
63.46
35.01
100.64
20.78
DiffAD 24
67.92
38.64
WoTE [
61.71
31.36
BridgeAD
0.71
50.06
22.73
PGS(Ours)
0.77
78.08
48.64
181.31
12.37
Tra- 27
1.70
59.90
30.00
76.54
18.08
ThinkTwice*
12
0.95
62.44
31.23
69.33
16.22
DriveAdapter
11
1.01
64.22
33.08
70.22
16.01
Table M summarizes the comparative open-loop and closed-loop planning performance on Bench2Drive. Compared to VAD-Base-aa the baseline model for our approach-PGGS reduces the open-loop L2 error from 0.91 to 0.77. more importantly, PGS achieves a remarkable Driving Score of 78.08, outperforming VAD-Bac (42.35) by 35.73 points in closed-looe evaluation. The Success Rate improves significantly from 15.00% to 48.64%. These improvements are primarily attributed to the enhanced causal reasoning capabilities introduced by self-supervised planning framework.
The comparison with contemporaneous methods 13 16 further validates the effectiveness of proposed paradigm. These methods improve closed-loop performance by adopting more complex architectures, leveraging multi-modal sensor inputs, employing diffusion models for multi-modal decision, combining trajectory generation with online ranking strategies, but still primarily rely on imitation learning and lack explicit consideration of causal reasoning. In contrast, the self- supervised, causality-drive PGS framework consistently outperforms these methods, highlighting the effectiveness of perception-gun guided self-supervision in capturing causal relationships.
Furthermore, PGS surpasses methods based on knowledge distillation (e.g., [27 12 III as well. Although distillation enhances planner robustness by transferring knowledge from expert models trained with noise-free privileged information, it fails model the causal dependencies between redundant perception outputs and ego planning. and results in causal confusion and suboptimal dicision-making policies. These results further validate the superiority of PGS in achieving causally grounded and robust driving performance.
Table 2 further compares the success rates of different approaches across specific driving scenarios. scenario is considered successful only if the ego vehicle reaches the designated destination without any collisions or infractions. Our model consistently outperforms competitors across several critical driving skills, achieving notably high success rates in Merging (35.00%), Overtaking (73.33%), Emergency Braking (55.00%), and Give Way (60.00%). It also obtains the highest overall ability score of 53.40%. These results highlight the strong generalization capability of our approach in handling complex and highly interactive scenarios.

---

Table 2: Multi-Ability Results of E2E-AD Methods. denotes expert feature distillation.
Ability (%)
Method
Merging
Overtaking
Emergency Brake
Give Way
Traffic Sign
Mean
AD-MLP 28
0.00
0.00
0.00
0.00
4.35
0.87
UniAD-Tiny
8.89
9.33
20.00
20.00
15.43
14.73
UniAD-Bas Base
14.10
17.78
21.67
10.00
14.21
15.55
VAD [14
8.11
24.44
18.64
20.00
19.15
18.07
DriveTranfformer
17.57
35.00
48.36
40.00
52.10
38.60
DiffAD
30.00
35.55
46.66
40.00
46.32
38.79
PGS (Ours)
35.00
73.33
55.00
60.00
43.68
53.40
TCP 27
16.18
20.00
20.00
10.00
6.99
14.63
TCP-ctrl*
10.29
4.44
10.00
10.00
6,45
8.23
TCP-traj
8.89
24.29
51.67
40.00
46.28
34.22
TCP-traj w/o distillation
17.14
6.67
40.00
50.00
28.72
28.51
Think [2
27.38
18.42
35.82
50.00
54.23
37.17
Driverd 11
28.82
26.38
48.76
50.00
56.43
42.08
4.4 Ablation Study on Bench2Drive
We conduct extensive ablation experiments to assess the contribution of each component in our self- supervised paradigm. For efficient closed-loop evaluation, we select the Merging and Overtaking scenarios, which are both complex and highly interactive. Together, they account for more than half of the total scenarios, making the evaluation metries on them sufficiently representative.
Table 3: Ablation Study of the Proposed PGS Framework.
Ability (%) t
Method
Avg. L2
Merging
Overtaking
Mean
VAD-Base
0.91
8.11
24.44
16.28
VAD-Tiny
1.15
9.33
11.11
10.22
PGS PGSBuse Base
0.87
16.46
13.33
14.89
PGS Buse+STPP
0.78
24.44
26.25
25.35
PGS BeseB
0.75
23.75
44.44
34.10
PGSA AUL
0.77
35.00
73.33
54.17
PGSNTPS
0.90
25.00
6.67
15.84
PGSself
2.89
31.25
35.56
33.40
As shown in Table PGS Base denotes our baseline model, where the perception network is trained with the perception loss used in VAD, and the planning head is trained with the imitation loss. Compared to VAD, it achieves slightly lower L,2 error and a comparable success rate. PGSsTrs introduces the centerline-aligne the alignment between road topology cues and the ego's planned trajectory, leading to significant improvements in both L2 error and success rate. Building upon this, PGS PS+ MTT Ps incorporates a relevant lane filter and reformulates the multi-modal ego decision as lane selection task within this filtered set. This design yields substantial performance boost in the Overtaking scenarios, where frequent lane changes occur. Finally, PGSAU further adds Negative Trajectory Planning Self- Superviirvi by identifying risky future positions of surrounding dynamic objects and penalizing ego trajectories that overlap with them. This additional constraint reduces collision risk in both selected scenarios. Overall, PGS au achieves mean success rate of 54.17%, with an improvement of over 39% compared to PGS which is trained purely via imitation learning. We further isolate the effect of Negative Trajectory Planning Self- suff--uueriiiion through a dedicated variant, PGSNTPS: to assess model behavior in unstructured environments. Despite the absence of structured road priors, this variant exhibits strong performance in Merging scenarios, demonstrating the capability of NTPS to mitigate collision risks in complex, geometry-agnosee contexts. However, its performance degrades Overtaking scenarios, likely due to overly conservative behavior the presence static obstacles and the lack of contextual lane information. These limitations effectively addressed
Spatial Trajectory Planning Self Self-Supervision, which strengthens

---

when NTPS is integrated with STPS and MTPS, as structured priors offer richer spatial cues and broaden the maneuver space, enabling more balanced and flexible decision-makee making.
Besides, we retrained a model, sGSsetf: using only PGS self-supervision, without any imitation loss. As expected, the 1.2 error increases significantly due the absence of expert trajectory knowledge. However, it still achieves a respectable success rate of 33.40% outperforming both PGS Base and V YAD by large margin. This underscores the importance of perception-consistet ego planning in causal modeling.
Conclusion & Limitation
We introduce perception-guided self-supervision paradigm for end-to-end au- tonomous driving. By leveraging road topology and dynamic agent motion as both inputs and supervisory signals, our approach aligns ego trajectory prediction with structured, causally relevant cues, enabling stronger causal reasoning and state-of- closed-loop performance. Extensive ablation studies further substantiate the efficacy of self-supervision mechanisms, highlighting promising direction for enhancing the real-world robustness of end-to-end autonomous driving.
Conclusion:
The framework's 's success relies accurate and robust perception of dynamic/static agents and road structures. Limited perception accuracy or generalization may impair planning performance, making perception robustness key challenge for future work.
Limitation:
References
[1] M. Bansal, A. Krizhevsky, and A. Ogale. Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst in roboties: Science and systems University of Freiburg, Freiburg im Breisgau, Germany, 2019.
[2] D. Barnes, M. Gadd, P. Murcutt, P. Newman, and Posner. The oxford radar robotcar dataset: A radar extension to the oxford robotcar dataset. In 2020 IEEE international conference robotics and automation (ICRA), pages 6433-6438. IEEE, 2020.
[3] H. Caesar, Kabzan, S. Tan, W. K. Fong, E. Wolff, Lang, Fletcher, O. Beijbom, and S. Omari. nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles. arXiv preprint arXiv:2106. 11810, 2021.
[4] M.-F. Chang, J. W. Lambert, P. Sangkloy, Singh, S. Bak, A. Hartnett, D. Wang, P. Carr, Lucey, D. Ramanan, and J. Hays. Argoverse: 3d tracking and forecasting with rich maps. In Conference Computer Vision and Pattern Recognition (CVPR), 2019.
[5] S. Chen, B. Jiang, H. Gao, B. Liao, Q. Xu, Q. Zhang, C. Huang, W. Liu, and X. Wang. Vadv2: End-to-end vectorized autonomous driving via probabilistic planning. arXiv preprint arXiv:2402. 13243, 2024.
[6] K. Chitta, A. Prakash, B. Jaeger, Z. Yu, K. Renz, and A. Geiger. Transfuser: Imitation with transformer-based sensor fusion for autonomous driving. IEEE transactions on pattern analysis and machine intelligence, 45(11):12878-1288)
2022.
[7] H. Gao, Chen, B. Jiang, B. Liao, Y. Shi, X. Guo, Pu, Yin, Li, X. Zhang, al. Rad: Training end-to-end driving policy via large-scale 3dgs-based reinforcement learning. arXiv preprint arXiv:2502.1 13144, 2025.
[8] Y. Hu, J. Yang, L. Chen, K. Li, C. Sima, Zhu, Chai, S. Du, T. Lin, W. Wang, et al. Planning- oriented autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11853-178622 2023.
[9] Huynh. Separating axis theorem for oriented bounding boxes. URL:: jkh. ceeefileesitorriaisseppannnn 20Axis% 20Theorem% 20for% 200riented% 20Bounding 20Boxes. pdf. 2009.
[10] X. Jia, 7. Yang. Li, 7. Zhang, and J. Yan. Bench2drive: Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving. The eiiht Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
10

---

[11] X. Jia. Y. Gao, L. Chen, J. Yan. P. L. Liu, and H. Li. Driveadapter: Breaking the coupling barrier of perception and planning end-to-end autonomous driving. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 7953-7963, 2023.
[12] X. Jia, Wu, L. Chen, J. Xie, C. He, J. Yan, and Li. Think twice before driving: To- wards scalable decoders for end-to-end autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21983-21994, 2023.
[13] X. Jia, J. You, Z. Zhang, and J. Yan. Drivetransformer: Unified transformer for scalable end-to-end autonomous driving. arXiv preprint S00.v:::5000006566 2025.
[14] B. Jiang, S. Chen, Q. Xu, B. Liao, J. Chen, H. Zhou, Q. Zhang, W. Liu, C. Huang, and X. Wang. Vad: Vectorized scene representation for efficient autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340-8350 2023.
[15] H. Li, M. Yuan, Y. Zhang, C. Wu, C. Zhao, C. Song, H. Feng, E. Ding, D. Zhang, and J. Wang.
XId: A cross-lane dataset for benchmarking novel driving view synthesis. CoRR, 2024.
[16] Y. Li, Y. Wang, Y. Liu, J. He, L. Fan, and Z. Zhang, End-to-end driving with online trajectory evaluation via bev world model. arXiv preprint arXiv:2504.0144 2025.
[17] Z. Li, K. Li, S. Wang, S. Lan, Z. Yu, Y. Ji, Z. Li, Z. Zhu, J. Kautz, Z. Wu, et al. Hydra- mdp: End-to-end multimodal planning with multi- tartee hydra-distillation. arXiv preprint arXiv:2406. 06978, 2024.
[18] Z. Li, W. Wang, H. Li, E. Xie, C. Sima, T. Lu, Q. Yu, and J. Dai. Bevformer: learning bird's- eye-view representation from lidar-camera via spatiotemporal transformers. IEEE Transactions on pattern Analysis and Machine Intelligence, 2024.
[19] Z. Li, Z. Yu, S. Lan, J. Li, J. Kautz, T. Lu, and J. M. Alvarez. Is ego status all you need for open-loop end-to-end autonomous driving? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11864-14873, 2024.
[20] C. Liang and X. Liu. The research of collision detection algorithm based on separating axis theorem. Int. J. Sci, 2(10): 2015.
[21] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations.
[22] H. Su, W. Wu, and J. Yan. Difsd: Ego-centric fully sparse paradigm with uncertainty de- noising and iterative refinement for efficient end-to-end autonomous driving. arXiv preprint MxXiv:2409.09777 2024.
[23] W. Sun, X. Lin, Y. Shi, C. Zhang, H. Wu, and S. Zheng. Sparsedrive: End-to-end autonomous driving via sparse scene representation. arXiv preprint 2.XXi::2005.96200 2024.
[24] T. Wang, C. Zhang, X. Qu, K. Li, W. Liu, and C. Huang. Diffad: unified diffusion modeling approach for autonomous driving. arXiv preprint arXiv:2503. /2/70, 2025.
[25] C. Wen, J. Qian, Lin, J. Teng, D. Jayaraman, and Gao. Fighting fire with fire: Avoiding dnn shortcuts through priming. In International Conference Machine Learning, pages 23723-23750. PMLR, 2022.
[26] X. Weng, B. Ivanovic, Y. Wang. Y. Wang, and M. Pavone. Para-drive: Parallelized architecture for real-time autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15449-15458, 2024.
[27] P. Wu, X. Jia, L. Chen, J. Yan, H. Li, and Y. Qiao. Trajectory-guided control prediction for end-to-end autonomous driving: A simple yet strong baseline. Advances in Neural Information Processing Systems, 35:61 19-6132, 2022.
(28) J. Zhai, 7. Feng, J. Du, Y. Mao, J.JJ. Liu, 7. Tan, Y. Zhang, X. Ye, and J. Wang. Rethink- ing the open-loop evaluation of end-to-end autonomous driving in nuscenes. arXiv preprint arXiv:2305. 10430, 2023.
[29] W. Zheng, R. Song, X. Guo, C. Zhang, and L. Chen. Genad: Generative end-to-end autonomous driving. In European Conference on Computer Vision, pages 87-104. Springer, 2024.

---

A Qualitative Analysis of Lane Centerline Perception
In CARLA, road topology information is stored using graph structure, where each node represents individual lane. The attriiutes of each node include the global coordinates of the lane's centerline, lane width, and other geometric properties. For each frame of training data, the ground-truth centerlines near the ego vehicle are generated by transforming all lane centerline coordinates from the global map into the ego-centric local coordinate system based the recorded ego pose.
(a) Perception results in straight road scenarios
(b) Perception results in curved road scenarios
(c) Perception results intersections
(d) Perception results in obstacle present scenarios
Figure 4: Visualization of lane centerline perception under diverse road conditions
The qualitative examples presented in Figure demonstrate that, even in the absence of salient visual features for lane centerlines, the map perception module in PGS can still reliably detect them by leveraging visual context from the scene. This ability is particularly evident in complex areas such as intersections. Such robust perception of centerlines forms the foundation for the effectiveness of the self-supervised MTPS and STPS components in framework.
B Relevant Lane Filtering and Target Lane Determination in MTPS
In the Multi-Modal Trajectory Planning Self-Supervision (MTPS), we frame the ego vehicle's multi- modal decision-making as a target lane selection task. The relevant lane filter extracts candidate lanes nanely the left, current, and right lanes- on the perceived road topology. Then, by referencing the ground-truth trajectory, the system identifies the target lane corresponding to the intended driving behavior.
Visualization of Supervision Signal Construction for Target Lane Selection
This subsection visually illustrates how the Multi-modal Trajectory Planning (MTP) module leverages the surrounding lane topology inform the selection of the ego vehicle's planning modality, as described in Section 3.2. Figure provides step-by-step illustration of the pipeline for constructing the supervision signal for target lane selection, starting from the perception outputs, moving through the filtering of relevant candidate lanes, and culminating in the determination of the target lane by referencing the endpoint of the ground-truth trajectory.
Classification Accuracy of the Target Lane Selection Task
To assess the effectiveness of topology-guidee supervision in MTPS, we evaluate the classification accuracy and recall of target lane prediction on the B2D open-loop validation set. As shown in
12

---

(a) Perceived lane centerlines
(b) Selected relevant lane set
(c) Selected target lane
Figure 5: Visualization of Target Lane Selection. Light pink points indicate the perceived lane centerlines, while dark red points represent the selected relevant lane set. The final target lane is also highlighted in dark red, with its name labeled in orange.
T training with the full 3-second ground-truth trajectory and standard cross- coos--nnropyy (CE) loss achieves high accuracy across all classes. However, due severe class imbalance, the recall for rare categories such as "oriented to left lane" and "oriented to right lane" remains relatively low, at 0.63 and 0.80 respectively.
Table 4: Accuracy and Recall of Target Lane Classification under Different Training Strategies.
Weighted CE 2s Trajectory
Metric
Label
CE
Weighted CE
Oriented to current lane 0.9685
0.9554
0.9690
Accuracy
Oriented to left lane
0.9785
0.9669
0.9748
Oriented to right lane
0.9887
0.9832
0.9909
Oriented current lane 0.9855
0.9586
0.9676
Recall
Oriented to left lane
0.6268
0.7344
0.8788
Oriented to right lane
0.7978
0.9011
0.9233
To mitigate this imbalance, we adopt a class reweighting strategy based on inverse class fre- quencies. Specifically, class weights are computed from the training data distribution as [1.074, 32.480, 26.505). corresponding to current, left, and right lane orientations, respectively. This adjustment significantly improves recall for the left and right lane categories by approximately 10 percentage points.
Furthermore, when shortening the trajectory horizon used for matching from 3 seconds to 2 seconds, both recall and precision improve. Shorter trajectories more accurately reflect immediate driving intentions, reducing ambiguity. Based on these results, we adopt inverse frequency weighting and 2-second trajectory matching as the default configuration for PGS training in MTPS.
C Spatial Trajectory Generation in STPS
In STPS, we construct spatial trajectory by combining the target lane centerline with the ground- truth trajectory, which is then used as supervision for the trajectory regression head facilitate better causal modeling.
13

---

(a) Ground-truth and STP trajectories in turning scenarios intersections
(b) Ground-truth and STP trajectories in straight-driving scenarios
Figure 6: Comparison of STP-generated and ground-truth trajectories. Perceived lane centerlines rendered light pink points, and other perceived lanes in orchid. Ground-truth trajectories visualized using yo-llow gradients, while STP trajectories are represented by blue gradients.
As illustrated in Figure 6 we present two representative scenarios. In Figure ba when navigating intersections, ground-truth trajectories occasionally deviate from the centerline of the designated outbound lane-either drifting left or right. Such sporadic deviations, likely due to labeling noise or imperfect execution, introduce spurious signals that hinder the model's 's ability to capture causally valid motion patterns. In contrast, STP trajectories, aligned with the intended lane centerline, exhibit high directional consistency and semantic alignment, thereby facilitating more robust causal learning.
Similarly, in Figure 6b) ground-truth trajectories in straightforward driving scenarios exhibit oscilla- tions around the lane centerline, potentially introducing ambiguity in lane-following behavior. The STP-generated trajectories, by contrast, maintain a stable, forward-directed course, offering clearer intent supervision and reducing trajectory-level noise.
D Separating Axis Theorem (SAT) Algorithm Description
In NTPS, the SAT algorithm is employed to generate negative supervision signals for the ego vehicle trajectory, conditioned on the predicted trajectories of surrounding agents.
SAT is a classical method for collision detection between convex polygons in 2D space. It is based on the principle that two convex polygons do not intersect if and only if there exists separating axis-an an axis along which the projections of the two polygons do not overlap. In practice, the set of candidate axes is constructed by computing the outward normals of all edges from both polygons. If a separating axis is found, the polygons are guaranteed to be disjoint. Otherwise, the polygons must intersect.
As shown Algorithm D the SAT algorithm iteratively tests all potential separating axes derived from the polygon edges.
14

---

Algorithm 1: Separating Axis Theorem (SAT)
Input: Vertex sets of polygons A and B
Output: Whether they intersect (true/false)
for each polygon P E {A, B3 do
for each edge of P do
Compute edge normal as projection axis axis; Project polygons A and B onto axis, obtaining intervals proja and projs: if proja and projø do not overlap then
return false;
// Separating axis found polygons don't intersect
return true:
// No separation found on any axis polygons intersect
E
Visualization of Closed-loop Evaluation
Representative Cases in Diverse and Complex Scenarios
Figure the closed-loog evaluation results of our model, PGSAu: Au. in the CARLA simulation environment. These visualizations are drawn from the full set of 220 closed-loop test scenarios in the Bench2Drive benchmark, encompassing a diverse spectrum of challenging conditions, such as adverse weather (e.g.. heavy rain, dense fog), varied lighting (e.g., daytime, nighttime), and complex traffic scenes (e.g., intersections, lane merging, overtaking, and traffic light negotiation). As illustrated, the model consistently produces smooth, goal-directed trajectories that respect the intended global route while dynamically responding to contextual hazards.
The results indicate that PGS au is capable of dynamically adjusting its trajectory to avoid obstacles while maintaining route fidelity. The model exhibits strong capacity align its predictions with the underlying semantic road structure, reflecting nuanced understanding of the causal relationships between environmental cues and appropriate driving behavior. This capability contributes to reliable and safe autonomous decision-making in closed-loop execution.
Limitations and Failure Case Analysis
While PGS Aul demonstrates robust performance across wide range of scenarios, observe several failure cases that expose current limitations in perception and causal reasoning.
As shown in Figure 8a the ego vehicle fails to avoid a parked car with an opened door. This failure is attributed the perception module treating such vehicles the same regular static obstacles, without distinguishing the opened door as separate semantic element. Consequently, the model fails to learn the causal relationship between door-opening events and the necessity of avoidance. Similarly, in Figure 8b) the model does not yield to approaching emergency vehicle from behind, likely due to the absence of semantic differentiation between emergency and regular vehicles the perception process.
These cases suggest the need enhance the perception module by introducing specialized object categories or refining bounding box representations (e.g., to cover opened doors or siren-bearing vehicles). Such improvements would allow the model to better capture the causal structures required for socially compliant and context-aware planning in these critical situations.
Supplementary Visualizations and Reproducibility
To further support analysis, provide extended set of visualizations in video format, accessible via the following GitHub repository: /Supplementary Materials Within this repository, the folder 0 O_reppreeentativeee ieeiiatttieeeeeeeee cases contains two subtolders, SiveerseChallengnngSeeeo and FailureScemarios which showcase representative cases that highlight both the strengths and limitations of the proposed model.
veace
Full closed-loop evaluation results are available the folder contains merged. jsso (overall results) and merged _uadd_abblttttyl jsso (per-scenario metrics). Frame- level logs for all 220 scenarios, including ego states, control commands (steering, throttle, brake), predicted traffic light states, and high-level decisions, are stored in eval_PGS all. The evaluation
which
15

---

(a) Overtaking stationary vehicle
S Delt E4
(b) Lane change under rainy weather conditions
Spted 2.67, Sebr -0.03 Ivotte: 3.75 boke 0.0. brotfe: RDD, 22200097750077000
(c) Driving at night with low visibility
(d) Pedestrian avoidance straight- road scenarios
16

---

(e) Traffic light recognition in the evening
(f) Emergency stop due to malfunctioning vehicle ahead
(g) Traffic light recognition under rainy weather conditions
geos DOS DJ
50
(h) Vehicle yielding right way
17

---

1./A cenmere
FOLL
(i) Merging into highway with yielding
4.0 bisil U. powa 14 ballic PEI., 5.40165 LO00
Merging into highway with parked vehicle ahead
14), -3.0 mrotte pottle S.83475440SL77S
231,
Baly Picket
Wily Roli
(k) Pedestrian violation crossing the road at night
(o Munii seda 8.75 aotenana LaNE FALIOA
noss
rom
(1) Obstacle avoidance when other vehicles merge into the lane
Figure 7: Visualization of representative closed-loop scenarios from the Bench2Drive benchmark. The wp way point denotes predicted trajectory points. The target is the mid-range goal issued by global route planner in simulation environment, encoded as input to the PGS planning network.
18

---

yotter MEN-0o-----00 39-00
FOLLON
(a) Failure avoid vehicle with opened door.
gpeed Koen GOS Pestle Andil 0.A. Inaiie FEll tists
(b) Failure to yield to an approaching emergency vehicle.
Figure 8: Representative failure cases due to incomplete perception and missing causal cues.
process be reproduced for each individual town by following the instructions in the Metric section of the Eval Tools
F PID Controller Configuration
We build upon the original PID controller parameters from Bench2Drive [0] and modify the aim point selection strategy. Instead of using fixed 4.0-meter target, we adopt a dynamic approach based on vehicle speed: selecting near aim point of 4.0 meters for low-speed scenarios (below 6.5 m/s) and a far aim point of 10.0 meters for high-speed scenarios (above 6.5 m/s). This design stems from the principle that at higher speeds, the vehicle requires longer look-ahead distance to follow the planned trajectory accurately. By providing sufficient foresight, the farther aim point facilitates smoother steering adjustments, thereby enhancing trajectory stability and reducing oscillations observed during high-speed maneuvers in our experiments.
G Experiments compute resources
All experiments were conducted with the following specifications:
Hardware Configuration
Each node is equipped with dual Intel(R) Xeon(R) Gold 6278C 2.60GHz processors, providing 52 physical cores and 104 threads per node. With 2 nodes in total, the system utilizes 4 CPU sockets and 208 logical processors.
CPU:
Memory: 512 GB RAM per node
GPU: 16 NVIDIA VI00 GPUs (32 GB each), with 8 GPUs per node across 2 nodes
19

---

Training Time
Stage 1 Training: Completed approximately day using 16 GPUs, equivalent to around 384 GPU hours.
Stage 2 Training: Completed in approximately day using 16 GPUs, equivalent to around 384 GPU hours.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction, we have thoroughly detailed the background, motivation, scope, main experimental results, and contributions of our work.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the work discussed in section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
21

---

Justification:
The full set of assumptions and complete proofs are provided in section 3 and section D of the main paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The proposed method implemented with a public closed-loop dataset benchmark. All the implementation details are reported in the main paper.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
22

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The datasets used this paper are publicly available, and the code will be released upon acceptance of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
wublic/guides/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https/cccccuclccuuueeeeeeeeeeeee Coaee//ddeeSebmmmcma ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental details are presented in section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We adopt standard evaluation datasets and metrics, which arcaccompanied by statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
23

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Computer resources are provided in both section 4 and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:h/pppppp..ppp.....o.opooooa
Answer: [Yes]
Justification:
The research conducted in the paper strictly conforms the code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The broader impacts of this work are discussed in the main paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The datasets and benchmarks used are all publicly available with licensees and are properly acknowledged in paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The code will be made public upon acceptance.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
26

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: No LLMs were involved in the development of the core methods presented in this paper.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
27


Reviews:
[{"review_id": "7N3apdGKtT", "reviewer": "Reviewer_qCVe", "summary": "The paper presents an algorithm to improve the popular imitation strategy for autonomous driving.  Specifically, it is argued that the conventional imitation learning strategy for autonomous driving is agnostic to the errors of safety concerns when minimizing the L2 loss during behavior cloning. This leads to critical safety errors in the output results. To address this, the paper introduces three kinds of losses during training to suppress such errors. Two of them aims to provide better target lane binding by quantizing the expert trajectory into lane-specific templates, and the third penalises the ego vehicle trajectory that collides with other dynamic road objects. Experiments on a closed-loop benchmark show that the proposed method can outperform other baselines in most scenarios under closed-loop metrics.", "questions": "- Will the quantisation of STPS cause some discontinuity in the target trajectory? How does that affect the performance?", "limitations": "See Strengths And Weaknesses.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "-  Strengths\n   - Interesting observation and motivation on the safety errors of behavior cloning for autonomous driving.\n   - Reasonable technical solutions and execution.\n   - Results look promising, with improvements over the baselines and other state-of-the-art methods with clear margins.\n\n- Weaknesses\n  - Some interesting cases are not tested. For instance, the first two introduced losses relies on map elements. What about unstructured cases? If would be helpful to show how PGS with only NTPS works in such cases.\n  - Some performance metric drops are not explained. It seems that the introduced losses may cause the efficiency and comforness degrade significantly in Table 1. What are the reasons for that? Shouldn't the inference cost of the proposed method the same as the VAD-base?\n  - Presentation, some notations are not clear or defined yet. E.g., what is the plus sign in $Q_{motion}$ between Lien 159 and Line 160? \"Limitation learning\" in Figure 1? It would also be helpful to provide some illustration for the quantities defined for section 3.1 through 3.4. Crucial equations should be numbered for easy reference too.", "was_revised": true, "final_justification": "The rebuttal addresses most my initial concerns and questions. I keep my original score.", "rebuttal": "## Response to Reviewer qCVe.\n\n> **Q1. For unstructured cases, show how PGS with only NTPS works.**\n\n**A1.** This is a reasonable suggestion, and thank you for pointing out our oversight. We have conducted additional experiments specifically addressing your concern by training a variant of our model (denoted as $PGS_{NTPS}$) using only the NTPS loss, thus entirely removing dependencies on structured map elements. As summarized in Table 1, compared to the complete PGS model, the NTPS-only variant demonstrates significantly improved performance in merging scenarios. This result can be attributed to NTPS’s strong guidance in collision-prone, complex intersection scenarios. Conversely, performance slightly decreased in overtaking scenarios, as NTPS may lead to overly conservative behavior when static obstacles are present in the ego lane. Upon further incorporating MTPS and STPS losses, structured road priors provide richer spatial guidance, effectively enlarging the decision space. This allows the model to learn more flexible maneuver patterns, such as overtaking by borrowing adjacent or opposing lanes. Consequently, the overall success rates improve in both _merging_ and _overtaking_ scenarios. Both quantitative and qualitative analyses confirm our original hypothesis: incorporating structured road priors significantly strengthens causal reasoning, enhancing decision robustness under structured driving conditions. In unstructured environments where road priors are unavailable, we expect the full PGS model to behave similarly to the NTPS-only variant.\n\n**Table 1: Ablation Study of the Proposed PGS Framework with NTPS-only PGS Model.**\n\n| **Method**             | **Avg. L2** | **Merging (%)** | **Overtaking (%)** | **Mean (%)** |\n| ---------------------- | ----------- | --------------- | ------------------ | ------------ |\n| VAD-Base               | 0.91        | 8.11            | 24.44              | 16.28        |\n| VAD-Tiny               | 1.15        | 9.33            | 11.11              | 10.22        |\n| $PGS_{Base}$           | 0.87        | 16.46           | 13.33              | 14.89        |\n| **$PGS_{NTPS}$**       | **0.90**    | **25.00**       | **6.67**           | **15.84**    |\n| $PGS_{Base+STPS}$      | 0.78        | 24.44           | 26.25              | 25.35        |\n| $PGS_{Base+STPS+MTPS}$ | 0.75        | 23.75           | 44.44              | 34.10        |\n| $PGS_{All}$            | 0.77        | 32.50           | 66.67              | 49.58        |\n| $PGS_{self}$           | 2.89        | 31.25           | 35.56              | 33.40        |\n> **Q2. Why the introduced losses may cause the efficiency and comforness degrade significantly in Table 1? Shouldn't the inference cost of the proposed method the same as the VAD-base?**\n\n**A2.** Thank you very much for this thoughtful comment. To clarify, the _efficiency_ metric is computed based on comparisons between the ego vehicle’s speed and that of surrounding traffic, rather than the inference cost. This terminology might have caused confusion. Since our method shares similar architectural complexity with VAD, it does not introduce additional computational overhead during inference. The _comfortness_ metric, on the other hand, is derived from acceleration and jerk measurements.\n\nFirst of all, prompted by your constructive question, we identified a critical issue in our evaluation process. Specifically, we inadvertently used an outdated version of the official `atomic_criteria.py` file (located at `Bench2Drive/scenario_runner/srunner/scenariomanager/scenarioatomics/atomic_criteria.py`). This earlier version incorrectly computed the efficiency metric using only cases with minimum-speed infractions (i.e., speed < 100%), while entirely omitting scenarios labeled as \"Perfect\" (i.e., completed without speed violations). This issue is confirmed by the JSON files included in our supplementary materials (`b.PGS_all_metric_files`).\n\nThe latest version of the official script, which we have now adopted, correctly follows the evaluation protocol described in the original paper by recording speed measurements at every 5% interval along the route to ensure comprehensive and fair metric computation.\n\nUnder the corrected protocol, our updated Efficiency and Comfortness scores are 180.9974 and 12.0039, respectively. Notably, our model achieves a new **state-of-the-art** result in Efficiency. These values will be updated in the final version of the paper.\n\nHowever, we would like to emphasize that Efficiency and Comfortness should be interpreted under comparable scenario success rates to ensure fair and meaningful comparisons. For example, if a vehicle becomes stuck at an intersection and fails to complete the scenario, the computed Comfortness may be deceptively high due to the lack of motion. Conversely, a model that completes the scenario at high speed—but with a collision—may report high Efficiency despite safety violations. Therefore, we argue that Driving Score and Success Rate are more indicative of real-world performance, and comparisons of secondary metrics such as Efficiency or Comfortness should be contextualized under similar success conditions.\n\nSecondly, in challenging driving scenarios such as overtaking by temporarily borrowing the opposing lane, our model prioritizes safety, often resulting in conservative maneuvers. Specifically, the model tends to decelerate or come to a full stop before accelerating rapidly to complete the maneuver safely. While this behavior closely resembles cautious human driving, it inevitably leads to increased acceleration and jerk, thereby negatively impacting the comfort score.\n\nMoreover, our method leverages lane-centerline-based pseudo trajectories as supervision signals. These pseudo trajectories effectively guide the model to adhere more accurately to road topology at intersections, thereby reducing erroneous lane choices. However, because these pseudo labels prioritize spatial correctness over temporal continuity, the resulting trajectories may exhibit slight discontinuities compared to human expert demonstrations, which further degrades the comfort metric. Addressing this temporal discontinuity to improve trajectory smoothness is a key part of our future work.\n\n\n> **Q3. Issues with notation clarity and paper formatting.**\n\n**A3.** We appreciate your careful review. We will revise the manuscript accordingly to improve the clarity of formatting, as per your suggestion. Specifically, the \"⊕\" sign indicates concatenation between features, The term “Limitation learning” in Figure 1 is a written error and will be corrected to “Imitation learning.”\n\n\n> **Q4. Will the quantisation of STPS cause some discontinuity in the target trajectory? How does that affect the performance?**\n\n**A4.** Your observation is correct. Quantizing spatial trajectories in STPS inevitably introduces minor discontinuities compared to smooth human expert trajectories. These discontinuities manifest as subtle velocity fluctuations, which contribute to reduced comfort. Addressing this concern, we plan to implement spatial smoothing techniques in future iterations to alleviate trajectory discontinuities and thus enhance comfort metrics."}, {"review_id": "tJEPLWWwvI", "reviewer": "Reviewer_M41J", "summary": "This paper proposes a new training paradigm for end-to-end autonomous driving, termed Perception-Guided Self-Supervision (PGS), which aims to mitigate causal confusion—a major limitation of imitation learning—by replacing expert trajectories with structured outputs from perception modules as supervision signals. The authors introduce three types of supervision:\nMTPS: Multi-Modal Trajectory Planning Self-Supervision, which reframes trajectory prediction as a lane selection problem.\nSTPS: Spatial Trajectory Planning Self-Supervision, which aligns the predicted trajectory with lane centerlines to avoid noisy human demonstration.\nNTPS: Negative Trajectory Planning Self-Supervision, which penalizes ego trajectories that intersect with future predicted trajectories of surrounding agents.\nExtensive experiments on the Bench2Drive closed-loop benchmark demonstrate that the proposed method outperforms prior state-of-the-art approaches, achieving significant improvements in both overall driving score and critical scenario success rate.", "questions": "1. Will the code be released? Open-sourcing the implementation would help the community reproduce and build upon the proposed method.\n\n2. Since the method relies heavily on perception outputs as supervision, how does it perform when perception results are inaccurate or noisy?\n\n3. Have the authors considered evaluating the method on real-world datasets such as NAVSIM to validate its applicability beyond simulation?\n\n4. Typo in Figure 1: Should \"Limitation learning\" be \"Imitation learning\"? Please clarify and correct if necessary.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The paper addresses the key challenge of causal confusion in end-to-end driving—a well-known problem that limits real-world deployment. The motivation is clear and well justified.\n\n2. The use of structured perception outputs (e.g., lane centerlines, predicted agent motion) as training supervision is intuitive, principled, and avoids the pitfalls of noisy expert demonstrations.\n\nWeaknesses:\n1. The proposed paradigm heavily depends on accurate perception outputs (e.g., lane centerlines, object trajectories). While this is acknowledged in the limitations, it raises concerns about robustness under real-world perception errors, especially in long-tail or rare scenarios.\n\n2. Terminology of “self-supervision” may be misleading. The term “self-supervision” typically refers to learning from raw data without any external labels. In this work, supervision is provided by another part of the network (perception), which is trained with labeled data. Some readers may find this use of the term confusing or inconsistent with common definitions.", "was_revised": true, "final_justification": "My concerns are mostly resolved, so I will maintain a positive rating. I agree with Reviewer RHTg and look forward to seeing results on a real-world dataset, such as NAVSIM, in the revised manuscript.", "rebuttal": "## Response to Reviewer M41J.\n\n> **Q1. What happens if the perception module fails catastrophically?**\n\n**A1.** We appreciate your thoughtful comment. We fully recognize that our framework depends on perception outputs, which naturally raises questions about robustness under perception errors. We would like to clarify our motivation, underlying assumptions, and the scope of our contributions in relation to this concern. \n\nThe design of our paradigm is motivated by a key empirical insight: in existing end-to-end driving systems, perception is rarely the dominant performance bottleneck. In the majority of scenarios, modern perception modules already provide sufficiently accurate outputs for downstream tasks. Despite this, end-to-end systems often suffer from causal confusion in their decision-making process—where models latch onto spurious correlations rather than semantically meaningful or causally relevant signals.\n\nBased on this observation, the primary objective of our work is to improve causal modeling within the planning and control module and to verify its impact on the performance of end-to-end systems. And we believe that the proposed improvements can enhance ego vehicle behavior in most scenarios where perception is effective.\n\nWe acknowledge that, like all perception-dependent systems, our method inherits the limitations of upstream perception. In rare or long-tail scenarios where perception fails, performance degradation is expected. However, this sensitivity is not unique to our method; rather, it is an intrinsic property of any perception-dependent autonomous driving approach—whether end-to-end or modular. Errors originating from the perception module naturally propagate downstream, cumulatively affecting planning and control decisions.\n\nCrucially, our framework neither introduces additional vulnerabilities nor exacerbates existing issues beyond those inherently present in conventional perception-driven approaches. In fact, enhancements in perception robustness directly translate into improved performance for our approach. Therefore, we see our contribution as complementary and orthogonal to existing perception improvement efforts.\n\nIn summary, while acknowledging the inherent sensitivity to perception quality, we emphasize that our method primarily targets the critical issue of causal confusion—demonstrably improving system robustness under standard perception conditions. Improving perception robustness itself remains a broader and longstanding goal in the autonomous driving research community, and advancements in this area would directly benefit the effectiveness of our proposed framework. \n\n> **Q2. The usage of the term “self-supervision” may be inconsistent with its conventional definition, as the supervision in this work comes from perception outputs trained with labels.**\n\n**A2.** Thank you for raising this insightful question. We had indeed discussed this point extensively prior to the submission, as your description precisely captures. We ultimately chose to use the term self-supervision because the main contributions of our work are organized around a novel learning paradigm for the planning and control module. This terminology was intended to emphasize the conceptual distinction from existing imitation learning paradigms. As a result, we finalized the term perception guided self-supervision after careful deliberation.\n\nThis term is meant to clarify the differences between our approach and standard definitions of self-supervision: specifically, our supervisory signals are derived from an on-the-shelf perception module. We will revise the manuscript to clarify the use of this terminology and more explicitly articulate its intended meaning, in order to avoid potential confusion.\n\n> **Q3. Will the code be released?**\n\n**A3.** Yes, we fully agree that open-sourcing the implementation would benefit the community by facilitating reproducibility and further research. We will release the code upon acceptance of the paper. \n\n>**Q4. How does the method perform when perception results are inaccurate or noisy?**\n\nThank you for your question. This concern closely aligns with the point raised in Weakness 1 of your comment. For a detailed response, please refer to our answer to Q1, where we provide a comprehensive discussion addressing this issue.\n\n> **Q5. Have the authors considered evaluating the method on real-world datasets such as NAVSIM?**\n\n**A5.** Thank you for your valuable suggestion.First, existing studies have demonstrated that although methods evaluated under open-loop settings may yield promising L2 error and collision rates, such models often exhibit significant performance degradation when deployed in real-world scenarios or evaluated under closed-loop settings. Therefore, our work focuses primarily on enhancing end-to-end model performance under closed-loop evaluation, which we believe holds greater practical value for autonomous driving systems.\n\nFor this reason, we chose Bench2Drive (B2D) as our primary benchmark. On one hand, this dataset encompasses a wide range of complex traffic scenarios, including 44 interactive categories such as _ParkingExit, ParkingCutIn, ParkedObstacleTwoWays, and HazardAtSideLane_, along with 23 different weather conditions (e.g., _sunny, foggy, rainy_), effectively covering the majority of real-world driving contexts. The evaluation set ensures comprehensive coverage across all categories. On the other hand, the evaluation is conducted in a fully closed-loop manner—meaning the behavior of surrounding agents dynamically responds to the ego vehicle’s actions—thereby better aligning with the dynamics of real-world deployment.\n\nIn contrast, the NAVSIM dataset is derived from nuPlan, which provides approximately 1,200 hours of human driving data. However, it does not directly include adverse conditions such as heavy rain or nighttime driving.  Furthermore, its evaluation is semi-closed-loop—or arguably still open-loop—as the behavior of other agents does not react to the ego vehicle's decisions. This makes the evaluation metrics insufficient to fully reflect the algorithm’s performance in real-world closed-loop conditions.\n\nSecond, the core motivation of our work is to address the causal confusion issues that frequently arise in closed-loop or real-world testing. A classic example of such confusion is observed in how autonomous vehicles behave at traffic light intersections: models suffering from causal confusion fail to learn the correct association between traffic light signals (red/green) and the ego vehicle's actions, instead basing their decisions on the behavior of surrounding vehicles. This represents one of the most critical and realistic forms of causal confusion, which is not captured in NAVSIM's evaluation metrics. NAVSIM focuses solely on metrics such as collision rate, drivable area violations, efficiency, and target goal achievement, without assessing whether the model correctly understands the causal relationship between traffic signals and ego behavior at traffic lights or stop lines. In contrast, the B2D benchmark explicitly reflects this type of causal failure: confusion regarding traffic light cues can lead to vehicles becoming stuck or violating traffic regulations, ultimately resulting in task failure. Furthermore, B2D includes a dedicated set of scenarios involving interactions with traffic signals and stop signs (i.e., the “traffic sign” scenario group), which allows for a more targeted and comprehensive evaluation of a model’s causal reasoning capabilities.\n\nThus, although B2D may have a larger domain gap from real-world data in terms of visual appearance and sensor modality, we argue that from an algorithmic perspective, B2D and its evaluation protocol more accurately reflect real-world performance.\n\nIn summary, due to the comprehensiveness of B2D's scenario coverage, its alignment with real-world deployment dynamics, and its ability to reflect the most challenging causal reasoning problems in end-to-end driving, we selected B2D as a more suitable and challenging benchmark. In contrast, NAVSIM may not be well-suited for addressing the specific issues our work targets.\n\nLastly, we sincerely hope to further validate and extend our method on benchmarks that combine real-world data with closed-loop evaluation. We have assessed the feasibility of adapting our method to NAVSIM, and found it would involve substantial effort, including dataset format conversion, constructing intermediate supervision signals (e.g., verifying whether NAVSIM includes lane-level annotations such as lane centerlines), validating perception-guided self-supervised signals (e.g., tuning thresholds for relevant lane filtering modules, and visual verification of MTPS, STPS, and NTPS losses). Given the limited time available during the rebuttal phase, it is technically infeasible to robustly adapt our method to NAVSIM within this timeframe.\n\nLuckily, we have noted that the B2D team announced in December 2024 the upcoming release of an extended version—Bench2Drive-R—which will include both real-world and simulation data. This extended benchmark will better fulfill both data-level and evaluation-level alignment with real-world conditions. We plan to validate our method on this dataset as soon as it becomes publicly available.\n\n> **Q6. Should \"Limitation learning\" be \"Imitation learning\" in Figure 1?**\n\n**A6.** Thank you for your careful review. Yes, this is a written error and should be corrected to “Imitation learning.” We will fix it in the revised version."}, {"review_id": "6DRnLSdBYn", "reviewer": "Reviewer_RHTg", "summary": "This paper introduces PGS, a new training paradigm for end-to-end autonomous driving. Rather than relying solely on expert trajectories for supervision, the authors propose using structured and interpretable perception outputs—such as lane centerlines and predicted motions of other agents—as both inputs and supervisory signals for decision-making modules. Three self-supervision losses are designed. Experimental results on the challenging Bench2Drive benchmark demonstrate that this perception-aligned self-supervised approach outperforms sota methods.", "questions": "See weaknesses.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strenths:\n1. The PGS approach is implemented via three well-defined loss functions (MTPS, STPS, NTPS) that are explained in detail mathematically and graphically.\n2. The approach is evaluated on the g closed-loop benchmark, which is more realistic than common open-loop metrics. Table 1 shows significant quantitative improvements in driving scores and success rates. Table 3 eliminates the effects of each component. The authors provide more examples in the supplementary material.\n3. The core idea of ​​PGS does not depend on certain architectures; instead, it is compatible with standard BEV-based processes and can be generalized to other frameworks.\n\nWeaknesses:\n- There seems to be no empirical evaluation of robustness to noise or imperfect perception, which may occur in real applications. For example, in safety-critical scenarios, does the system performance collapse if the perception system fails to detect some vehicles?\n\n- Did the authors try to implement and test PGS in any real-world datasets/simulations (there are indeed some recent works that perform closed-loop simulations on real-world scenarios, i.e. DriveArena, NavSim)?", "was_revised": true, "final_justification": "I assume the authors will incorporate this core discussion of real-world transferability into future versions of the manuscript (if the paper is accepted).\n\nAuthors  response fully address my concerns, and I  maintain my positive rating", "rebuttal": "## Response to Reviewer RHTg.\n\n> **Q1. Does the system performance collapse if the perception system fails to detect some vehicles?**\n\n**A1.** Thank you for your valuable question. This is indeed a critical consideration, especially in the context of real-world deployment. A similar concern was also raised by Reviewer M41J, Q1, to which we have provided a detailed response. We kindly refer you to that reply for a comprehensive discussion of this issue.\n\n> **Q2. Did the authors try to implement and test PGS in any real-world datasets/simulations such as DriveArena, NavSim?**\n\n**A2.** Thank you for your valuable suggestion. We have carefully considered your comment. DriveArena is currently primarily used for data generation and reinforcement learning in closed-loop training settings. In contrast, our method is specifically developed for training on offline datasets—collected from real vehicles—to enhance closed-loop performance via novel algorithmic and architectural improvements. Indeed, training on off-line datasets remains the predominant paradigm for deploying end-to-end autonomous driving models both in academia and industry, particularly due to the practical constraints associated with online training. Furthermore, DriveArena currently lacks standardized evaluation protocols and publicly available benchmarks, which makes it difficult to adopt at this stage. Nonetheless, we fully agree that closed-loop training represents a promising future research direction, and we intend to explore this in subsequent extensions of our work. Regarding experiments on the NavSim, please refer to our response to Reviewer M41J, Q5."}, {"review_id": "FrB76FYZt9", "reviewer": "Reviewer_gAQK", "summary": "Conventional imitation-learning-based end-to-end autonomous driving systems achieve strong performance in open-loop evaluations by leveraging large-scale expert demonstrations, yet they suffer severe degradation in closed-loop (real-time) settings due to causal confusion. To address this issue, the authors introduce Perception-Guided Self-Supervision (PGS). PGS treats perception outputs, such as lane centerlines and predicted trajectories of nearby agents, as the primary supervisory signals for the decision-making module, and applies positive and negative self-supervision to align the ego vehicle’s trajectory with these perception cues, thereby alleviating causal confusion.", "questions": "1. The proposed PGS is applied only to the Ego Planning Head. Could it also be leveraged for the branches that output object trajectories and mode-classification scores?\n2. What does $\\beta$ represent in line 240?\n3. Section 4.3 reports high driving-score and success-rate figures, yet noticeably lower scores for efficiency and comfort. What accounts for this imbalance?\n4. Are “PPGSS” and “PGS” in Tables 1 and 2 the same? If not a typo, please clarify what PPGSS stands for.\n5. The paper says it adopts VAD as the backbone architecture and retains the same training objectives outside the PGS paradigm. Nevertheless, the ablation study in Section 4.4 shows a performance gap between VAD-Base and PGS_Base, even though they are trained with identical losses. What causes this discrepancy?", "limitations": "The authors adequately discuss the limitations in the paper and throughout the work. They point out that their proposed framework is heavily dependent on perception accuracy and generalization, and list this reliance as one of its key limitations.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths\n- The paper is clearly written overall, and the figures and equations are well organized, making the proposed framework easy to understand.\n- The experiments demonstrate higher task success rates than previous studies.\n- The authors include extensive visualizations of the results and provide a thorough analysis of limitations and failure cases.\n\n## Weaknesses\n- While the related works discussed in Section 2.3 and studies such as [1] encourage networks to autonomously discover causal cues directly from data, the proposed PGS framework introduces auxiliary losses based on perception module outputs, such as lane centerlines and the predicted future trajectories of surrounding objects, to inject elements that are considered important from a human perspective, like lane keeping and collision avoidance. Although these elements are not implemented as manually crafted rules or explicit filters, but rather are incorporated into the learning process through network training, and can be seen as a practical compromise to reduce imitation noise and improve safety, I am somewhat concerned that this explicit intervention may diminish the strength of the work in comparison to purely end-to-end imitation learning approaches.\n- I believe the paper’s contribution could be emphasized further by adding experiments in environments built on real-world datasets such as nuScenes [2], in addition to the CARLA simulation where surrounding vehicles behave ideally.\n\nReferences:\n\n- [1] De Haan, Pim, Dinesh Jayaraman, and Sergey Levine. \"Causal confusion in imitation learning.\" *Advances in neural information processing systems* 32 (2019).\n- [2] Caesar, Holger, et al. \"nuscenes: A multimodal dataset for autonomous driving.\" *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*. 2020.", "was_revised": true, "final_justification": "I will maintain my original rating. The authors have proposed an approach to improve existing end-to-end autonomous driving methods and have demonstrated its effectiveness through experiments. As some other reviewers noted, questions remain regarding its applicability to real-world scenarios, and there may be debate over whether the method fundamentally resolves the causal confusion issue. Nonetheless, I continue to hold a positive overall view of the work and will therefore keep my initial rating.", "rebuttal": "## Response to Reviewer gAQK.\n> **Q1. Do this explicit intervention may diminish the strength of the work in comparison to purely end-to-end imitation learning approaches?**\n\n**A1.** Thank you for your insightful feedback. While our PGS framework introduces auxiliary losses derived from perception outputs—such as lane geometry or predicted trajectories—it remains a fully trainable end‑to‑end model. All perception, planning, and policy modules are jointly optimized without introducing hard-coded or frozen components.\n\nThis approach offers a principled way to improve safety and reduce imitation noise by guiding the network to focus on causal features (e.g., lane following and collision avoidance cues), without sacrificing flexibility or learning capacity. \n\nBy contrast, purely end-to-end imitation often encounters causal confusion, where models learn spurious correlations rather than underlying causal mechanisms—especially under distributional shift. As empirically documented, more observational inputs can indeed degrade performance if the model incorrectly infers cause from effect. The auxiliary supervision in PGS can help mitigate such confusion by nudging the model toward robust, causally relevant patterns during training.\n\n> **Q2. Adding experiments in environments built on real-world datasets such as nuScenes will be ideally.**\n\n**A2.** Thank you for this valuable recommendation. We fully acknowledge the significance of evaluating autonomous driving methods on real-world datasets. Indeed, nuScenes is a highly influential benchmark with rich annotations that has notably advanced research in perception and prediction. However, for assessing the performance of end-to-end driving systems, nuScenes exhibits several key limitations:\n\nFirstly, nuScenes provides primarily open-loop data. Recent studies (e.g., Reference [19]) have explicitly demonstrated that open-loop performance often fails to reflect realistic closed-loop capabilities accurately. Remarkably, simple baseline models—such as a single-layer MLP conditioned only on the ego vehicle’s current state—can achieve superficially strong open-loop performance metrics on nuScenes, underscoring this limitation.\n\nSecondly, nuScenes predominantly comprises simple, straight-driving scenarios, which constrains its capacity to probe causal reasoning and to evaluate generalization in rare yet safety-critical situations.\n\nIn contrast, our work focuses explicitly on improving closed-loop performance in interactive and high-risk settings. Bench2Drive encompasses a wide range of driving scenarios and offers relatively comprehensive evaluation metrics, which constitutes the primary reason for selecting it as our main evaluation benchmark.\n\nLastly, we sincerely hope to further validate and extend our method on benchmarks that combine real-world data with closed-loop evaluation.As part of our future work, we plan to explore real-world evaluation using Bench2Drive-R, which provides real-world driving data with richer map annotations and supports closed-loop benchmarking. We believe this will offer a more principled and reliable framework for evaluating decision-making performance in the real world. \n\n> **Q3. Could PGS also be leveraged for the branches that output object trajectories and mode-classification scores?**\n\n**A3.** Thank you for your valuable suggestion. The proposed PGS framework can indeed be extended to the mode-classification branch, as it shares similar decision-level semantics with ego planning.\n\nHowever, applying PGS directly to object trajectory prediction is less straightforward. In ego planning, the future intention (e.g., turning left or right in form of target ponint) is clearly defined and remains applicable during both training and deployment phases. In contrast, for surrounding objects, such intent information is typically unavailable or ambiguous, especially at intersections. To effectively apply PGS in this context, additional intent estimation or behavior modeling would be required. We consider this a promising direction for future work. We believe exploring such extensions could further enhance the model’s holistic scene understanding and planning reliability.\n\n> **Q4. What does $\\beta$ represent in line 240?**\n\n**A4.** $\\beta$ represents the threshold distance used to determine potential collisions. In our model, we set $\\beta$ = 0.5 meters to define a safe margin between agents.\n\n> **Q5. What accounts for the imbalance of high driving-score and success-rate and lower scores for efficiency and comfort?**\n\n**A5.** Thank you for your thoughtful question. Please refer to our response to Reviewer qCVeQ, Q2 for a detailed explanation.\n\n> **Q6. Are “PPGSS” and “PGS” in Tables 1 and 2 the same?**\n\n**A6.** We really appreciate your attention to detail. Yes, \"PPGSS\" is a typo and will be corrected to \"PGS\" in the revised version.\n\n> **Q7. Why there is a performance gap between VAD-Base and PGS_Base?**\n\n**A7.** Thank you for your thoughtful question. As stated in line 154 of the main paper, the proposed $ PGS_{base}$ model does not adopt the cascaded prediction–decision architecture used in VAD. Instead, it employs a unified decoder for both ego trajectory planning and object motion prediction. This architectural difference inherently introduces changes in representation and decision coupling. \n\nFurthermore, we would like to offer an important clarification. Our model does not incorporate the auxiliary planning losses used in the VAD framework. The original intent was to adopt the full loss suite from VAD’s perception module only. We sincerely apologize for the confusion and will revise the manuscript to clearly state this distinction."}]

Author rebuttal:

Meta-review: This paper introduces Perception-Guided Self-Supervision (PGS), a novel approach to mitigate causal confusion in end-to-end autonomous driving by leveraging perception outputs as supervisory signals. The authors effectively demonstrate the method's potential, achieving state-of-the-art results on the Bench2Drive benchmark and providing strong empirical evidence for its effectiveness. The work addresses a fundamental limitation in imitation learning and offers a principled way to improve causal modeling.

Reviewers highlighted the significance of the problem addressed, the novelty of the approach, and the quantitative improvements achieved.

However, some concerns were raised, primarily concerning the robustness of the method to perception errors in real-world scenarios, the clarity of the term "self-supervision" as used in the paper, and the limited evaluation on real-world datasets. The authors provided detailed responses to these concerns, clarifying their methodology, addressing evaluation issues, and explaining their rationale for benchmark selection. Their explanations regarding real-world transferability and the necessity of closed-loop evaluation for addressing causal confusion were well-received.

Overall, the paper presents a technically sound and significant contribution to the field of autonomous driving. The authors have addressed reviewer concerns, and the proposed PGS framework offers a promising direction for future research. The reviewers' final assessments reflect a positive view of the work.
Decision: Accept (poster)