Title: Prioritizing Perception-Guided Self-Supervision: A New Paradigm for Causal Modeling in End-to-End Autonomous Driving
Venue: NeurIPS 2025
Accepted: yes
Keywords: End to end autonomous driving
Abstract: End-to-end autonomous driving systems, predominantly trained through imitation learning, have demonstrated considerable effectiveness in leveraging large-scale expert driving data. Despite their success in open-loop evaluations, these systems often exhibit significant performance degradation in closed-loop scenarios due to causal confusion. This confusion is fundamentally exacerbated by the overreliance of the imitation learning paradigm on expert trajectories, which often contain unattributable noise and interfere with the modeling of causal relationships between environmental contexts and appropriate driving actions.

To address this fundamental limitation, we propose Perception-Guided Self-Supervision (PGS)—a simple yet effective training paradigm that leverages perception outputs as the primary supervisory signals, explicitly modeling causal relationships in decision-making. The proposed framework aligns both the inputs and outputs of the decision-making module with perception results—such as lane centerlines and the predicted motions of surrounding agents—by introducing positive and negative self-supervision for the ego trajectory. This alignment is specifically designed to mitigate causal confusion arising from the inherent noise in expert trajectories.

Equipped with perception-driven supervision, our method—built on a standard end-to-end architecture—achieves a Driving Score of 78.08 and a mean success rate of 48.64\% on the challenging closed-loop Bench2Drive benchmark, significantly outperforming existing state-of-the-art methods, including those employing more complex network architectures and inference pipelines. These results underscore the effectiveness and robustness of the proposed PGS framework, and point to a promising direction for addressing causal confusion and enhancing real-world generalization in autonomous driving.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nConventional imitation-learning-based end-to-end autonomous driving systems achieve strong performance in open-loop evaluations by leveraging large-scale expert demonstrations, yet they suffer severe degradation in closed-loop (real-time) settings due to causal confusion. To address this issue, the authors introduce Perception-Guided Self-Supervision (PGS). PGS treats perception outputs, such as lane centerlines and predicted trajectories of nearby agents, as the primary supervisory signals for the decision-making module, and applies positive and negative self-supervision to align the ego vehicle’s trajectory with these perception cues, thereby alleviating causal confusion.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- The paper is clearly written overall, and the figures and equations are well organized, making the proposed framework easy to understand.\n- The experiments demonstrate higher task success rates than previous studies.\n- The authors include extensive visualizations of the results and provide a thorough analysis of limitations and failure cases.\n\n## Weaknesses\n- While the related works discussed in Section 2.3 and studies such as [1] encourage networks to autonomously discover causal cues directly from data, the proposed PGS framework introduces auxiliary losses based on perception module outputs, such as lane centerlines and the predicted future trajectories of surrounding objects, to inject elements that are considered important from a human perspective, like lane keeping and collision avoidance. Although these elements are not implemented as manually crafted rules or explicit filters, but rather are incorporated into the learning process through network training, and can be seen as a practical compromise to reduce imitation noise and improve safety, I am somewhat concerned that this explicit intervention may diminish the strength of the work in comparison to purely end-to-end imitation learning approaches.\n- I believe the paper’s contribution could be emphasized further by adding experiments in environments built on real-world datasets such as nuScenes [2], in addition to the CARLA simulation where surrounding vehicles behave ideally.\n\nReferences:\n\n- [1] De Haan, Pim, Dinesh Jayaraman, and Sergey Levine. \"Causal confusion in imitation learning.\" *Advances in neural information processing systems* 32 (2019).\n- [2] Caesar, Holger, et al. \"nuscenes: A multimodal dataset for autonomous driving.\" *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*. 2020.\n\nQUESTIONS:\n1. The proposed PGS is applied only to the Ego Planning Head. Could it also be leveraged for the branches that output object trajectories and mode-classification scores?\n2. What does $\\beta$ represent in line 240?\n3. Section 4.3 reports high driving-score and success-rate figures, yet noticeably lower scores for efficiency and comfort. What accounts for this imbalance?\n4. Are “PPGSS” and “PGS” in Tables 1 and 2 the same? If not a typo, please clarify what PPGSS stands for.\n5. The paper says it adopts VAD as the backbone architecture and retains the same training objectives outside the PGS paradigm. Nevertheless, the ablation study in Section 4.4 shows a performance gap between VAD-Base and PGS_Base, even though they are trained with identical losses. What causes this discrepancy?\n\nLIMITATIONS:\nThe authors adequately discuss the limitations in the paper and throughout the work. They point out that their proposed framework is heavily dependent on perception accuracy and generalization, and list this reliance as one of its key limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper introduces PGS, a new training paradigm for end-to-end autonomous driving. Rather than relying solely on expert trajectories for supervision, the authors propose using structured and interpretable perception outputs—such as lane centerlines and predicted motions of other agents—as both inputs and supervisory signals for decision-making modules. Three self-supervision losses are designed. Experimental results on the challenging Bench2Drive benchmark demonstrate that this perception-aligned self-supervised approach outperforms sota methods.

STRENGTHS AND WEAKNESSES:
Strenths:
1. The PGS approach is implemented via three well-defined loss functions (MTPS, STPS, NTPS) that are explained in detail mathematically and graphically.
2. The approach is evaluated on the g closed-loop benchmark, which is more realistic than common open-loop metrics. Table 1 shows significant quantitative improvements in driving scores and success rates. Table 3 eliminates the effects of each component. The authors provide more examples in the supplementary material.
3. The core idea of ​​PGS does not depend on certain architectures; instead, it is compatible with standard BEV-based processes and can be generalized to other frameworks.

Weaknesses:
- There seems to be no empirical evaluation of robustness to noise or imperfect perception, which may occur in real applications. For example, in safety-critical scenarios, does the system performance collapse if the perception system fails to detect some vehicles?

- Did the authors try to implement and test PGS in any real-world datasets/simulations (there are indeed some recent works that perform closed-loop simulations on real-world scenarios, i.e. DriveArena, NavSim)?

QUESTIONS:
See weaknesses.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new training paradigm for end-to-end autonomous driving, termed Perception-Guided Self-Supervision (PGS), which aims to mitigate causal confusion—a major limitation of imitation learning—by replacing expert trajectories with structured outputs from perception modules as supervision signals. The authors introduce three types of supervision:\nMTPS: Multi-Modal Trajectory Planning Self-Supervision, which reframes trajectory prediction as a lane selection problem.\nSTPS: Spatial Trajectory Planning Self-Supervision, which aligns the predicted trajectory with lane centerlines to avoid noisy human demonstration.\nNTPS: Negative Trajectory Planning Self-Supervision, which penalizes ego trajectories that intersect with future predicted trajectories of surrounding agents.\nExtensive experiments on the Bench2Drive closed-loop benchmark demonstrate that the proposed method outperforms prior state-of-the-art approaches, achieving significant improvements in both overall driving score and critical scenario success rate.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper addresses the key challenge of causal confusion in end-to-end driving—a well-known problem that limits real-world deployment. The motivation is clear and well justified.\n\n2. The use of structured perception outputs (e.g., lane centerlines, predicted agent motion) as training supervision is intuitive, principled, and avoids the pitfalls of noisy expert demonstrations.\n\nWeaknesses:\n1. The proposed paradigm heavily depends on accurate perception outputs (e.g., lane centerlines, object trajectories). While this is acknowledged in the limitations, it raises concerns about robustness under real-world perception errors, especially in long-tail or rare scenarios.\n\n2. Terminology of “self-supervision” may be misleading. The term “self-supervision” typically refers to learning from raw data without any external labels. In this work, supervision is provided by another part of the network (perception), which is trained with labeled data. Some readers may find this use of the term confusing or inconsistent with common definitions.\n\nQUESTIONS:\n1. Will the code be released? Open-sourcing the implementation would help the community reproduce and build upon the proposed method.\n\n2. Since the method relies heavily on perception outputs as supervision, how does it perform when perception results are inaccurate or noisy?\n\n3. Have the authors considered evaluating the method on real-world datasets such as NAVSIM to validate its applicability beyond simulation?\n\n4. Typo in Figure 1: Should \"Limitation learning\" be \"Imitation learning\"? Please clarify and correct if necessary.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an algorithm to improve the popular imitation strategy for autonomous driving.  Specifically, it is argued that the conventional imitation learning strategy for autonomous driving is agnostic to the errors of safety concerns when minimizing the L2 loss during behavior cloning. This leads to critical safety errors in the output results. To address this, the paper introduces three kinds of losses during training to suppress such errors. Two of them aims to provide better target lane binding by quantizing the expert trajectory into lane-specific templates, and the third penalises the ego vehicle trajectory that collides with other dynamic road objects. Experiments on a closed-loop benchmark show that the proposed method can outperform other baselines in most scenarios under closed-loop metrics.\n\nSTRENGTHS AND WEAKNESSES:\n-  Strengths\n   - Interesting observation and motivation on the safety errors of behavior cloning for autonomous driving.\n   - Reasonable technical solutions and execution.\n   - Results look promising, with improvements over the baselines and other state-of-the-art methods with clear margins.\n\n- Weaknesses\n  - Some interesting cases are not tested. For instance, the first two introduced losses relies on map elements. What about unstructured cases? If would be helpful to show how PGS with only NTPS works in such cases.\n  - Some performance metric drops are not explained. It seems that the introduced losses may cause the efficiency and comforness degrade significantly in Table 1. What are the reasons for that? Shouldn't the inference cost of the proposed method the same as the VAD-base?\n  - Presentation, some notations are not clear or defined yet. E.g., what is the plus sign in $Q_{motion}$ between Lien 159 and Line 160? \"Limitation learning\" in Figure 1? It would also be helpful to provide some illustration for the quantities defined for section 3.1 through 3.4. Crucial equations should be numbered for easy reference too.\n\nQUESTIONS:\n- Will the quantisation of STPS cause some discontinuity in the target trajectory? How does that affect the performance?\n\nLIMITATIONS:\nSee Strengths And Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-07"}