RAP: 3D Rasterization Augmented End-to-End Planning
Authors: Lan Feng
Yang Gao
Eloi Zablocki
Quanyi Li
Wuyang Li
Sichao Liu
Matthieu Cord
Alexandre Alahi
Venue: ICLR 2026 Poster (2026)
Keywords: Autonomous Driving
Planning
Sim-to-Real
Primary area: applications to robotics, autonomy, planning
TL;DR: Photorealism is unnecessary for robust end-to-end driving; our RAP framework leverages lightweight rasterization and feature alignment to scale training with large-scale synthetic samples, achieving state-of-the-art performance across benchmarks.

Abstract: Imitation learning for end-to-end driving trains policies only on expert demonstrations. Once deployed in a closed loop, such policies lack recovery data: small mistakes cannot be corrected and quickly compound into failures. A promising direction is to generate alternative viewpoints and trajectories beyond the logged path. Prior work explores photorealistic digital twins via neural rendering or game engines, but these methods are prohibitively slow and costly, and thus mainly used for evaluation. In this work, we argue that photorealism is unnecessary for training end-to-end planners. What matters is semantic fidelity and scalability: driving depends on geometry and dynamics, not textures or lighting. Motivated by this, we propose 3D Rasterization, which replaces costly rendering with lightweight rasterization of annotated primitives, enabling augmentations such as counterfactual recovery maneuvers and cross-agent view synthesis. To transfer these synthetic views effectively to real-world deployment, we introduce a Raster-to-Real (R2R) feature-space alignment that bridges the sim-to-real gap at the representation level. Together, these components form the Rasterization Augmented Planning (RAP) pipeline, a scalable data augmentation framework for planning. RAP achieves state-of-the-art closed-loop robustness and long-tail generalization, ranking 1st on four major benchmarks: NAVSIM v1/v2, Waymo Open Dataset Vision-based E2E Driving, and Bench2Drive. Our results demonstrate that lightweight rasterization with feature alignment suffices to scale end-to-end training, offering a practical alternative to photorealistic rendering. Project page: https://alan-lanfeng.github.io/RAP/.

Published as a conference paper at ICLR 2026
RAP: 3D RASTERIZATION AUGMENTED END-TO-END PLANNING
Lan Fengbt Yang Gao
Éloi Zablocki
Quanyi
Wuyang Li
Sichao Liu'i
Matthieu 23rd
Alexandre Alahi
EPFL. Switzerland ffirs firstname lastea
Valeo. France
3 SSoroonne Université, France
ffirstname. lastname@valeo.com
ABSTRACT
Imitation learning for end-to-end driving trains policies only on expert demonstra- tions. Once deployed in a closed loop, such policies lack recovery data: small mistakes cannot be corrected and quickly compound into failures. A promis- ing direction is to generate alternative viewpoints and trajectories beyond the logged path. Prior work explores photorealistic digital twins via neural render- ing or game engines, but these methods are prohibitively slow and costly, and thus mainly used for evaluation. In this work, we argue that photorealism is unnecessary for training end-to-end planners, What matters is semantic fidelity and scalability: driving depends on geometry and dynamics, ing. Motivated by this, we propose 3D Rasterization, which replaces costly ren- dering with lightweight rasterization of annotated primitives, enabling augmen- tations such as counterfactual recovery maneuvers and cross-agent view synthe- sis. To transfer these synthetic views effectively to real-world deployment, we introduce a Raster-to-Rea feature-space alignment that bridges the sim-to-real gap in the feature space. Together, these components form the Rasterization Augmented Planning (RAP), scalable data augmentation pipeline for plan- ning. RAP achieves state-of-the-ann closed-loop robustness and long-tail gener- alization, ranking 1st four major benchmarks: NAVSIM v1/v2, Waymo Open Dataset Aision-based E2E Driving, and Bench2Drive. Our results show that lightweight rasterization with feature alignment suffices scale E2E train- ing, offering a practical alternative to photorealistic rendering. Project page: attps:// aal//alan-aaaaa anfeng. github........ io/RAP/
textures light-
INTRODUCTION
End-to-end (E2E) autonomous driving maps sensory inputs directly waypoints control commands, offering a scalable alternative to modular pipelines (Hu et al. [023] Diang et al.] [023] Bartoccioni al. 2025). Most existing methods rely offline imitation learning (II.), where poli- cies are trained only on expert demonstrations from large-scale logs. While effective in open-loop evaluations, such training suffers from covariate shift (Ross et al., [011] and provides few recovery examples. Once deployed in a closed-loop, small errors cannot be corrected and quickly escalate into unrecoverable states, leaving IL-based planners brittle in practice (Chen et al. 2024
A natural solution is to augment training with synthetic scenarios. Recent work has explored pho- torealistic digital twins built with 3D neural rendering, such as NeRFs (Mildenhall al.. 2020) or Gaussian Splatting (Kerbl et all]]202), as well as engine-based simulators (Dosovitskiy et al.) 2017]]
Li al. 2023b). These pipelines enable counterfactual augmentation and closed-loop training by synthesizing inputs beyond the logged trajectory. By producing synthetic views that are visually indistinguishable from real images (Figure 1| left), these techniques achieve superior fidelity com- pared to simulator-based reconstructions (Li et al. 2023a), which also enable closed-loop rollout but fail to match visual appearance for the E2E model inference. However, despite their visual fidelity, they remain prohibitively slow and costly, making large-scale training impractical. In practice, they
are mostly restricted to policy evaluation (Ljungbergh et al. 2024 Cao et al. 2025 Zhou et al. 2024 Jiang al. 2025)

---

Published as a conference paper at ICLR 2026
Minimizing Gap
Space
Neural/Engine based
Minimizing Gap Feature Space
3D Rasterization
Driving Logs
X High Computational Cost
Driving Loge
Scalable
Training free
Figure I: Comparison of rendering paradigms for end-to-end driving, Neural or engine-based methods (left) aim to minimize the sim-to-real gap in pixel space, but incur high computational cost. In contrast, our approach (right) leverages 3D rasterization, which is scalable and fully controllable, and aligns rasterized inputs with real images in feature space.
In this work, we take different stance: robust E2E driving training does not require photorealistic rendering, but rather semantic accuracy and scalability. Driving decisions fundamentally depend on geometry, semanties, and multi-agent dynamics rather than visual details like textures or lighting. Moreover, humans readily transfer driving skills between video games and the real world, suggest- ing that aligning latent task-relevant features is more important than pixel-level appearance. These observations motivate favor lightweight rendering, combined with feature-space alignment, as a more scalable and transferable alternative to costly photorealistic approaches.
To this end, introduce 3D rasterization pipeline that reconstructs driving scenes by projecting annotated primitives (such as lane polylines and agent cuboids) into perspective views (Figure I right). Unlike neural/engine rendering, rasterization training-free, fast, and highly controllable, while still capturing the semantic and dynamic cues necessary for driving. This design enables us to go beyond the fixed dataset by generating non-trivial data augmentations, including: (1) Recovery- oriented perturbations, where ego trajectories are perturbed to simulate recovery maneuvers, di- rectly targeting II. brittleness; (2) Cross-agent view synthesis, where scenes are re-rendered from other agents' viewpoints, expanding both scale and interaction diversity. Moreover, in contrast to neural rendering, which seeks minimize the gap in pixel space, we propose a Raster-to-Real (R2R) alignment module that minimizes the gap in feature space, where semantic and geometric structures are more compact and easier to align. Together, these components form Rasterization Augmented Planning (RAP), scalable data augmentation framework for robust E2E planning.
We equip RAP with multiple plamners, showing that its benefits hold across diverse model designs. Extensive experiments show that RAP achieves strong closed-loop robustness and long-tail gen- eralization, ranking 1st place on the following 4 major E2E planning benchmarks: NAVSIM v1/v2 (Dauner et al.) 2024 Cao et al.] 2025). Waymo Open Dataset Vision-based E2E Driving (WOD-E2E) (Xu et al. (2025), and Bench2Drive (Jia et al.) (024). We find that efficient rasteri- zation with feature-space alignment provides the scalability and robustness needed for end-to-end planning, without requiring photorealistic rendering.
Our work makes the following contributions:
A scalable 3D rasterization pipeline that reconstructs driving scenes from annotations by project-
ing geometric primitives into camera views.
Kaster-to-Real (R2R) alignment module that bridges rasterized and real inputs in feature space, through combination of distillation and adversarial adaptation.
The 3D Rusterization Augmented end-to-end Planning (RAP) framework, which augments im- itation learning with counterfactual scene generation and cross-agent view synthesis, achieving state-of-the-at closed-loop robustness and long-tail generalization on multiple benchmarks.
2 RELATED WORK
End-to-end planning. E2E motion planners map sensory observations directly to future trajec-
tories or controls. Reinforcement learning approaches (Chekroun et al. 2023] Toromanoff et al.) 2020 Liang al. 2018) have been explored in simulators but remain bottlenecked by sample in- efficiency and the need for scalable environments. Imitation learning (IL) from real-world expert
driving logs (Pomerleau) 1988 Buhet al. 2020
Hu et al. 2022 Chitta et al. 2023 Hu et al.
2

---

Published as a conference paper at ICLR 2026
Feature Projector
Projected Features
Image Encoder
Pairect Inputss
eic... Annotated
Driving Logs
Cross-agent
Recovery-oreente Perturbation
Demain-Aligned
View Synthesis
Backbone
Rasterized- 0
Large-scale
Perception
Trajectory
Scoring
Pooling
Gradient
Reversal
Domain Classifeer +
Real-1
Synthetic Samples
Detached
Spatial-evel Alignment
Global-level Alignment
a. Data Augmentations via 3D Rasterization
b. Raster-to-Real Alignment
Figure 2: Overview of the proposed RAP. (a) Data Augmentations via 3D Rasterization: anno- tated driving logs converted into large-scale synthetic samples through cross-agent view synthe- sis and recovery-oriented perturbation. (b) Raster-to-Ree Real Alignment: paired real and rasterized inputs are processed by a frozen image encoder and learnable feature projector. Spatial-level alignment uses MSE loss against detached raster features, while global-level alignment employs gradient reversal layer and domain classifier enforce domain confusion.
2023 is more widely adopted and has achieved strong performance open-loop evaluations (Jaagg et al. 2023 Guo et al. 2025 Liao et al.) 2025). However, IL-trained policies suffer from covariate shift: they lack examples of recovery from mistakes and fail to generalize to rare long-tail events (Chen al. 2 To mitigate this, prior work has explored adversarial scene generation with challenging traffic behaviors (Bansal et al. 2019 Hanselmann et al. 2022 Rempe et al.] 2022 Yin jtt al. 2024), but these efforts restricted to bird's s-eye view and have only been validated in mid-to-end planning. In contrast, our work tackles the end-to-end setting from camera inputs, using scalable 3D rasterization generate diverse recovery and counterfactual scenarios.
Rendering for Driving. Classical simulators such as CARLA (Dosovitskiy et al., 2017), LGSVL (Rong al., 2020), and MetaDrive (Li et al. 2023b) rely physically based render- ing (PBR) with handcrafted 3D assets, but creating diverse, realistic worlds is costly, and model- ing traffic behavior remains challenging. MetaDrive (Li et al., 2023b) instead synthesizes digital scenes from real-world logs, while VISTA (Amini et al.) 2020; 2022) reprojects real images to nearby viewpoints, though only for small ego deviations. Neural rendering approaches such as NeRF (Mildenhall et al., 2020) and 3D Gaussian Splatting (3DGS) (Kerbl et al. (023) reconstruct logs with high fidelity and support counterfactual replay, but suffer from poor scalability, costly optimization, and visual artifacts when views deviate significantly. Voxel- and occupancy-basee re- constructions (Huang et al., 2023 Jiang al. 2024 Li et al.) 2025 2025| Chambon al., 2025) trade off fidelity for efficiency but require dense labeling (Huang al. (2024). Attempts at training planners with these renderings, e.g., RAD (Gao al.) 2025), remain small-scale and siatorraliim-fouuseee Closed-loop or seeudo-closed-lopy evaluations have been explored in NeuroNCAP (Ljungbergh al. 2024). HUGSIM (Zhou et al.] (024), RealEngine (Jiang et al. (2025), and NAVSIM v2 (Cao et al. 2025), but all rely on expensive photorealistic pipelines. In contrast, we advocate lightweight rasterization, which preserves semantic and geometric accuracy, avoids pixel-level artifacts, and enables scalable and mon-trivial data augmentation.
3 3D RASTERIZATION AUGMENTED PLANNING
RAP builds on real- driving logs by generating additional training data that goes beyond expert demonstrations. Instead of aiming for photorealistic images, goal is to capture the geometry, semantics, and dynamics that matter for driving. To this end, we design a lightweight rasteriza- tion pipeline that reconstructs controllable views of traffic scenes (Section 3.1). enabling fast and
3

---

Published as a conference paper at ICLR 2026
t=
t=1 t=2 t=3
Figure 3: Real rasterized views across 5 seconds. Top row: real front-cament inputs. Bottom row: corresponding rasterized views produced by our pipeline. Rasterization retains scene geometry and agent dynamics while discarding unnecessary appearance details.
large-scale data augmentation (Section 3.2), and we further introduce feature-alignmen module to ensure the network can effectively transfer from rasterized inputs to real images (Section 3.3)
3.1 3D RASTERIZATION
Our design prioritizes rendering speed and scala- bility, enabling the generation of large-scale aug- mented views. Instead of photorealism, we focus on preserving the geometric and semantic cues most relevant for driving (such agent positions, orientations, and interactions with the map) while discarding textures and lighting details that do not affect planning.
Real Image
3D Rasterization
Scene representation. At each log frame we reconstruct the scene from annotations. Static
map elements (e.g., road surfaces, crosswalks) represented as polylines M = (Py} in world coordinates, where each Ps R7 denotes a polyline with nę vertices in the (x, y, =) plane.
Figure 4: PCA visualization of frozen DI- NOv3 features. It shows rasterized and real inputs share similar structures, supporting ab- straction as a perceptually valid substitute.
Traffic-reevaatt objects (vehicles, bicycles, pedestrians, traffic cones, barriers, construction signs, and generic objects) are approximated by oriented cuboids
B, (li, wi, h, T.), = [4//2 tw:/2 0, ha]
where 4, w;, h4 are the length, width, and height actor i, T, E SE(3) is its rigid-body pose in world coordinates, and C; R8x3 gives the eight 3D corner points of the cuboid. Traffic lights modeled as upright cuboids with fixed dimensions, color-coded according to their state (red, yellow, green).
World-to-image projection. We model the ego camera as a standard pinhole camera with intrin- sics K RR3x and extrinsics T Tw-re E S3)... Any 3D point p.. R is lifted homogeneous coordinates P.. = [p 1JT E RA, and projected to the image plane as
Uge
π (Pw)
K T wo-e Pw:
After perspective division, the pixel coordinates are
(u, v) =
Min My
with depth uz. Points with uz Snear discarded.
4

---

Published as a conference paper at ICLR 2026
Rasterization. All primitives rasterized into an RGB canvas I E R xx33 using depth-aware compositing, where each fragment stores depth d with a fading weight a = max(0. blended through single buffer to resolve occlusions; primitives crossing the view boundary clipped with the Sutherland-Hodgman (Sutherland & Hodgman, 1974) operator to ensure stable visibility. Focusing geometric and semantic fidelity while discarding photorealistic details, rasterizer delivers spatial cues essential for planning at a fraction of the cost, enabling large-scale counterfactual generation. As shown in Figure 4 features extracted by a frozen DINOv3 encoder and visualized via PCA remain qualitatively consistent between rasterized and real images, indicat- ing that abstraction preserves perceptual cues crucial for downstream learning. Figure 3 shows qualitative comparison between real and rasterized views across 5 horizon.
d e/dmxx)
3.2 DATA AUGMENTATIONS VIA 3D RASTERIZATION
Beyond rendering ego-centric views, our rasterization pipeline naturally supports diverse non-trivial augmentation strategies, allowing us expand the training corpus and expose the planner to richer and rarer distributions of driving scenarios. We focus on two complementary techniques (Figure 2 left): Recovery-oriented perfurbations simulate recovery from off-distribution states, and cross- agent view synthesis to diversify viewpoints and interactions and thereby extract more value from an annotated log.
Recovery-orientee perturbations. We enrich the dataset by perturbing the logged ego trajectory. Given a ground-truth trajectory - r²()). we apply controlled lateral and longitudinal offsets together
with stochastic noise:
(t) = (t) + dtn(() + ōtong (t) + Ex.
where Stat. ôjong are perturbations sampled from predefined ranges and e, denotes Gaussian noise. The perturbed trajectory is then re-rendered with 3D rasterization, creating counterfactual scenes in which the ego vehicle drifts away from the expert path. These samples encourage the planner to recover from distribution shifts, improving robustness in closed-loop evaluation.
Cross-agent view synthesis. Each nuPlan traffic scenario contains trajectories for n agents (in- cluding the ego). Instead of rendering only from the ego perspective, replace the ego trajectory with that of another agent while keeping the original camera parameters fixed. This produces realis- tic views from other agents without requiring new sensors.
Together, these augmentations scale beyond the limitations of logged data, yielding over 500k raster- ized training samples that cover diverse viewpoints, richer interactions, and rare recovery scenarios.
3.3 RASTER-TO-REEE ALIGNMENT
Synthetic rasters already produce features similar to real images Figuuee 4). suggesting they can strengthen learning when combined. To make this benefit reliable, we introduce Raster-to-Real (R2R) alignment (Figure 2] right), enforcing feature consistency between rasterized and real inputs both spatial and global levels.
Spatial-level alignment. For each real sample x" with paired rasterized rendering xº, visual encoder Φ(-) to extract projected features:
F" ==="(xx),
F F==
F Nxd"
where N denotes the number of spatial locations (patch tokens for ViT or feature-map positions for CNNs), and d' the projected feature dimension. We freeze the raster features F and update the real features F" by minimizing a mean-squared alignment loss:
N P
1
Eapuatill
F
N
Since raster features come from high-quality annotations and omit distracting details, they as a strong proxy for perception. With their abundance compared to real images, aligning real features to them offers cleaner and denser supervision.
5

---

Published as a conference paper at ICLR 2026
Table 1: NAVSIM v1 benchmark (navtest). Bold/underlined indicates the best/second-
Method
Input
NC
DAC 1
TTC1
Comf.
EPT
PDMS
IDM-Closed (Rule-based) Dauner et A 2023
Perception
94.6
99.8
86.9
99.9
89.9
89.1
Human
100
100
100
99.9
87.5
94.8
Midra-MDPP-appp W-EP (P (Li 2024
Cam Lidar
98.3
96.0
94.6
100
78.7
86.5
DiffusionDriive (Liao et al. 2025
Cam & Lidar
98.2
96.2
94.7
100
82.2
88.1
DiffusionDrive- Canera Only (Koproduced)
Camera
97.9
94.6
93.6
80.7
86.0
PARA-Drive (Wenig
2024
Camera
97.9
92.4
93.0
99.8
79.3
84.0
Aut (Zhou et a. 2025
Camera
98.4
95.6
98.0
99,94
81.9
89.1
iPad (uoo al 2022
Camera
98.6
98.3
94,9
100
88.0
91.7
Centaur (Sma cra
2025
Camera
99.2
98.7
99.97
86.0
MAP-DiffusionDDiiv--Cneee Only
Camera
98.5
98.5
95.4
100
83
89.2
RAP-iPud
Camera
98.2
98.6
94.6
100
90.1
92.5
RAP-DINO (Ours)
Camera
99.1
98.9
96.7
100
90.3
93.8
Table 2: Public leaderboard for the NAVSIM v2 benchmark (navhard). Bold indicates the best result.
Method
Stage
NC
DAC
DDC
TLC1
EP
TTC1
LK 1
HC 1
EC f
EPDMS
LTF (Chitta ul.
96.22
79.56
99.11
99.56
84.12
95.11
94.22
97.56
79.11
23.12
2023
2
77.76
70.21
84.29
98.07
85.
75.67
45.41
95.75
75.98
23.12
1
97.11
94.44
98.78
99.78
83.86
96.89
94.67
96.44
66.22
36.93
RAP-DINO (Ours)
2
83.20
83.88
87.39
98.02
86,87
80.39
52.27
95.18
52.41
36.93
Global alignment. Global alignment complements spatial-level supervision by keeping the over- all feature distributions consistent. For instance, rasterized views may contain pure black back- grounds absent real data; aligning globally mitigates such biases and improves generalization. Besides, since rasterized samples vastly outnumber real-raster pairs, unsupervised domain adapta- tion (Ganin & Lempitsky 2015) provides an effective way to exploit the full synthetic corpus by enforcing global consistency even without paired supervision.
For each sample, we compute global representation g E Rd by average pooling its feature map F E Nxd' We then train domain classifier D to predict whether 9 comes from real or rasterized data. Following Ganin & Lempitsky (205)), gradient reversal layer is inserted before D, such that the encoder is optimized to maximize domain confusion while D) is optimized to minimize classification error with E {0, 1} the domain label:
Cglobal
(g))) [y log D(g) + +(1 y) log(1 - D(g)))
(3)
Overall objective. The final training objective combines task supervision with both levels of R2R alignment with Lusk the total planning loss (Guo et al. (0255: L = Lrask Xs Lopatial >, Leghhhal. where Xa Ag control the strength of spatial-level and global alignment.
4 EXPERIMENTS
We build our rasterized data from OpenScene, a compact distribution of the nuPlan dataset (Caesar jet al., (02T) that contains over 1,200 hours of annotated driving logs. Among these, about 120 hours provide ego-centric real camera sensors. We rasterize both the ego vehicle and other agents' trajectories across all 1,200 hours of logs. To diversify recovery behaviors, we additionally perturb ego trajectories in randomly selected 10% subset of ego logs.
Dataset Curation. We extract 7-second clips, using the first 2 seconds as input and the following seconds as output. For ego trajectories, we follow the NAVSIM (Dauner al. 2024)'s Planning- aware Driving Metric Score (PDMS) filtering strategy, removing trivial cases where the constant- velocity baseline already has high scores and human demonstrations have low scores. For other vehicles, since nuPlan lacks route annotations and PDMS cannot be computed, we instead filter clips by ADE of the constant-veloctt baseline (ADE 0.5) and ensure validity. After filtering, curated dataset consists of 85k samples with paired real and rasterized sensors, 8.5k rasterized

---

Published as a conference paper at ICLR 2026
Table 3: Top-6 entries on the public leaderboard for the WOD-E2E Driving Challenge (up to September 2025).
Method
ADE@Ss
ADE@3s
RFS (Spotlight)
RFS (Overall)
DiflusionLII
2.89
1.36
6,41
7.72
HMVLM
3.07
1.33
6.73
7.85
UniPlan
2.84
1.27
6,92
7.78
ViT-Adapte--GRU
2.89
1.44
6.67
7.99
Poutine (Rowe al. 2025
2.74
1.21
6.89
7.99
RAP-DINO (ours)
2.65
1.17
7.20
8.04
samples with perturbations, 272k/200k: rasterized samples from ego trajectories and other vehicles' trajectories.
Models. We instantiate our RAP framework with model we call RAP-DINO. which combines a frozen DINOv3-H+ backbone (Siméoni et al. 2025) with a learnable MLP projector and an iter- ative deformable attention decoder adapted from (Guo et al. 2025). The full model size is ~888M parameters. For ablation and efficient closed-loop inference on Bench2Drive, we also introduce a lightweight RAP-ResNet variant with only ~29M parameters. In addition, to demonstrate the model-agnostic nature of RAP, we apply our framework to existing architectures, yielding RAP- iPad (Guo et al. 2025) and RAP-DiffusionDrive (Liao et al. 2025).
Training. We train on the curated dataset with two supervision heads: a multi-modal trajectory head supervised by future trajectories, and trajectory scoring head trained with PDMS scores. Training uses 4 H100 GPUs and takes about 80 hours. The trained model is directly evaluated on NAVSIM 1/v2 benchmarks. For Waymo, we fine-tune on the train/val split using the official RFS scorer. For Bench2Drive, we follow the official bench2drive-ben our 85k real-raster pairs and 8.5k perturbed samples. this case, we use smaller ResNet34 (He et al. 2016) image backbone (RAP-ResNet) for faster closed-loop inference. See (Sectinn A.4 for more details.
dataset and augment it with
4.1 LEADERBOARD EVALUATION
NAVSIM v1/v2. The NAVSIM benchmarks (Dauner et al. 2024 Cao et al. 2025) evaluate plan- ner's ability to generate safe and efficient 4-second future trajectory (sampled at 2Hz) based on seconds of historical ego states and multi-view camera inputs. The benchmark is derived from the NuPlan dataset, specifically curating challenging scenarios with intention changes while filtering out trivial straight-driving segments. NAVSIM v1 employs the PDMS, composite score aggre- gating sub-metrics such as No at-fault Collision (NC), Drivable Area Compliance (DAC), Time-to- Collision (TTC), Ego Progress (EP), and Comfort (Comf.). To better assess closed- robustness, NAVSIM v2 introduces the Two-stage Extended PDMS (EPDMS), which incorporates additional criteria: Traffic Light Compliance (TLC), Driving Direction Compliance (DDC), Lane Keeping (LK), and Extended Comfort (EC). In its second stage, NAVSIM v2 further employs 3D Gaussian Splat- ting (3DGS) synthesize counterfactual camera views after policy deviations, thereby simulating closed-loop evaluation from logged data. We report results on the navhard split.
Table shows the NAVSIM v1 benchmark, where RAP-DINO achieves the highest overall PDMS score of 93.8, surpassing all prior camera-based methods. Beyond our strongest model, applying RAP to existing architectures also brings consistent gains: RAP-DiffusionDrive improves PDMS by +3.2 over the original DiffusionDrive, while RAP-iPad yields +0.7 improvement over iPad. These results confirm that RAP is both effective its own right and broadly beneficial when applied to other state-of of-the-art planners. RAP-DINO also sets a new state-of- tftt-oof-the-an art on NAVSIM v2 (Table 2), achieving an overall EPDMS of 36.9, substantially higher than LTF, while maintaining strong performance across both Stage and the more challenging Stage 2 counterfactual evaluation.
WOD Vision-based E2E Driving. Given 4s of multi-camera inputs plus ego history and route, the task is to predict a 5 future trajectory, evaluated and ranked by the Rater Feedback Score. The dataset includes 4021 segments (2037 train, 479 val, and the rest test) curated for long-tail events

---

Published as a conference paper at ICLR 2026
Table 4: Closed-loop Results on Bench2Drive Benchmark (Jia al. 2024).
Method
Efficiency
Comfortness f
Success Rate (%c
Driving Score
AD-MLP Zhai al. 2023
48.45
22.63
0.00
18.05
UniAD-Tiny (uu 2023
123.92
47.04
13.18
40.73
UniAD-Base al.
129.21
43.58
16.36
45.81
VAD Jiang 2025
157.94
46.01
15.00
42.35
Drive Transforee al. 2025
100.64
20.78
35.01
63.46
iPad (Guo 2025
161.31
28.21
35.91
65.02
RAP-ResNct (Ours)
165.47
23.63
37.27
66.42
Table 5: Ablation study on 3D rasteriza-
Table 6: Ablation on recovery-oriented perturbations. Evaluation is conducted NAVSIM v1 and v2 (Section 4.2)
tion design choices (Section 4.2
ID
Face Rendering
Depth
Decay
Background
MinADE
NAVSIM v1 PDMS
NAVSIM v2
Training Set
Colored
Yes
Black
0.91
two-stage EPDMS
B Transparent
Yes
Black
0.98
Colored
No
Black
1.05
W/ Perturbed
92.5
32.5
Colored
Yes
Natural
1.33
Perturbed
92.5
36.9
such as construction detours, pedestrian accidents, and unexpected freeway obstacles. These cases occur with a frequency below 0.003% in daily driving, underscoring the dataset's focus on rare but safety-critical scenarios.
Table shows the results. Our RAP-DINO achieves the best overall results, with RFS (Over- all) of 8.04 and the lowest 5s (2.65) and ADE@ 3s (1.17). Notably, RAP surpasses Poutine (Rowe et al. 2025), strong B--ccale rision-languwe model, while also attaining the highest RFS (Spotlight) of 7.20, significantly outperforming all competing entries. These results highlight RAP's strong generalization and robustness in long-tail scenarios.
Bench2Drive (Jia et al. 2024). We use the CARLA (Dosovitskiy et al. 2017) simulator, employ- ing the Bench2Drive benchmarks for closed-loop performance evaluation of RAP. Bench2Drive provides 1,000 clips from the expert model Trink2Drive, with 950 for training and 50 for open- loop evaluation. Closed-loop evaluation covers 220 routes across CARLA towns, each containing safety-critica event, and reports four metries: Success Rate, Driving Score, Efficiency, and Com- fortness. As shown in Table 4 our RAP-ResNet achieves the best overall performance, attaining the highest Efficiency (165.47) and Driving Score (66.42), well the highest Success Rate (37.27% These results highlight that RAP significantly improves route completion and robustness in closed- loop evaluation, establishing new state-of-the-art on Bench2Drive.
4.2 ABLATION STUDY
Unless otherwise specified, all ablations are conducted on the navtrain subset NAVSIM. The training set consists of 85k paired samples with both real and rasterized views, along with ad- ditional 100k raster-only samples generated from ego trajectories. Evaluation is performed on the validation split (12k samples) using Minimum Average Displacement Error (MinADE) as the metric. We use Res-ResNet for efficient training and accelerated experimentation.
3D Rasterization Design Choices. We ablate three factors the rasterization pipeline: whether the object cuboids use solid colors or semi-transparemt faces, whether to apply depth-based inten- sity decay, and whether the background is pure black or natural sky-ground split. See appendix Section A.2 for visualizations.
Results Table 5 show that solid-colored faces, depth decay, and a black background together achieve the best ADE. Removing any of these components leads to noticeable degradation, firming that semantic cues, depth-aware rendering, and minimal background distractions are all important for effective learning.
8

---

Published as a conference paper at ICLR 2026
1.2
MinADE
Spatial & Global
Spatial Only
1.07
w/o aligmmtt
1.03
1.1
1.0
1%
0.97
1.0
5%
20%
50%
100%
0.021
1.2173
0.94
0.93
R-
0.9942
Samplex
1E+04
1E+05
1E+06
10%
100%
Figure 6: Scaling curve for cross-agent view synthesis (Section 4.2), showing con- sistent gains more synthetic samples are added.
Figure 5: Ablation on R2R alignment Section 4.2), showing that both spatial and global alignment improve performance.
Ablation on Recovery-oriented Perturbations. We further examine the effect of recovery- oriented perturbation, which exposes the planner to counterfactual recoveries during training. Two training sets compared: the navtrain split with 85k real samples, and the same split aug- mented with 8.5k perturbed ego trajectories. Since this modification is designed to improve closed- loop robustness, perturbation has no effect on vl (both 92.5), but significantly improves v2 from 32.5 to 36.9. This expected, as NAVSIM v2 adopts two-stage evaluation protocol that better reflects closed-loop performance.
evaluate NAVSIM v1 and v2 using the PDM score. Table shows that
Ablation on R2R Alignment. We evaluate R2R alignment the navtrain subset NAVSIM, which provides 85k real-raster paired samples. For training, we vary the fraction of real data kept (f1%, 5%, 20%, 30%, 100%)) while replacing the remainder with rasterized samples, keeping the total size fixed. All models are evaluated on the validation split (12k real samples) using MinADE. Fig. compares three settings: no alignment, spatial alignment, and spatial+global alignment. Both alignment strategies improve performance over the baseline, and combining spatial and global align- ment yields the strongest results across all replacement ratios. This confirms that aligning synthetic and real features bottures-both locally and globally-effeective reduces the domain gap and enables better transfer of supervision. Moreover, synthetic augmentation itself is beneficial: with 50% synthetic data, performance surpasses training 100% real samples. Overall, (i) R2R alignment system- atically improves domain transfer, and (ii) large-scale rasterized data serve not only as a substitute but also powerful augmentation to limited real-world data.
Ablation on Cross-Agent View Synthesis. We next analyze the effect of scaling cross-agent syn- thesis, where trajectories of non-ego vehicles are rasterized to produce additional training samples. Starting from the navtrain in split with 85k real samples, we progressively add {1k, 10k, 100k, 500k, 1000k} synthetic samples of other vehicles while keeping all other factors fixed. Figure exhibits clear log-scaling trend: the relation between sample count x and MinADE is well fitted by = -0.021 In(x) + 1.2173, R² 0.9942, demonstrating diminishing but consistent gains more synthetic samples are added. This log-scaling behavior closely mirrors findings from prior studies on data scaling in end-to-end driving (Baniodch et al.) 2025 Zheng et al. 2024). Notably, this finding is significant because the added data consists of rasterized samples derived from other agents' trajectories, showing that even secondary viewpoints contribute to scaling laws and improve planner robustness.
CONCLUSION
We presented RAP, scalable framework for end-to-end driving that leverages lightweight 3D rasterization and feature-space alignment as an alternative to costly photorealistic rendering. By enabling recovery-oriented perturbations and cross-agent synthesis, RAP scales training large- scale counterfactual scenes while preserving semantic and geometric fidelity. Extensive experi- ments across four benchmarks demonstrate that RAP consistently improves closed-loop robustness

---

Published as a conference paper at ICLR 2026
and long-tail generalization, establishing a practical and effective recipe for scaling end-to-end tonomous driving. Limitations & Future Work: a key limitation our approach is that remains within the imitation learning paradigm, which inherits issues such as causal confusion. In future work, we aim to extend 3D rasterization into a full simulator to support closed-loop reinforcement learning, enabling richer interaction and policy improvement beyond offline demonstrations.
ACKNOWLEDGMENTS
We would like to thank Ellington Kirby and Alexandre Boulch for their valuable help. The project was partially funded by Valeo, SwissAI, Sportradar, the Swedish Research Council (Veten- skapsrådet) under award 2023-00493, and the NAISS under award 2025/22-1173.
ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. Our research focuses on developing scalable training methods for end-to-end autonomous driving planners using synthetic rasterized data. No human subjects were involved, and all datasets used (nuPlan, Waymo, NAVSIM, and Bench2Drive) are publicly released under their respective licenses. We acknowledge that autonomous driving research raises safety and fairness concerns. By emphasizing reproducibility, transparent methodology. and evaluation on diverse benchmarks, we aim to mitigate risks of overclaiming model capabiliiiess The proposed method does not introduce new privacy, security, discrinination risks beyond those already present in the underlying datasets.
REPRODUCIBILITY STATEMENT
We have made efforts to ensure that our work is reproducible. The paper provides detailed descrip- tions of the model architecture (Section 3), training setup (Section 4), and ablation studies (Sec-) [ion 4.2). Additional hyperparameters and implementation details are provided in the Appendix. All datasets used are publicly available (nuPlan, WOD, Bench2Drive), and we describe the curation and filtering process in (Section 4] Source code and configuration files will be submitted and released with the camera-ready version to facilitate full reproducibility. These instructions apply to everyone, regardless of the formatter being used.
REFERENCES
Alexander Amini, Igor Gilitschenski, Jacob Phillips, Julia Moseyko, Rohan Banerjee, Sertac Kara- man, and Daniela Rus. Learning robust control policies for end-to-end autonomous driving from data-driven simulation. RAL, 2020.
Alexander Amini, Tsun-Hsuan Wang, Igor Gilitschenski, Wilko Schwarting. Zhijian Liu, Song Han, Sertac Karaman, and Daniela Rus. Vista 2.0: An open, data-driven simulator for multimodal sensing and policy learning for autonomous vehicles. 2022.
Mustafa Baniodeh, Kratarth Goel, Scott Ettinger, Carlos Fuertes, Ari Seff, Tim Shen, Cole Gulino, Chenjie Yang, Ghassen Jerfel, Dokook Choe, Rui Wang, Benjamin Charrow, Vinutha Kallem, Sergio Casas, Rami A1-Rfou, Benjamin Sapp, and Dragomir Anguelov. Scaling laws of motion forecasting and planning - technical report, 2025.
Mayank Bansal, Alex Krizhevsky, and Abhijit S. Ogale. Chauffeurnet: Learning to drive by imi- tating the best and synthesizing the worst. In Robotics: Science and Systems XV, University of Freiburg, Freiburg im Breisgau, Germany, June 22-26, 2019, 2019.
Florent Bartoccioni, Elias Ramzi, Victor Besnier, Shashanka Venkataramanan, Tuan-Hung Vu, Yihong Xu, Loick Chambon, Spyros Gidaris, Serkan Odabas, David Hurych, Renaud Marlet, Alexandre Boulch, Mickaël Chen, Éloi Zablocki, Andrei Bursuc, Eduardo Valle, and Matthieu Cord. Vavim and vavam: Autonomous driving through video generative modeling. CoRR, 15672, 2025.
10

---

Published as a conference paper at ICLR 2026
Thibault Buhet, Émilie Wirbel, Andrei Bursuc, and Xavier Perrotton. PLOP: probabilistic polyno- mial objects trajectory prediction for autonomous driving. In CoRL, 2020.
Holger Caesar, Juraj Kabzan, Kok Seang Tan, Whye Kit Fong, Eric M. Wolff, Alex H. Lang, Luke Fletcher, Oscar Beijbom, and Sammy Omari. nuplan: A closed-loop ml-based planning bench- mark for autonomous vehicles. CoRR, abs/2106. 11800, 2021.
Wei Cao, Marcel Hallgarten, Tianyu Li, Daniel Dauner, Xunjiang Gu, Caojun Wang, Yakov Miron, Marco Aiello, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, Andreas Geiger, and Kashyap Chitta. Pseudo-simulation for autonomous driving. In CoRL, 2025.
Loick Chambon, Eloi Zablocki, Alexandre Boulch, Mickaël Chen, and Matthieu Cord. Gaussrender: Learning 3d occupancy with gaussian rendering. In ICCV, 2025.
Raphael Chekroun, Marin Toromanoff, Sascha Hornauer, and Fabien Moutarde. Gri: General rein- forced imitation and its application to iision-based autonomous driving. Roboties, 2023.
Li Chen, Penghao Wu, Kashyap Chitta, Bernhard Jaeger, Andreas Geiger, and Hongyang Li. End- to-end autonomous driving: Challenges and frontiers. IEEE Trans. Pattern Anal. Mach. Intell., 46(12):10164-10183, 2024.
Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. Transfuser: Imitation with transformer-based sensor fusion for autonomous driving. PAMI, 2023.
Daniel Dauner, Marcel Hallgarten, Andreas Geiger, and Kashyap Chitta. Parting with miscon- ceptions about learning-bas based vehicle motion planning. In Conference Robot Learning, pp-
1268-1281. PMLR, 2023.
Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng. Zhiyu Huang. Zetong Yang. Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, Andreas Geiger, and Kashyap Chitta. NAVSIM: data-driven non-reat non-reactive autonomous vehicle simulation and benchmarking. In NeurlPS, 2024.
Alexey Dosovitskiy, Germán Ros, Felipe Codevilla, Antonio M. López, and Vladlen Koltun. CARLA: an open urban driving simulator. 2017.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference machine learning, pp. 1180-1189, PMLR, 2015.
Hao Gao, Shaoyu Chen, Bo Jiang, Bencheng Liao, Yiang Shi, Xiaoyang Guo, Yuechuan Pu, Haoran Yin, Xiangyu Li, Xinbang Zhang, al. Rad: Training an end-to-end driving policy via large-scale 3dgs-based reinforcement learning. arXiv preprint arXiv:2502.131: 13144 2025.
Ke Guo, Haochen Liu, Xiaojun Wu, Jia Pan, and Chen Lv. ipad: Iterative proposal-centric end-to- end autonomous driving. arXiv preprint arXiv:2505. 15/11, 2025.
Niklas Hanselmann, Katrin Renz, Kashyap Chitta, Apratim Bhattacharyya, and Andreas Geiger. KING: generating safety-critical driving scenarios for robust imitation via kinematics gradients. In ECCV, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Anthony Hu, Gianluca Corrado, Nicolas Griffiths, Zak Murez, Corina Gurau, Hudson Yeo, Alex Kendall, Roberto Cipolla, and Jamie Shotton. Model-based imitation learning for urban driving. In NewrIPS, 2022.
Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tian- wei Lin, Wenhai Wang, Lewei Lu, Xiaosong Jia, Qiang Liu, Jifeng Dai, Yu Qiao, and Hongyang Li. Planning-oriente autonomous driving. In CVPR, 2023.
Yuanhui Huang, Wenzhao Zheng, Yumpeng Zhang, Jie Zhou, and Jiwen Lu. Tri-perspective view for vision-based 3d semantic occupancy prediction. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9223-9232, 2023.

---

Published as a conference paper at ICLR 2026
Yuanhui Huang, Wenzhao Zheng. Borui Zhang, Jie Zhou, and Jiwen Lu. Selfoce: vision-basee based 3d occupancy prediction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19946-19956, 2024.
Self-supervised
Xiaosong Jia, Zhenjie Yang, Qifeng Li, Zhiyuan Zhang, and Junchi Yan. Bench2drive: Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving. In NeurIPS, 2024.
Xiaosong Jía, Junqi You, Zhiyuan Zhang, and Junchi Yan. Drivetransformer: Unified transformer for scalable end-to-end autonomous driving. arXiv preprint arXiv:2503.07655 2025.
Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vectorized scene representation for efficient au- tonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pp. 8340-8350, 2023.
Haoyi Jiang, Tianheng Cheng, Naiyu Gao, Haoyang Zhang, Tianwei Lin, Wenyu Liu, and Xing- gang Wang. Symphonize 3d semantic scene completion with contextual instance queries. Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 20258- 20267, 2024.
Junzhe Jiang, Nan Song, Jingyu Li, Xiatian Zhu, and Li Zhang. Realengine: Simulating autonomous driving in realistic context. CoRR, abs/2505.16902, 2025. doi: 10.48550/ARXIV.2500. 16902.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139:11139: 2023.
Quanyi Li, Zhenghao Peng, Lan Feng, Zhizheng Liu, Chenda Duan, Wenjie Mo, and Bolei Zhou. Scenarionet: Open-source platform for large-scale traffic scenario simulation and modeling, 2023a.
Quanyi Li, Zhenghao Peng, Lan Feng, Qihang Zhang, Zhenghai Xue, and Bolei Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE Trans. Put- tern Anal. Mach. Intell., (5(3))3461-34755 2023b. doi: 10.1 10.110//PAMM2222.119977777
Wuyang Li, Zhu Yu, and Alexandre Alahi. Voxdet: Rethinking 3d semantic occupancy prediction as dense object detection. arXiv preprint arXiv:2506.04622 2025.
Zhenxin Li, Kailin Li, Shihao Wang, Shiyi Lan, Zhiding Yu, Yishen Ji, Zhiqi Li, Ziyue Zhu, Jan Kautz, Zuxuan Wu, et al. Hydra-mdp: End-to-end multimodal planning with multi-target hydra- distillation. arXiv preprint 24Ki::2006006778 2024.
Xiaodan Liang, Tairui Wang, Luona Yang, and Eric Xing. Cirl: Controllable imitative reinforcement learning for vision-bases self-driving. In ECCV, 2018.
Bencheng Liao, Shaoyu Chen, Haoran Yin, Bo Jiang, Cheng Wang, Sixu Yan, Xinbang Zhang, Xiangyu Li, Ying Zhang, Qian Zhang, et al. Diffusiondrive: Truncated diffusion model for end- to-end autonomous driving. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 12037-12047, 2025.
William Ljungbergh, Adam Tonderski, Joakim Johnander, Holger Caesar, Kalle Aström, Michael Felsberg, and Christoffer Petersson. Neuroncap: Photorealistic closed-loop safety testing for autonomous driving. In ECCV, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 10110:17 2017.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020,
Dean Pomerleau. ALVINN: an autonomous land vehicle neural network. In NIPS, 1988.
Davis Rempe, Jonah Philion, Leonidas Guibas, Sanja Fidler, and Or Litany. Generating useful accident- driving scenarios via learned traffic prior. In CVPR, 2022.
12

---

Published as a conference paper at ICLR 2026
Guodong Rong. Byung Hyun Shin, Hadi Tabatabuee, Qiang Lu, Steve Lemke, Märtins Možeiko, Eric Boise, Geehoon Uhm, Mark Gerow, Shalin Mehta, et al. Lgsvl simulator: A high fidelity simulator for autonomous driving, arXiv preprint arXhv:2005.03778 2020.
Stephane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning, 2011.
Luke Rowe, Rodrigue de Schaetzen, Roger Girgis, Christopher Pal, and Liam Paull. Poutine: aaaar-laanaagg-rraecooooo pre-training and reinforcement learning post-training enable robust end-to-end autonomous driving, arXiv preprint arXiv:2506. 1234, 2025.
Chonghao Sima, Kashyap Chitta, Zhiding Yu, Shiyi Lan, Ping Luo, Andreas Geiger, Hongyang Li, and Jose Alvarez. Centaur: Robust end-to-end autonomous driving with test-time training- arXiv preprint arXiv:2503. 1650, 2025.
Oriane Siméoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Mare Szafraniec, Seung Eun Yi, Michaël Ramamonjisoa, Francisco Massa, Daniel Haziza, Luca Wehrstedt, Jianyuan Wang, Timothée Darcet, Théo Moutakanni, Leonel Sentana, Claire Roberts, Andrea Vedaldi, Jamie Tolan, John Brandt, Camille Couprie, Julien Mairal, Hervé Jégou, Patrick Labatut, and Piotr Bojanowski. Dinov3. CoRR, 2iss/250..10044 2025.
Ivan E Sutherland and Gary W Hodgman. Reentrant polygon clipping. Communications of the ACM, 17(1):32-42, 1974.
Marin Toromanoff, Emilie Wirbel, and Fabien Moutarde. End-to-end model-free reinforcement learning for urban driving using implicit affordances. In CVPR, 2020.
Xinshuo Weng, Boris Ivanovic, Yan Wang, Yue Wang, and Marco Pavone. Para-drive: Parallelized architecture for real-time autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15449-15458, 2024.
Runsheng Xu, Hubert Lin, Wonseok Jeon, Hao Feng, Yuliang Zou, Liting Sun, John Gorman, Kate Tolstaya, Sarah Tang, Brandyn White, Sapp Ben, Mingxing Tan, Jyh-Jing Hwang, and Dragomir Anguelov. Wod-e2e: Waymo open dataset for end-to-end driving in challenging long-tail scenar- ios. arXiv preprint 1rXi::2510.26125, 2025.
Yuan Yin, Pegah Khayatan, Éloi Zablocki, Alexandre Boulch, and Matthieu Cord. Regents: Real- world safety-critical driving scenario generation made stable. In Alessio Del Bue, Cristian Can- ton, Jordi Pont-Tuset, and Tatiana Tommasi (eds.), Computer Vision ECCV 2024 Workshops Milan, Italy, September 29-October 4, 2024, Proceedings, Part VII, volume 15629 of Lecture Notes in Computer Science, pp. 262-276. Springer, 2024. doi: 10.1007/998833-01--17777 1007/ 978007778-00 3- 031 91767-77777 7
URL https::/doi..oo.
Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong. Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. arXiv preprint arXiv:2410.0694/ 2024.
Jiang-Tian Zhai, Ze Feng, Jinhao Du, Yongqiang Mao, Jiang-Jiang Liu, Zichang Tan, Yifu Zhang, Xiaoqing Ye, and Jingdong Wang. Rethinking the open-loop evaluation of end-to-end autonomous driving in nuscenes. arXiv preprint vrXiv:2305.104:3 2023.
Yupeng Zheng, Zhongpu Xia, Qichao Zhang, Teng Zhang, Ben Lu, Xiaochuang Huo, Chao Han, Yixian Li, Mengjie Yu, Bu Jin, et al. Preliminary investigation into data scaling laws for imitation 2024.
learning-based end-to-end autonomous driving. arXiv preprinl
Hongyu Zhou, Longzhong Lin, Jiabao Wang, Yichong Lu, Dongfeng Bai, Bingbing Liu, Yue Wang, Andreas Geiger, and Yiyi Liao. HUGSIM: A real-time, photo-realiste and closed-loop simulator for autonomous driving. CORR, abs/2412.01718, 2024. doi: 10.48550/AR.122442..1777
Zewei Zhou, Tianhui Cai, Seth Zhao, Yun Zhang, Zhiyu Huang, Bolei Zhou, and Jiaqi Ma. Au- tovla: A languagee action model for end-to-end autonomous driving with adaptive reason- ing and reinforcement fine-tuning, arXiv preprint arXiv:2506.13755 2025.
13

---

Published as a conference paper at ICLR 2026
A
APPENDIX
A.1 DISCUSSIONS
Does Simplified 3D Rasterization Risk Overlook Critical annotated geometric elements of rasterization pipeline simplifies appearance by rendering only the the scene, such as lanes, vehicles, and traffic signals. This choice reflects deliherate trade-off. Rasterization gives us the speed and controllability needed to generate large-scale, diverse training data. High-fidelity rendering methods cannot reach this scale. At the same time, rasterization does not aim to replace real images. It complements them.
Visual Cues? Our 3D
Despite this simplification, we find that the model retains strong perception and planning ability when applied to real-world images. Qualitative results on nuPlan (Figure 7 show that RAP correctly reacts to visual elements that are not present in the rasterization ontology. such as unannotated "Keep Left" signs or LED displays on a truck. Additional long-tail cases from WOD-E2E (Figure 8) further confirm that the model can handle subtle and unexpected signals inference time.
These results indicate that simplified 3D rasterization does not hinder the model's ability to reason about fine- mne-grained or OOD cues. Instead, it provides a strong geometric foundation that supports robust, scalable learning. Real images, seen both during training and always at inference, supply the remaining richness that rasterization cannot encode. The strong performance of RAP on the corner-case-oocuse WOD-E2E benchmark (Table 3) reinforces this conclusion.
Does Real-to-Sim Feature Alignment Cause Information Loss? A natural concern that align- ing real-world features to simplified rasterized features may suppress unannotated or subtle visual cues. Our results show that this does not occur, largely due to the multi-task learning design of our planner. The Real-to-Raste alignment provides stable geometric prior, while the planning and per- ception objectives applied on real features compel the model to retain all task-relevant information necessary for accurate decision-make making in complex traffic scenes.
To further understand this effect, we compare three alignment strategies: Raster-to-Real, symmetric alignment, and our Real-to-Raster variant. Follow- ing the same ablation setup in [Section 4.2 we fix the real-data ratio to 50%. The results in Table 7 show that Real-to-Rand Raster yields the lowest ADE, outper- forming both alternatives. This supports our hypoth- esis that raster features serve as clean structural scaffold, while the planning loss ensures that seman- tic richness from the real image preserved. These findings align with recent representation alignment literature Yu et al. 2024), where aligning to a well-structured feature space improves semantic abstraction without discarding perceptual detail. Future work may explore adaptive task-aware alignment schemes to further reduce potential in- formation loss.
Table 7: Comparison of alignment directions under 50% real data. Lower ADE is better.
Alignment Variant
MinADE.
Raster-to-Real Symmetric Alignment
1.12
1.14
Real-to-Raster (ours) 1.02
WOD Vision-based E2E Driving Finetuning. For finetuning on the WOD-E2E Driving bench- mark, we use a learning rate of x 10-3 and unfreeze the visual encoder. Finetuning is performed in two stages: first on the official training split, and then the validation split with batch size of 16. For the final test submission, we apply non-maximum suppression (NMS) ensembling over two checkpoints to improve robustness.
Bench2Drive Mixed- Sraedd-raining and Dataset Alignment. To enable mixed training with nuPlan and Bench2Drive, we align their data formats before batching. Specifically, reorder camera views to a unified sequence, resize images to a common 576 x 1024 resolution, and adjust camera calibration matrices with rotation and scaling match nuPlan's 's convention. Ego kinematics and target trajectories are normalized to the same coordinate frame. This alignment ensures that samples from both datasets share consistent geometry and calibration, allowing stable joint optimization.
14

---

Published as a conference paper at ICLR 2026
Real Camera
Rasterized Camera
Scenario A
Scenario B
Figure 7: RAP-DINO retains fidelity to fine-grained real-world cues. This figure ablates our single, fully-trained model by comparing predictions (red trajectory) against the ground truth (green trajectory) when conditioned two different inputs: the raw real-world image (Top) vs. the simplified rasterized image (Bottom). (Left) Scenario A: An unannotated "Keep Left" sign, absent from the raster data, is correctly perceived from the raw image (Top), leading to correct trajectory. The cnster-conditione model (Bottom) fails. (Right) Scenario B: challenging OOD dynamic LED arrow, also absent from the raster, is successfully identified from the raw image (Top), resulting in safe lane change.
Figure 8: Qualitative Results the WOD Vision-Based E2E Driving Dataset. The red trajec- tory shows the predictions of RAP-DINO, and the green trajectory denotes the ground-truth future path. Across the four examples, RAP-DINO successfully identifies stop sign, traffic cones, and oncoming vehicles at night, leading to accurate and safe trajectory plans.
A.2 VISUALIZAATIOSS FOR ABLATION STUDY ON 3D RASTERIZATION
To better illustrate the impact of different design choices in our 3D rasterization pipeline, we pro- vide qualitative visualizations in Fig. 9 We compare object face rendering (colored vs. transparent), depth decay (enabled disabled), and background style (pure black natural sky-ground split). These examples highlight how each choice affects visual cues such as object semantics, depth per- ception, and training stability.
15

---

Published as a conference paper at ICLR 2026
Natural Background
Transparent Faces
No Depth Decay
Baseline
Figure 9: Qualitative comparison of 3D rasterization choices. Columns correspond to different design choices: background (natural vs. black), face rendering (transparent vs. colored), and depth decay (off vs. on). The configuration with colored faces + depth decay black background (right- most) provides the most informative yet stable representation.
A.3 LLM USAGE
In compliance with ICLR 2026 policy, we acknowledge the use of large language models (LLMs) in preparing this paper. LLMs were used to polish the writing, assist retrieving related work, and provide limited support for brainstorming experimental designs. All research ideas, experiments, analyses, and conclusions are the sole work and responsibility the authors.
Table 8: Training hyperparameters RAP.
Parameter
Value
Number of GPUs Total batch size
x H100
128 for Navsim 64 for Weiv
Optimizer
Adam W (Loshchilov & Hutter 2017
Learning rate (initial)
lc-4
Learning rate schedule
Cosine Learning Rate Decay le-4
Weight decay
Epochs (pretraining)
Epochs (finetuning)
Gradient clipping
0.0
Domain alignment loss weights (Aspatial, Agtobal)
(0.002, 0.1)
Dropout
0.1
A.4 MORE TRAINING DETAILS
Hyperparameters. We summarize the training hyperparameters of RAP in Table Unless oth- erwise noted, the same configuration used across benchmarks, with dataset-specific finetuning schedules described in Sec. 4
Gradient Reversal Layer: For adversarial global alignment, we adopt gradient reversal layer following Ganin & Lempitskyl (2015). During the forward pass, the GRL acts identity map- ping, while in the backward pass it multiplies the gradients by - X, forcing the encoder to learn Nomain-invarian features. We schedule À with smooth annealing function
A(p) = 0.1 * exp(-7p) 1), E [0
where p is training progress and 7 = 10 by default. Our domain classifier is a lightweight MLP applied to aggregated features.
16


Reviews:
[{"review_id": "qPHVBNGTiL", "reviewer": "Reviewer_eDtm", "summary": "This paper presents an end-to-end driving framework that uses 3D rasterization instead of costly photorealistic rendering. By focusing on semantic and geometric fidelity and aligning synthetic with real data in feature space, RAP enables more flexible data augmentation and achieves robustness and generalization across multiple driving benchmarks.", "questions": "- do the authors plan to evaluate RAP outside of the IL setting? rl evaluations would demonstrate if the model can generalize under agent exploration\n- can the authors provide additional examples showing reconstruction results on very OOD scenes? e.g. scenes with objects that have very little representation in the training dataset.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- replaces expensive photorealistic rendering, offering a controllable way to generate diverse driving scenarios.\n- provides experiments and ablations across multiple major benchmarks (NAVSIM, Waymo, Bench2Drive), consistently achieving strong results\n- bridges synthetic and real data efficiently, reducing the sim-to-real gap without costly photorealistic rendering", "weaknesses": "- RAP remains within the imitation learning framework, so it still can display issues like causal confusion and lack of active policy improvement.\n- the simplified rasterized scenes may miss fine-grained visual cues or rare real-world conditions that could matter for extreme edge-cases", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tIxp5DXwAp", "reviewer": "Reviewer_HRhi", "summary": "This paper propose that 3D rasterization of annotated primitives are semantically similar to images, so it can be used as input for end-to-end driving. By doing so, multiple data augmentation tricks can be applied to end-to-end driving, without costly image rendering. A raster-to-real alignment is further proposed to align the two representations in feature spaces.", "questions": "* Any rules considered in Recovery-oriented perturbations, for example, not collide with nearby vehicles?\n* Why use minADE as metric in ablation study?\n* In Fig.5, why the performance of 100% real data is not the best?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* For the input of end-to-end driving models, a 3D rasterization representation is proposed to be the substitute of raw images , which is interestingly to already have similar semantics in DINOv3 feature space.\n* With this representation, two types of augmentation are applied in the training of end-to-end driving model, with the need for costly image-space rendering in previous methods.\n* A raster-to-real alignment module is proposed to enforce feature consistency between rasterized and real inputs at both spatial and global levels.", "weaknesses": "* On NAVSIM and WOD benchmarks, the method achieves SOTA performance. However, on the real close-loop benchmark Bench2Drive, the performance gain is incremental compared with baseline, which is contrary to Recovery-oriented perturbations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WBcqgVpb7G", "reviewer": "Reviewer_y7Na", "summary": "This paper addresses the critical problem of brittleness in end-to-end autonomous driving models trained via imitation learning. The authors argue that this brittleness stems from the lack of \"recovery\" data, as models are only exposed to expert demonstrations. To overcome this, they propose Rasterization Augmented Planning (RAP), a scalable data augmentation framework.\n\nThe core contribution is a lightweight 3D rasterization pipeline that generates synthetic camera views from annotated geometric primitives (e.g., agent cuboids, lane polylines). This approach deliberately forgoes photorealism, arguing that semantic and geometric fidelity are sufficient for training robust driving planners, and that this method is far more scalable than computationally expensive alternatives like neural rendering or game engines.\n\nThis rasterization pipeline enables two key data augmentations: (1) **Recovery-oriented perturbations**, which simulate off-expert-path maneuvers to teach the policy how to recover from mistakes, and (2) **Cross-agent view synthesis**, which re-renders scenes from the perspective of other agents to dramatically increase the volume and diversity of training data.\n\nTo bridge the domain gap between the abstract rasterized views and real-world images, the paper introduces a Raster-to-Real (R2R) alignment module. R2R operates in the feature space, using a combination of spatial-level feature distillation (aligning real features to \"cleaner\" raster features) and global-level adversarial adaptation to enforce domain invariance.\n\nThe authors demonstrate the effectiveness of RAP through extensive experiments, achieving state-of-the-art results on four major benchmarks: NAVSIM v1/v2, the Waymo Open Dataset (WOD) Vision-based E2E Challenge, and Bench2Drive. The results show consistent improvements in closed-loop robustness and long-tail generalization, not only for their proposed model but also when RAP is applied to existing state-of-the-art planners.", "questions": "1. Regarding the R2R alignment module: The spatial alignment loss (Eq. 2) updates the real features `F^r` to match the frozen raster features `F^s`, treating the latter as a \"clean\" supervision signal. Could you elaborate on the intuition behind this directional alignment? Did you experiment with a symmetric loss or alternative objectives where both feature distributions are jointly optimized?\n2. Regarding the scaling laws (Fig. 6): The log-scaling behavior with cross-agent data is very promising. What do you theorize is the limiting factor for this scaling? Does the performance plateau come from the finite variety of agent behaviors in the log, or from the inherent domain gap that cannot be fully closed by R2R alignment?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. **Pragmatic and Well-Motivated Core Idea:** The central thesis—that photorealism is not necessary for training robust E2E planners and that semantic/geometric fidelity is sufficient—is compelling and challenges a prevailing trend in the community. This provides a practical, scalable, and computationally efficient alternative to the prohibitively expensive photorealistic simulation pipelines (NeRFs, 3DGS, etc.). This insight is a significant conceptual contribution.\n2. **Exceptional Experimental Validation:** The empirical evidence supporting the paper's claims is extensive and highly convincing.\n    - **Breadth:** Achieving #1 ranking on four diverse and challenging benchmarks (NAVSIM, WOD, Bench2Drive) is a remarkable feat and strongly validates the method's effectiveness and generalizability.\n    - **Rigor:** The evaluation covers both open-loop metrics (WOD) and, more importantly, pseudo-closed-loop (NAVSIM v2) and full closed-loop (Bench2Drive) simulations, directly addressing the core problem of closed-loop brittleness.\n    - **Model-Agnosticism:** The authors demonstrate that RAP is not just a component of their specific model but a general framework that provides significant gains when applied to other SOTA models (e.g., `RAP-iPad`, `RAP-DiffusionDrive`). This greatly strengthens the paper's impact.\n3. **Thorough and Insightful Ablation Studies:** The paper includes a comprehensive set of ablations that methodically validate the key design choices.\n    - The study on rasterization design (Table 5) clarifies why specific choices like solid faces and depth decay are important.\n    - The ablation on recovery perturbations (Table 6) directly links this augmentation to improved performance on the closed-loop-oriented NAVSIM v2 benchmark.\n    - The analysis of the R2R alignment module (Fig. 5) clearly shows the benefit of both spatial and global alignment.\n    - The scaling curve for cross-agent synthesis (Fig. 6) is particularly strong, demonstrating a clear log-scaling law that mirrors findings on real data scaling. This provides powerful evidence for the value of the generated synthetic data.\n4. **High-Quality Presentation:** The paper is well-written, clearly structured, and easy to follow. The figures, especially the comparative illustration in Figure 1 and the system overview in Figure 2, are excellent and effectively communicate the core concepts.", "weaknesses": "1. **Oversimplification of the Static World:** The rasterization pipeline represents the world as a set of key primitives against a simple (often black) background. While the ablations show this is effective, it discards a vast amount of visual context from the static environment (buildings, foliage, non-annotated road signs, etc.). This simplification might be a vulnerability in complex urban scenes where such context is semantically important for driving decisions (e.g., navigating a complex, non-standard construction zone).\n2. **Information Loss.** A primary weakness of the RAP framework is the inherent information loss resulting from its reliance on abstract 3D rasterization. By training the real-world encoder to align its features with a \"clean\" representation derived solely from annotated primitives (e.g., agent cuboids and lane lines), the model is implicitly taught to ignore any visual information not present in the labels. This poses a significant safety risk, as the system may fail to perceive critical, unannotated hazards such as road debris, potholes, temporary construction signs, or police officers directing traffic. Furthermore, this abstraction discards subtle but predictive visual cues, like wet road surfaces or holes in the road, which are crucial for nuanced, proactive driving. While this approach demonstrably improves robustness against common visual variations like weather and lighting, it does so at the cost of creating a potential \"blindness\" to novel or out-of-annotation long-tail events, a critical limitation for real-world deployment.\n3. **RL Training.** A weakness is the paper's narrow focus on imitation learning, which fails to exploit the full potential of the proposed RAP framework. The authors have effectively built a fast and scalable driving simulator, an ideal environment for reinforcement learning (RL). By restricting their method to data augmentation for IL, they miss the opportunity to use RL to train an agent that could learn through active exploration, optimize for more complex rewards, and potentially discover policies superior to the original expert. This confinement to IL undersells the power of their core contribution, leaving its most transformative application as an RL training engine unexplored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)