VR-Drive: Viewpoint-Robust End-to-End Driving with Feed-Forward 3D Gaussian Splatting
Authors: Hoonhee Cho
Jae-Young Kang
Giwon Lee
Hyemin Yang
Heejun Park
Seokwoo Jung
Kuk-Jin Yoon
Venue: NeurIPS 2025 poster ()
Keywords: End-to-End Autonomous Driving
Autonomous Driving
3D Reconstruction
Primary area: applications

Abstract: End-to-end autonomous driving (E2E-AD) has emerged as a promising paradigm that unifies perception, prediction, and planning into a holistic, data-driven framework. However, achieving robustness to varying camera viewpoints, a common real-world challenge due to diverse vehicle configurations, remains an open problem. In this work, we propose VR-Drive, a novel E2E-AD framework that addresses viewpoint generalization by jointly learning 3D scene reconstruction as an auxiliary task to enable planning-aware view synthesis. Unlike prior scene-specific synthesis approaches, VR-Drive adopts a feed-forward inference strategy that supports online training-time augmentation from sparse views without additional annotations. To further improve viewpoint consistency, we introduce a viewpoint-mixed memory bank that facilitates temporal interaction across multiple viewpoints and a viewpoint-consistent distillation strategy that transfers knowledge from original to synthesized views. Trained in a fully end-to-end manner, VR-Drive effectively mitigates synthesis-induced noise and improves planning under viewpoint shifts. In addition, we release a new benchmark dataset to evaluate E2E-AD performance under novel camera viewpoints, enabling comprehensive analysis. Our results demonstrate that VR-Drive is a scalable and robust solution for the real-world deployment of end-to-end autonomous driving systems.

VR-Drive: Viewpoint-Robust End-to-End Driving with Feed-Forward 3D Gaussian Splatting
Hoonhee Cho
Jae-Young Kang Giwon Lee' Hyemin Yang!
Heejun Park'
Seokwoo Jung' Kuk-Jin Yoon'
KAIST 422ot
https:/wvddiveneuiis.gghuuu gittuuu
Abstract
End-to-end autonomous driving (E2E-AD) has emerged as a promising paradigm that unifies perception, prediction, and planning into a holistic, data-driven frame- work. However, achieving robustness varying camera viewpoints, common real-world challenge due to diverse vehicle configurations, remains open prob- lem. In this work, we propose VR-Drive, novel E2E-AD framework that addresses viewpoint generalization by jointly learning 3D scene reconstruction as auxiliary task to enable planning-awe aware view synthesis. Unlike prior scene-specific synthe- sis approaches, VR-Drive adopts ferd-forward inference strategy that supports online training-tima time augmentation from sparse views without additional annota- tions. To further improve viewpoint consistency, we introduce viewpoint-mixed memory bank that facilitates temporal interaction across multiple viewpoints and a viewponnt-consistent distillation strategy that transfers knowledge from original to synthesized views. Trained a fully end-to-end manner, VR-Drive effectively mitigates synthesis-induced noise and improves planning under viewpoint shifts. In addition, we release a new benchmark dataset to evaluate E2E-AD performance under novel camera viewpoints, enabling comprehensive analysis. Our results demonstrate that VR-Drive is a scalable and robust solution for the real-world deployment of end-to-end autonomous driving systems.
Introduction
The end-to-end autonomous driving (E2E-AD) system refers to the integration of all modules, including perception, prediction, and planning nodes. The end-to-end driving paradigm g [1
55 10
4 16 54. 17 [22 has consistently gained attention holistic approach, wherein the perception and prediction tasks are effectively integrated to support planning. This integration enhances both performance and efficiency, favoring unified model for the entire driving task. This data-driven approach, compared to traditional rule-based planning, designed to function robustly in complex scenarios by integrating various perception tasks (e.g., detection, tracking, mapping, etc.). During the training process, incorporates vast amounts of data and annotations to enhance its capabilities.
Despite significant advancements and strong performance across various scenarios, existing end-to- end autonomous driving (E2E-AD) must evolve into scalable and flexible holistic models become viable industry solutions. Recent E2E-AD systems [14]
67
63
31
73
50
g 32 in particular, aim
to achieve comparable performance using only raw camera input. However, the viewpoint of the camera 30 45 can vary depending on the vehicle's type and make, and systems that can effectively adapt to these changes are crucial from a real-world application perspective. A straightforward solution to this challenge would be to collect data using a variety of vehicles and camera rigs, and
Equal contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Ground truth
Ours
DiffusionDrive
Roof
Alvil
Poot Fucul
Original
Ameohald Ancoerhen
Reathed
Novel
Ground truth
Original/Ours Original/DiffusiooDriie
Novel/Ours
Novel/DiffusionDrive
Figure Example scenario where surrounding vehicles have stopped a traffic signal. In the original training view, both our VR-Drive and DiffusionDrive [35] perform well in perceiving nearby vehicles and planning. However, with lowered camera height, DiffusionDrive fails to detect surrounding vehicles, leading to a trajectory that collides with the front vehicle, posing safety risk. In contrast, VR-Drive maintains accurate perception (except for those occluded due to the lowered camera height) and plans trajectories as effectively as in the original view.
then use this data during the training process. However, this approach impractical because it is impossible to pre-build camera viewpoints for every type of vehicle. Additionally, E2E-AD networks require annotations for various tasks, which incur significant costs, making an impractical direction. Furthermore, to be deployable across different types of vehicles, the model must be flexible and robust not only to the predefined data but also to out-of-distribution (OOD) data. Therefore, the network must also ensure its generalization ability during the training process.
To this end, we tackle the critical real-world challenge of generalization to diverse camera rigs in end-to-end autonomous driving (E2E-AD) systems. Specifically, we propose VR-Drive, which jointly learns 3D scene reconstruction as an auxiliary modular task within E2E-AD to augment the diversity of camera viewpoints. While numerous prior works 22
have explored novel view synthesis through 3D reconstruction, these methods are typically scene-optimized and require significant computational resources, making them unsuitable for real-time downstream tasks. Therefore, we advocate for an online scene reconstruction approach that operates effectively with sparse views. To this end, adopt a feed-forward inference strategy (53 58
3
74 29
3 ensure efficiency. Rather than training a separate novel view synthesis model, we integrate it as a joint modular task within the end-to-end framework, thereby reducing training complexity. Moreover, to prevent errors in view synthesis from propagating and degrading the final planning performance, VR-Drive introduces unified framework that incorporates 3D reconstruction an auxiliary task within E2E-AD, enabling novel view synthesis without requiring additional annotations, To learn viewpoint-robust and consistent feature space, VR-Drive utilizes a viewpoint-mixed memory bank that encourages interaction between features from different viewpoints in the sequential training process by allowing them to mix in 3D space. Additionally, to mitigate the potential noise embedded in the features extracted from transfers knowledge from the original view features to guide the learning these synthesized features. Benefiting from its end-to-end joint training, this planning-aware synthesis strategy ensures that the model remains effective under viewpoint shifts and contributes to improved downstream planning. As shown in Fig. VR- Tr-Drive maintains robust performance under varying camera viewpoints, unlike
auppentt images, we propose a distillation strategy that

---

existing E2E-AD methods that are sensitive to such changes, demonstrating its potential as scalable and reliable end-to-end autonomous driving solution for real-world deployment.
The main contributions and unique aspects of our work are summarized as follows:
We tackle viewpoint robustness in end-to-end autonomous driving (E2E-AD) by jointly learning 3D reconstruction for planning-awarr view synthesis, enabling training data augmentation across diverse viewpoints and improving generalization to unseen camera configurations.
We propose a viewpoint-mixed memory bank that enables temporal interaction between features from different viewpoints, and introduce a viewpoint-consistent distillation strategy that transfers knowledge from original viewpoint images to their corresponding augmented novel view synthesis images in a 3D space.
We introduce a new benchmark dataset for E2E-AD to evaluate robustness under novel camera viewpoints unseen during training.
2
Related Works
2.1 End-to-End Autonomous Driving
End-to-end autonomous driving (E2E-AD) aims generate final driving plans directly from raw sensor inputs within an integrated framework, in contrast to conventional methods that separately train perception, prediction, and planning modules. Previous E2E-AD works can be largely categorized into two major directions: (1) focusing on architecture and task exploration. and (2) leveraging high-level information distillation. Architecture-be basud approaches, such as (2 23 69, demonstrate that submodules within an integrated framework can be optimized to enhance the final planning performance. The following works 27 33
further improved planning accuracy by removing certain
auxiliary tasks, such as occupancy prediction and motion prediction. In contrast, [60] reorganized traditionally sequential auxiliary tasks into parallel structure, while (33) proposed task-aware training strategy to better leverage task relationships in parallel settings.
Architecture-basse methods rely on large-scale annotated data, but often struggle in diverse scenarios due to biased training distributions, leading to issues such as causal confusion and long- tail errors. To address this, several studies have explored distilling actions and feature information from rule- based or reinforcement learning (RL)-based experts trained in privileged settings [62] 77 25| 24). Additionally, there has been research utilizing language models for scene representation, prediction, and planning, enhancing situational understanding and adaptability through the general knowledge embedded in large-scale foundation models
48 49
Despite various research directions in E2E-AD, no prior work has addressed the development of model architectures that are robust to novel sensor viewpoints. This challenge is particularly critical, as sensor viewpoint variation is an inevitable and realistic factor in real-world deployments, arising from differences in vehicle types, sensor configurations, and mounting positions. However, it remains difficult to address within existing E2E-AD architectures and training paradigms, which are heavily dependent the sensor inputs seen during training. In this work, take the first step toward overcoming this limitation by proposing a method that enhances robustness to unseen sensor views.
2.2
Viewpoint-Roben Representations and Scene Reconstruction
have shown that neural networks are vulnerable to viewpoint changes, especially under distribution shifts. While these studies explored adversarial viewpoints in 2D have extended this line of research to address viewpoint robustness in 3D perception tasks. They typically leverage novel view synthesis to generate images under varying camera viewpoints, aiming to train perception algorithms that are robust across diverse views. Research on novel view synthesis via 3D scene reconstruction 665 61
Early studies 40
perception, more recent efforts [30 [8] 2
36
56 has
advanced significantly, particularly with the emergence of Neural Radiance Fields (NeRF) 43 and
3D Gaussian Splatting (3DGS) (28). However, most methods scene-specific and require long training times, as they rely on scene-by seee-by-scenee optimization.
To be applicable to scalable E2E-AD, view augmentation strategy must satisfy two key requirements. (1) Since the test-time camera viewpoint not fixed and can vary widely, the model must be robust to arbitrary views. This requires synthesizing diverse novel views during training, which in turn
3

---

Original-view learning
Perception-plamiing learning
Viewpoint- Mixid Memory Bank
Image
w/o Aug:
Original View
+
Random Select
Gaussians
Features
Detection
Mapping
Planning
w/ Aug.
3D Gauss
Original
Original
Image Encoder
Only used Aug.
Mult-view
Novel View
Features
Viewpolnt-Consleee Distillation)
Novel-view learning
Figure Overall framework of VR-Drive. Our overall framework consists of three main components, as follows: (1) original-view learning, (2) novel-view learning, and (3) perception-planning learning. For novel-view learning, the perception-planning head is randomly assigned to either the original or a novel view during training, allowing the model to generalize across different viewpoints.
demands real-time online processing for both training and inference. (2) To be effective in driving scenes, the method must support 3D reconstruction even with sparse or low-overlap observations. To adopt feed-forwart 3D gaussian splatting [52 71 3 531 that is both
meet these requirements, generalizable and capable of online training and inference. By incorporating 3D scene reconstruction sub-task within E2E-AD, enhance scene-level understanding and achieve performance gains even for the original viewpoints. Furthermore, by jointly training the view synthesis and driving tasks in end-to-end manner, account for potential synthesis emros and demonstrate the feasibility of extending novel view synthesis as practical means to improve viewpoint robustness in E2E-AD.
3 Methods
3.1 Overall Framework
Given multi-view images, end-to-end autonomous driving (E2B-AD) models jointly learn perception and motion prediction to produce accurate motion plans for the ego vehicle. In addition to the standard pipeline of existing E2E-AD approaches, the proposed VR-Drive incorporates scene reconstruction as an auxiliary task, leveraging 3D Gaussian Splatting (3DGS) (28] The overall framework of VR-Drive shown in Fig. 2 VR-Drive comprises three components, each targeting distinct objective: (I)
original-view learning, (2) novel-view learning, and (3) perception-plamning learning.
the original view the default input of the pipeline. Given multi-view images, the image encoder (ResNet50 (19) first extracts original multi-view feature maps, / R-NxCxHX where N is the number of camera views. These generated feature maps are utilized not only for perception and planning in autonomous driving, but also for learning and rendering novel views via 3DGS. We build on the original 3DGS framework 28 which represents scene using Gaussian primitives spherical harmonics for color c. The covariance matrix constructed by combining the scaling factor and rotation quaternion r. Unlike the original 3DGS that relies on tructure-from-mootie for optimizing μ4, predict primitives in a feed-forward [53]. pixel-wise manner directly from input images. Similar to previous work [51 that treated depth estimation as an auxiliary task within E2E-AD, we jointly learn depth as part of the E2E-AD framework. The estimated depth is then used to infer the position of Gaussian primitives E IR3, We use the predicted depth map D and image feature map /, as input to a Gaussian network composed of multiple convolutional layers. This network predicts the remaining parameters of each Gaussian primitive, including the scaling factor s E RRS rotation quaternion
Original-view learning:
During training,
(Jr, Σ, a c), defined by position J4, covariance Σ,, opacity o, and
RA, opacity a [0, 1), and color e E IR* represented by k-degree spherical harmonics. To ensure valid ranges, apply softplus to H. and sofumax to H

---

Perception Pipeline
Viewpoint-Mixee
Memory Bank
Origin
Memory
Initial Detection
Viewpoint- Mixed
Memory Bank
Viewpoint Cont--onsttttt
In
Distillation
Refinement
Memory
Out
Novel
Mixed
Anchor & Feature
Key
Viewpoimt Cross Attentitnnn
Origin
Value
Self AiffAAttention
Dis Distilaaioon
Laistill
Query
Viewpoint-Mixxe
Cross-view
Single
Feature
Novel
2D Features
View
Figure 3:
Illustration of the perception pipeline. VR-Drive includes two complementary techniques
to ensure consistent feature representations across camera viewpoints: Viewpoint- Memory Bank and Viewpoint- Conpoint-Consistent Distillation.
enforeing E T and a E [o, 1]. The feed-forward design enables online inference on novel views and generalization new inputs without scene-specifie constraints.
Novel-view learning: VR-Drive aims to achieve robust planning performance by generating con- sistent feature representations even for camera viewpoints that were not observed during training. Specifically, at test time, seeks to replicate the feature space of the original view across diverse, unseen viewpoints. To this end, randomly sample camera extrinsies and render multi-view feature maps from arbitrary perspectives using the Gaussian primitives generated from the original view. Given the rendered multi-view images from a novel view, generate novel view features, TER NxCx may differ in distribution from the original, we guide the model to generate feature representations that closely align with those of the original view. We observe that feed-forward 3DGS facilitates scene-level 3D understanding, which proves beneficial even under novel viewpoints. To encourage robustness, additionally employ cyclic reconstruction loss that trains the model regenerate the original view from a novel one.
using shared image encoder with the original view. Since the novel view features
Perception- planniing learning: VR-Drive selectively trains original and novel views during the training to achieve robustness across diverse camera viewpoints. The image features extracted from the selected view passed to the perception and planning heads, enabling planning based on the corresponding perception representation. Following 335 26 27). we adopt 3D object detection and mapping as perception tasks. More specifically, achieve efficient representation,
utilize same sparse architecture that leverages anchor- and instance feature-based designs [38 39 for both detection and mapping tasks. Since the two tasks differ only in the dimensionality of the anchors, we provide all descriptions and definitions in the context of detection, which are equally applicable mapping. We first generate initial bounding box proposals using the detection module (38), E TR.xNE where is the number of anchors and Ng is the dimensionality of each anchor. For each proposal, we also extract the corresponding instance features F' = {F"2, _ FM, E IR xN,, where N; is the dimension of the instance feature. This allows us encode the surrounding agents in the 3D space based on the extracted image features. As illustrated in Fig. we insert viewpoint-robuss modules into the perception pipeline for detection and mapping, in addition to the conventional detection components. Specifically, we introduce two dedicated components within the perception stage of VR-Drive: the Viewpoint-Mixed Memory Distillation strategy, designed address feature variations
denoted as B [B1, B Bd)
Bank and the Neewponnt-Cosissee across viewpoints and promote canonical feature learning. We obtain the final perception results by refining the viewpoint-robust features through an additional detection decoder. Finally, to enable planning that interacts with the predicted agents, adopt the motion planner proposed in (35).
5

---

3.2 Viewpoint-Mixed Memory Bank
As mentioned in Sec. 3.1 the perception and planning pipeline randomly receives features from either the original view or a novel view during training. To promote canonical 3D feature learning from image imputs across diverse viewpoints with different distributions, we encourage interaction between 3D features extracted from diverse view during training. Rather than simply using single pair of original and novel views, limiting the model to observing only two viewpoints within a single forward pass, we adopt a memory bank strategy that stores and updates features from continuously changing novel views to promote broader viewpoint generalization. Let F E R x be the instance features retrieved from the viewpoint-mixed memory bank, where M' is the number of sampled features. Following the method proposed in [38. we align F' to the current frame by leveraging the velocities of the anchor box and the status of the ego vehicle to compensate for temporal shifts between viewpoints. Our objective is to generate interactive features between F" and the instance features from the current view F. To achieve this, we leverage attention mechanisms [57] to fuse features from the memory bank and the current view, resulting in the following mixed feature representation:
F
Aotiss-Attetiion(( (aunntiiinnuuuyauaaa = F, Key = F', Value = =²).
(1)
The mixed feature, F, further processed through a self-attention mechanism to model interactions among agents, and are then passed to the viewpoint-consisten distillation module. The viewpoint- mixed memory bank is updated by selecting the top-K high-confidence instances after the final refinemett, while the oldest instances in the bank are discarded in an FIFO manner.
3.3 Viewpoint-Consiseee Distillation
One potential challenge in learning viewpoint robustness through novel view synthesis is that the synthesized images may contain rendering artifacts, especially in occluded or texture-less regions. Moreover, novel view settings often involve more extreme side-facing camera angles, which can be more challenging for autonomous driving due to reduced visibility or increased uncertainty in object localization. To address this, adopt a distillation strategy in which the original view, typically containing more reliable and informative features due to better visibility and camera positioning, guides the learning of novel views. One simple strategy is force alignment between two view features by projecting one onto the other using depth and pose. However, such alignment often excludes regions that are perceptually important for downstream tasks. Instead, we utilize the instance features F and their corresponding anchor boxes B to selectively distill information that is crucial from a planning perspective. Motivated by [59], we aim to extract representative object features by computing a learnable offset p and weight w for each instance i based on its instance feature F,, defined as p; = f(;;) E xx and w, = gFi) E RNxx where f and y are learnable keypoint and weight generations. Here, and N denote the number of sampled points and cameras, respectively. Then, we compute the j-th 3D sampled point as follows:
P Pt.j + position (B:),
where position (B:) denotes the 3D center coordinates (as, y, v)= of the bounding box B,. We project the sampled 3D points onto the image plane of each camera view using the corresponding transformation matrix, and extract image features at the n-th camera view via bilinear sampling, defined as:
f BilinearSamplee(T ), E C
where II, is the camera transformation matrix and I,, is the original view 2D image feature map from the n--hh camera. Then, we define the aggregated feature at anchor index as:
S. = Σ Σ Wa,i.j fn.is-
The same procedure is applied to the novel view image feature map j, resulting in S. To align the sampled features 5, from the novel view with the corresponding features S, from the original view, we apply mean squared eгror (MSE) loss between features. We restrict the loss of distillation to high-confidence anchors avoid distillation in the background or noisy boxes. Let I be the set of anchors whose confidence scores exceed a predefined threshold r. The viewpoint-consistent distillation loss is defined as:
Ldistill
š stopgrad
(5)

---

Original
Pitch +5
Pitch -10"
Height +1.0 m
Height -0.7 Depth +1.0 m
Front
Front left
Figure Variant camera viewpoints at test time, differing from the original training distribution.
where stopgrad(-) indicates gradient detachment.
Note that the viewpoint-mixed memory bank is always used, whereas the viewpoint-consistent distillation is only applied when a novel view image is used as the input for perception and planning.
3.4 Loss Functions
The loss functions consist of various tasks. For motion prediction and planning, we apply the winner- takes-all strategy [34] In the planning task, extra regression loss is introduced handle ego status. For classification, we utilize focal loss [37], while L.1 loss is used for regression in both detection and mapping tasks. Furthermore, we incorporate the viewpoint-consistent distillation loss. We also use rendering loss for scene reconstruction, as described below.
loss is also employed for depth estimation. Additionally,
Rendering Loss. We use both L2 and LPIPS [68] losses as the rendering objective. Since ground truth for various viewpoints unavailable during training, apply rendering loss through two alternative strategies, depending on whether novel view augmentation is used.
Original Reconstruction Loss. The reconstruction loss encourages the model to render novel views from input images using Gaussian primitives. As real data lacks paired novel views, we simulate them by synthesizing adjacent-time views via splat-based rendering and apply the loss to the generated outputs.
Cyclic Reconstruction Loss. When novel view is given input for perceptioonpllnneee heads, supervision using adjacent time-step images, as done with the original view, is not feasible due to the absence of paired frames. To support effective 3D scene learning with Gaussian primitives and depth, we adopt a cyclic rendering strategy that reconstructs the original view from the novel view.
The overall loss function for end-to-end training is:
C = Ldet. + Limap + Letepth + Limotion + Lptann + Lrender-
(6)
4 End-to-End Autonomous Driving Benchmark with Viewpoint Variations
4.1 Training and Evaluation Setup
Our work pioneers research on camera viewpoint variations in end-to-end autonomous driving (E2E- AD) and aims establish a framework for training and evaluation in future studies. Considering the challenges of acquiring data with varying rigs during vehicle operation in real-world applications, we fix the rig single setup during the training process. Furthermore, our goal to evaluate the model's 's robustness across various out-of-distrrie ontto--distribution data and assess its performance under different camera settings with distinct distributions. To achieve this, we introduce sensor variations at test time, deviating from the original camera configuration used during training, including: +5" pitch, -10 pitch, +1.0m height, -0.7m height, and +1.0m depth. These variations are configured based on the sensor settings from (30 to evaluate robustness.
7

---

Table Open-loop planning performance in nuScenes dataset, Metric calculation follows ST-P3 The best performance in each setting is highlighted in bold. denotes the usage of ego-status.
20
L2 (m)
Collision (%)
Camera Setting
Methods
2x
Avg.
2s
Avg.
AD-MI.P
(66
0.20
0.26
0.41
0.29
0.17
0.18
0.24
0.19
BEV-Planner*
0.28
0.52
0.84
0.55
0.13
0.17
0.36
0.22
VAD
0.41
0.70
0.72
0.07
0.17
0.22
Original
SparseDrive
0.29
0.58
0.96
0.61
0.01
0.05
0.18
0.08
DiffusionDrive
35
0.27
0.54
0.90
0.57
0.03
0.05
0.16
0.08
VR-Drive (Ours)
0.29
0.57
0.95
0.60
0.01
0.03
0.14
0.06
AD-MLP 66
0.20
0.26
0.41
0.29
0.17
0.18
0.24
0.19
BEV-Planner*
0.29
0.56
0,91
0.59
0.27
0.31
0.54
0.37
VAD 27
0.38
0.66
0.68
0.11
0.21
0.51
0.28
pitch +5
SparseDrive [1
0.32
0.63
1.03
0.66
0.02
0.08
0.35
0.15
DiffusionDrrive 35
0.33
0.64
1.04
0.67
0.00
0.09
0.24
0.11
VR-Drive (Ours)
0.29
0.57
0,94
0.60
0.00
0.02
0.14
0.06
AD-MLP" 66
0.20
0.26
0,41
0.29
0.17
0.18
0.24
0.19
BEV-Planner" 33
0.27
0.51
0,86
0.54
0.64
0.73
0.93
0.76
VAD [27
0.70
1.01
1,35
1.02
0.55
0.82
1.27
0.88
pitch
SparseDrive 611
0.46
0.91
1,50
0.96
0.03
0.15
0.50
0.23
DiffusionDrive
K5
0.45
0.91
1.52
0.96
0.02
0.16
0.55
0.24
VR-Drive (Ours)
0.34
0.66
1.10
0.70
0.02
0.08
0.24
0.11
AD-MLP 66
0.20
0.26
0.41
0.29
0.17
0.18
0.24
0.19
BEV-Planner 33
0.28
0,54
0.88
0.57
0.20
0.22
0.44
0.29
VAD [27
0.41
0.70
1.07
0.73
0.14
0.45
0.80
0.47
height +1.0
SparseDrive S1
0.42
0.83
1,36
0.87
0.10
0.45
1.08
0.54
DiffusionDrivve 35
0.81
1.44
2.14
1.46
0.17
0.78
1.47
0.81
VR-Drive (Ours)
0.34
0.66
0.69
0.05
0.11
Unseen
AD-MLP
0.20
0.26
0.41
0.29
0.17
0.18
0.19
BEV-Planner* (3
0.29
0.55
0.89
0.58
0.49
0.61
0.82
0.64
VAD 27
0.41
0.71
1.09
0.74
0.09
0.17
0.39
0.22
height -0.7 m
SparseDrive 5
0.50
0.97
1,56
1.01
0.01
0.20
0.68
0.30
DiffusionDrive 35
0.64
1,18
1,82
1.21
0.00
0.12
0.49
0.20
VR-Drive (Ours)
0.34
0.66
1.09
0.69
0.03
0.11
0.28
0.14
AD-MLP 166
0.20
0.26
0,41
0.29
0.17
0.18
0.24
0.19
BEV-Planner 33
0.29
0.55
0.89
0.58
0.17
0.23
0.43
0.28
VAD pi
0.39
0.68
1.05
0.71
0.09
0.19
0.48
0.26
depth +1.0
SparseDrive [1
0.66
1.23
1.91
1.27
0.05
0.25
0.62
0.31
DiffusionDrive 35
0.87
1.55
2.30
1.57
0.12
0.37
0.75
0.41
VR-Drive (Ours)
0.37
0.69
1.11
0.72
0.02
0.11
0.27
0.13
AD-MLP 66
0.20
0,26
0.41
0.29
0.17
0.18
0.24
0.19
BEV-Planner
0.28
0.54
0.88
0.57
0.36
0.42
0.63
0.47
VAD
0.75
1.11
0.78
0.20
0.37
0.42
Average
SparseDrive
51
0.91
1.47
0.95
0.04
0.23
0.31
Difv
35
1.14
1.17
0.36
0.41
VR-Drive (Ours)
0.34
0.65
0.68
0.01
0.07
0.24
0.11
4.2 Dataset Generation Protocol
We use the nuScenes I benchmark, which is widely used in recent E2E-AD works 38 26
5)
However, since the nuScenes dataset does not provide images from variant camera viewpoints, performed offline scene optimization as a method to obtain data from various viewpoints. We per- formed offline scene optimization (7). showcasing high-peeformance and strong geometric alignment, on nuScenes test sequences. This process enabled rendering various views for each sequence, as shown in Fig. After manually inspecting the test sequences, we excluded 4 sequences from the original 150 due to unsatisfactory quality, leaving 146 test sequences for unseen viewpoints. Note that this offline scene optimization requires significant training time for each scene, making impractical for datasets with a large number of sequences. On an NVIDIA TITAN RTX, each sequence took over hours train, and the total time of optimization and rendering took more than 3 weeks. This underscores the practicality of our online novel view synthesis approach for training.
Experiments
Experiment Setup. We evaluate the model using Average Displacement Error (ADE) and Col- lision Rate. For comparison, existing end-to-end models, including AD-MLP (66), BEV.

---

Table 2: Ablation for design choices. "SR" and "VMM" indicate scene reconstruction and viewpoint- mixed memory bank. "CR" and "VCD" indicate cyclic reconstruction loss and siewpoint-consitter distillation, respectively. A indicates that scene reconstruction is learned jointly, but the generated novel view images are not used as perception and planning input.
Modules
Unseen Average
L.2 (m)
4
Collision (%)
4.
L.2
(m)
Collision (%)
ID
SR VMM CR VCD
2s
36
Avg.
1s
2s
3x
Avg.
1s
28
3x
Avg.
1s
2x
3s
Avg.
0.31
0.60
0.98
0.63
0.13
0.10
0.19
0.14
0.47
0.88
1.38
0.91
0.17
0.25
0.48
0.30
2
A
0.28
0.56
0.93
0.59
0.00
0.05
0.17
0.07
0.46
0.87
1.36
0.90
0.04
0.20
0.53
0.26
3
0.31
0.60
0.97
0.63
0.08
0.08
0.20
0.10
0.40
0.76
1.20
0.79
0.04
0.16
0.36
0.19
0.31
0.59
0.95
0.62
0.02
0.37
0.70
1.12
0.73
0.03
0.13 0.36
0.17
5
/
/
V
0.29
0,56
0.93
0.59
0.04
0.06
0.19
0.09
0.33
0.64
1.06
0.68
0.04
0.13
0.31
0.16
6
0.31
0.59
0.94
0.61
0.02
0.05
0.16
0.08
0.37
0.70
1.14
0.73
0.02
0.09
0.31
0.14
7
5 0.29
0.57
0.95
0.60
0.01
0.03
0.14 0.06
0.34
0.65
1.06
0.68
0.01
0.07 0.24
0.11
Table 3: Analysis of the range of random extrinsics for novel views during the training process.
Seen
Unseen Average
L2 (m)
Collision
(%)
L2 (m)
Collision (%)
Settings
1s
28
38
Avg.
Is
2x
38
Avg.
1s
2s
3s
Avg.
28
3s
Avg.
0.29
0.57
0.95
0.60
0.01
0.03
0.14
0.06
0.34
0.65
1.06
0.68
0.01
0.07
0.24
0.11
Superset
0.29
0.57
0.95
0.60
0.00
0.03
0.15
0.06
0.33
0.65
1.06
0.68
0.03
0.09
0.25
0.12
Subset
0.30
0.60
0.99
0.63
0.00
0.05
0.16
0.07
0.41
0.79
1.27
0.82
0.02
0.09
0.27
0.13
Planner 33 VAD (27). SrirseDrive [51] and DiffusionDrive [35]. During training, we rendered
random novel view images with pitch in the range [-100, 5%), height in [-0.7m, 1.0m], and depth in 00.2m, 1.0m]. which broadly covers the test configurations. Additional implementation details will be described in the supplementary material.
Experimental Results. Table shows the performance of E2E-AD models on both original and novel views, where "unscen" refers to data that was not provided during training. When focusing on the performance in both the original and unseen domains, we begin by comparing the performance of our proposed VR-Drive with DiffusionDrive as example. On the original domain, both models show similar performance. However, when evaluated on the unseen domain, DiffusionDrive experiences a significant increase in both ADE and collision rate. In contrast, our method demonstrates performance comparable to the original view, even in more challenging camera viewpoints under previously unseen distributions.
Ablation Study
Effect of the components. We conducted an ablation study on each module, shown in Table 2 Notably, comparing ID-1 and ID-2 reveals that simply enabling joint learning of scene reconstruction already improves performance on both original viewpoints. This suggests that online joint optimiza- tion with 3DGS contributes to improving the scalability of E2E-AD systems, likely by encouraging a more precise comprehension of 3D geometry. Such enhanced geometric understanding facilitates more informed and reliable planning decisions. The most significant performance gain emerges ID-3, where the novel view generated via scene reconstruction is used as an additional input to the model. Beyond this, the proposed modules further contribute to performance improvements. Interestingly, our method does not suffer from a trade-off where improved performance on novel views comes at the cost of degraded performance on original views. Instead, the proposed components enhance the model's overall capability, even in the original views. This suggests that novel views serve an effective form of augmentation during training, and the introduced modules help guide the model to learn better representations, ultimately benefiting both original and novel view settings.
Range of random extrinsics. We study the distribution shift between training and testing in terms of camera viewpoint diversity, as summarized in Table 3 For the experiments in Table ! we set the training-time random extrinsic ranges to pitch E 1 .5%, height [-0.7m, 1.0m], and depth E [-0.2m, 1.0m]. To examine generalization beyond the test distribution, the "Superset" setting expands the training sensor range to pitch E [-15", 100), height E [-100m, 1.5m]. and depth E [-0.5m, 1.5m], covering viewpoints that go beyond the test distribution. This allows us to investigate whether the model remains robust when trained with broader range of viewpoints.

---

Conversely, the "Subset" setting limits the sensor range to pitch E 1- 29], height E [-0.3m, 0.5m], and depth E [-0.1m, 0.5m], ensuring that the training views do overlap with any of the test-time configurations. Our model performs consistently across the Superset, Subset, and original settings, demonstrating robustness to continuous viewpoint variation.
Closed-loop Evaluation on the CARLA dataset
Table 4: Closed-loop test on CARLA dataset.
Unseen
Original
Methods
pitch +5
pitch -10
height +1.0 0
height -0.7
depth +1.0 m
Average
DS
RC
DS
RC
DS
RC
DS
RC
DS
RC
DS
RC
DS
RC
Town0S-Nov
20
44.24
100.00
41.00
100.00
23.85
100.00
25.83
100.00
28.60
100.00
32.06
100.00
30.27
100.00
TCP 62
92.73
92.73
70.33
80.33
4.65
4.65
88.51
88.51
0.00
0.00
91.11
91.11
50,92
52.92
AD-MLP
66
13.59
32.83
13.59
32.83
13.59
32.83
13.59
32.83
13.59
32.83
13.59
32.83
13.59
32.83
BEV-Planner
33
17.25
28.70
7.30
28.89
7.74
28.83
8.51
28.95
7.69
28.70
7.75
28.95
7.80
28.86
Baseline
76.47
99.20
69.41
89.60
45.65
99.38
48.67
100.00
41.59
86.76
35.95
98.60
48.25
94.87
VR-Drive (Ours)
84.04
99.04
75.00
100.00
91.26
98.76
98.44
98.99
80.67
97.32
95.88
96.35
88.25
98.28
We use the CARLA 0.9.10.1 simulator 15|
for closed-loop testing. For the closed-loop test, we evaluate performance using the Town05short benchmark. We collect the training data from Town01, 02, 03, 04, 06, 07, and 10, using scenario routes based on previous work [46]. For the evaluation, we assessed each model's performance based on two key metries: Driving Score (DS) and Route Completion (RC). To provide a comprehensive comparison, we included several established end- to-end autonomous driving models. Specifically, we evaluated ST-P3(20) TCP(62), AD-MLP [66]. BEV-Planner (333, and baseline alongside our proposed method. As the baseline, we adopt the ID-1 setting from Table 2 removing all proposed modules.
Following existing works to enable training and evaluation on novel viewpoints, we establish a new benchmark. Specifically, we sample 20% of sequences from Town05 Short construct Town05-Nov, which serves as our novel-view evaluation set. For training data, we follow Transfuser [46] and collect samples using the autopilot, but only from original viewpoints. For fair comparison with prior works, we handle baselines based on their available resources. In the case of ST-P3 (20 and TCP 62). since pretrained checkpoints on Town05 are publicly released, we directly evaluate these models without retraining.
27 46),
we adopt the Town05 benchmark for simulation. However,
Table f shows the closed-loop evaluation results on the Town05-Nov benchmark. Existing end-to-end autonomous driving approaches tend struggle with planning unseen test scenarios, sometimes failing to initiate driving in particularly challenging cases. Notably, the DS metric is more adversely affected compared RC, experiencing degradation in perception performance when faced with novel viewpoint inputs. In contrast, our method demonstrates performance on unseen tests that is comparable that of the original viewpoint.
Conclusion
In this work, present VR-Drive, unified end-to-end autonomous driving framework that leverages novel view synthesis and viewpoint-robus learning. To the best of our knowledge, we are the first to study camera viewpoint variation in E2E-AD for real-world applications. We benchmark VR-Drive on the nuScenes dataset and the CARLA simulator, achieving state-of-the-art performance across diverse camera viewpoints and diitrf--iitributiion conditions.
Limitation and Future Work. The performance of VR-Drive is influenced by the accuracy of camera calibration. While errors in calibration may lead to suboptimal results, the system could be made more robust to such errors. Addressing this issue and enhancing the system's 's robustness to calibration inaccuracies could be an important focus for future work.
10

---

Acknowledgement
We thank 42dot for funding this research. We also appreciate the support and valuable dis- cussions from the members of the 42dot research team. This work was supported by the National Research Foundation of Korea(NRFF grant funded by the Korea governmen((MSIT) (NRF2022R AAA0022RRA2BBR A2B5B0030066))))
References
[1] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621-1 1631, 2020.
[2] G. Chang, J. Lee, D. Kim, J. Kim, D. Lee, Ji, S. Jang, and S. Kim. Unified domain gener- alization and adaptation for multi-view 3d object detection. Advances in Neural Information Processing Systems, 37:58498-58524 2024.
[3] D. Charatan, S. L. Li, A. Tagliasacchi, and V. Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable reconstruction. In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pages 19457-19467, 2024.
[4] L. Chen, P. Wu, K. Chitta, B. Jaeger, A. Geiger, and H. Li. End-to-end autonomous driving: Challenges and frontiers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[5] S. Chen, B. Jiang, H. Gao, B. Liao, Q. Xu, Q. Zhang, C. Huang, W. Liu, and X. Wang. Vadv2: End-to-end vectorized autonomous driving via probabilistic planning, arXiv preprint arXiv:2402. 13243, 2024.
[6] Y. Chen, Xu, C. Zheng, B. Zhuang, M. Pollefeys, Geiger, T.-J. Cham, and Cai. Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images. In European Conference Computer Vision, pages 370-386. Springer, 2024.
[7] Z. Chen, J. Yang, J. Huang, R. de Lutio, J. M. Esturo, B. Ivanovic, O. Litany, Z. Gojeic, S. Fidler, M. Pavone, et al. Omnire: Omni urban scene reconstruction. arXiv preprint 4rXiv:2408.16760 2024.
[8] Z. Chen, M. Ye, S. Xu, T. Cao, and Q Chen. Ppad: Iterative interactions of prediction and planning for end-to-end autonomous driving. In European Conference on Computer Vision, pages 239 239-256. Springer, 2024.
[9] Z. Chen, Z. Yu, J. Li, L. You, and X. Tan. Dualat: Dual attention transformer for end-to-end autonomous driving. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 16353-16359, IEEE, 2024.
[10] K. Chitta, A. Prakash, and A. Geiger. Neat: Neural attention fields for end-to-end autonomous driving. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 15793-15803, 2021.
[11] Codevilla, M. Müller, A. López, Koltun, and A. Dosovitskiy. End-to-end driving via conditional imitation learning. In 2018 IEEE International Conference Robotics and Automation (ICRA), pages 4693-4700, 2018. doi: LLLLL//ICRAA2211.86664000
[12] B. Coors, A. P. Condurache, and A. Geiger. Nova: Learning see in novel viewpoints and domains. In 2019 International Conference on 3D Vision (3DV), pages 116-125. IEEE, 2019.
[13] T. Do, K. Vuong, S. Roumeliotis, and S. Park. Surface normal estimation of tilted images via spatial rectifier. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, pages 265-280. Springer, 2020.
[14] Doll, N. Hanselmann, L. Schneider, R. Schulz, M. Cordts, M. Enzweiler, and H. Lensch. Dualad: Disentangling the dynamic and static world for end-to-end driving. In Proceedings of the IEEF/CV Conference on Computer Vision and Pattern Recognition, pages 14728-14737, 2024.
11

---

[15] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun. Carla: An open urban driving simulator. In Conference on robot learning, pages 1-16. PMLR, 2017.
[16] Y. Duan, Q. Zhang, and R. Xu. Prompting multi-modal tokens to enhance end-to-end au- tonomous driving imitation learning with Ilms. In 2024 IEEE International Conference Robotics and Automation (ICRA), pages
IEEE, 2024.
[17] K. Feng, C. Li, D. Ren, Y. Yuan, and G. Wang. On the road to portability: Compressing end-to- end motion planner for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15099-15108, 2024.
[18] R. Gao, K. Chen, Z. Li, L. Hong, Z. Li, and Q Xu. Magicdrive3d: Controllable 3d generation for any-view rendering in street scenes. arXiv preprint arXiv:2405.14477 2024.
[19] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[20] S. Hu, L. Chen, Wu, H. Li, J. Yan, and D. Tao. St-p3: driving via spatial-temporal feature learning. In European Conference Computer Vision, pages 533-549. Springer, 2022.
End-to-end vision-bases autonomous
[21] Y. Hu, J. Yang, L. Chen, K. Li, C. Sima, X. Zhu, Chahi,, S. Du, T. Lin, W. Wang, et al. Planning- oriented autonomous driving. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17853-17862, 2023.
[22] N. Huang, X. Wei, W. Zheng, P. An, M. Lu, W. Zhan, M. Tomizuka, K. Keutzer, and Zhang. S3gaussian: Self-supervised street gaussians for autonomous driving. arXiv preprint arXiv:2405.203: 2024.
[23] B. Jaeger, K. Chitta, and A. Geiger. Hidden biases of end-to-end driving models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2240-8249, 2023.
[24] X. Jia, Y. Gao, L. Chen, J. Yan, P. L. Liu, and H. Li. Driveadapter: Breaking the coupling barrier of perception and planning end-to-end autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7953-7963, 2023.
[25] X. Jia, P. Wu, L. Chen, J. Xie, C. He, J. Yan, and H. Li. Think twice before driving: To- wards scalable decoders for end-to-end autonomous driving. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 21983-21994, 2023.
[26] X. Jia, J. You, Z. Zhang, and J. Yan. Drivetransformer: Unified transformer for scalable end-to-end autonomous driving. arXiv preprint arXiv:2503.07653 2025.
[27] B. Jiang, S. Chen, Q. Xu, B. Liao, J. Chen, H. Zhou, Q. Zhang, W. Liu, C. Huang, and X. Wang. Vad: Vectorized scene representation for efficient autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340-8350, 2023.
[28] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis. 3d gaussian splatting for real-time real-time radiance field rendering. ACM Trans. Graph., 42(4):139 2023.
[29] M. Khan, Fazlali, D. Sharma, T. Cao, D. Bai, Y. Ren, and B. Liu. Autosplat: Constrained gaus- sian splatting for autonomous driving scene reconstruction. arXiv preprint 200.:2240,.02598, 2024.
[30] T. Klinghoffer, J. Philion, W. Chen, O. Litany, Z. Gojeic, J. Joo, R. Raskar, S. Fidler, and J. M. Alvarez. Towards viewpoint robustness in bird's eye view segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8515-8524, 2023.
[31] P. Li and D. Cui. iavigation-guider sparse scene representation for end-to-end autonomous driving. In The Thirteenth International Conference on Learning Representations.
[32] Y. Y. Fan, He, Y. Wang, Y. Chen, Zhang, and T. Tan. Enhancing end-to-end autonomous driving with latent world model. arXiv preprint
2024.
12

---

[33] Z. Li, Z. Yu, S. Lan, J. Li, Kautz, T. Lu, and J. M. Alvarez. Is ego status all you need for open-loop end-to-end autonomous driving? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14864-148735 2024.
[34] M. Liang, B. Yang, R. Hu, Y. Chen, R. Liao, S. Feng, and R. Urtasun. Learning lane graph representations for motion forecasting. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pages 541-556. Springer, 2020.
[35] B. Liao, S. Chen, H. Yin, B. Jian, C. Wang, S. Yan, X. Zhang, X. Li, Y. Zhang, Q. Zhang, et al. Diffusiondrive: Truncated diffusion model for end-to-end autonomous driving. arXiv preprint arXiv:2411.15139, 2024.
[36] J. Lin, 2. Li, X. Tang, J. Liu, S. Liu, J. Liu, Y. Lu, X. Wu, S. Xu, Y. Yan, et al. Vastgaussian: Vast 3d gaussians for large scene reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5166-5175, 2024.
[37] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
[38] X. Lin, T. Lin, Z. Pei, L. Huang, and Su. Sparse4d: Multi-view 3d object detection with sparse spatial-tempon fusion. arXiv preprint arXiv:221 I. 1.1-8/1, 2022.
[39] X. Lin, Z. Pei, T. Lin, L. Huang, and Z. Su. Sparse4d v3: Advancing end-to-end 3d detection and tracking. arXiv preprint 1r.1:2231..1172. 2023.
[40] S. Madan, T. Henry, J. Dozier, H. Ho, N. Bhandari, T. Sasaki, Durand, H. Pfister, and X. Boix. When and how cnns generalize to outof-distribution aategory-viewpolie combinations. arXiv preprint arXiv:2007.0803
2021.
[41] S. Madan, T. Sasaki, T.-M. Li, X. Boix, and H. Pfister. Small in-distribution changes in 3d perspective and lighting fool both cnns and transformers. arXiv preprint arXiv:2106. 16198, 3, 2021.
[42] J. Mao, Y. Qian, J. Ye, H. Zhao, and Y. Wang. Gpt-driver: Learning to drive with gpt. arXiv preprint arXiv:2310.01415, 2023.
[43] B. Mildenhall, P. Srinivasan, M. Tancik, T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[44] C. Pan, B. Yaman, T. Nesti, A. Mallik, A. G. Allievi, S. Velipasalar, and L. Ren. Vlp: Vision language planning for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14760-14769, 2024.
[45] J. Philion and S. Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pages 194-210. Springer, 2020.
[46] A. Prakash, K. Chitta, and A. Geiger. Multi-modal fusion transformer for end-to-end au- tonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7077-7087, 2021.
[47] H. Shao, L. Wang, R. Chen, S. L. Waslander, H. Li, and Y. Liu. Reasonnet: End-to-end driving with temporal and global reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13723-13733, 2023.
[48] H. Shao, Y. Hu. L. Wang, G. Song, S. L. Waslander, Y. Liu, and H. Li. Lmdrive: Closed-loop end-to-end driving with large language models. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 15 15120-15130, 2024.
[49] C. Sima, K. Renz, K. Chitta, L. Chen, H. Zhang, C. Xie, J. BeiBwenger, P. Luo, A. Geiger, and H. Li. Drivelm: Driving with graph visual question answering. In European Conference on Computer Vision, pages 256-274. Springer, 2024.
13

---

[50] Z. Song, C. Jia, L. Liu, H. Pan, Y. Zhang, J. Wang, X. Zhang, S. Xu, L. Yang, and Y. Luo. Don't t shake the wheel: Momentum-awwe aware planning end-to-end autonomous driving. arXiv preprint 2.Xiv:5503.0312: 2025.
[51] W. Sun, X. Lin, Y. Shi, C. Zhang, H. Wu, and S. Zheng. Sparsedrive: End-to-end autonomous driving via sparse scene representation. arXiv preprini arXiv:2405. 19620, 2024.
[52] S. Szymanowiez, C. Rupprecht, and A. Vedaldi. Splatter image: Ultra-fatt single-view 3d reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10208-10217, 2024.
[53] Q. Tian, Tan, Y. Xie, and Ma. Drivingforward: ferd-forward 3d gaussian splatting for driving scene reconstruction from flexible surround-view input. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 7374-7382, 2025.
[54] W. Tong, C. Sima, Wang, L. Chen, S. Wu, H. Deng, Y. Gu, L. Lu, P. Luo, D. Lin, et al. Scene occupancy. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8406-8415, 2023.
[55] M. Toromanoff, E. Wirbel, and Moutarde. End-to-end model-free reinforcement learning for urban driving using implicit affordances. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[56] H. Turki, V. Agrawal, S. R. Buld, L. Porzi, P. Kontschieder, D. Ramanan, M. Zollhöfer, and C. Richardt. Hybridnerf: Efficient neural rendering via adaptive volumetric surfaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19647-19656, 2024.
[57] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, E. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[58] I. Wang. S. W. Kim, J. Yang, C. Yu, B. Ivanovic, S. Waslander, Y. Wang. S. Fidler, M. Pavone, and P. Karkus. Distillnerf: Perceiving 3d scenes from single-glance images by distilling neural fields and foundation model features. Advances in Neural Information Processing Systems, 37: 62334-62361, 2024.
[59] Y. Wang, V. C. Guizilini, T. Zhang, Y. Wang, H. Zhao, and J. Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In Conference on Robot Learning. pages 180-191. PMLR, 2022.
[60] X. Weng, B. Ivanovic, Y. Wang, Y. Wang, and M. Pavone. Para-drive: Parallelized architecture for real-time autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15449-15458, 2024.
[61] G. Wu, T. Yi, J. Fang, L. Xie, X. splatting for real-time dynamic scene rendering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 20310-20320, 2024.
Zhang, W. Wei, W. Liu, Q. Tian, and X. Wang. 4d gaussian
[62] P. Wu, X. Jia, L. Chen, J. Yan, H. Li, and Y. Qiao. jnjectory-guider control prediction for end-to-end autonomous driving: simple yet strong baseline. Advances in Neural Information Processing Systems, 35:61 19119-66 6132, 2022.
[63] Z. Xing, X. Zhang, Y. Hu, B. Jiang, T. He, Q. Zhang, X. Long, and W. Yin. Goalflow: Goal- driven flow matching for multimodal trajectories generation in end-to-end autonomous driving. arXiv preprint arXiv:2503.0 055689 2025.
[64] Y. Xu, Y. Hu, Z. Zhang, G. P. Meyer, S. K. Mustikovela, S. Srinivasa, E. M. Wolff, and X. Huang. Vlm-ad: End-to-end autonomous driving through vision-language model supervision. arXiv 2024.
preprint 242
[65] Y. Yan, H. Lin, C. Zhou, W. Wang, H. Sun, K. Zhan, X. Lang, X. Zhou, and S. Peng. Street gaussians: Modeling dynamic urban scenes with gaussian splatting. In European Conference on Computer Vision, pages 156-173. Springer, 2024.
14

---

[66] J. Zhai, Z. Feng. J. Du, Y. Mao. J.-J. Liu, Z. Tan, Y. Zhang, X. Ye, and J. Wang. Rethink- ing the open-loop evaluation end-to-end autonomous driving in nuscenes. arXiv preprint arXiv:2305. 10430, 2023.
[67] B. Zhang, N. Song, X. Jin, and L. Zhang. Bridging past and future: End-to-end autonomous driving with historical prediction and planning. arXiv preprint arXiv:2503. 14182, 2025.
[68] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features a perceptual metric. In Proceedings the IEEE conference computer vision and pattern recognition, pages 586-595, 2018.
[69] Y. Zhang, D. Qian, D. Li, Y. Pan, Y. Chen, Z. Liang, Z. Zhang, S. Zhang, H. Li, M. Fu, al. Graphad: Interaction scene graph for end-to-end autonomous driving. arXiv preprint arXiv:2403. /9098, 2024.
[70] 7. Zhang, A. Liniger, D. Dai, F. Yu, and I.. Van Gool, End-to-end urban driving by imitating reinforcement learning coach. In Proceedings of the IEEE/CVE international conference on computer vision, pages 15222-15232, 2021.
[71] S. Zheng, B. Zhou, R. Shao, B. Liu, S. Zhang, L. Nie, and Y. Liu. Gps-gaussian: Generalizable pixel-wise 3d gaussian splatting for real-time human novel view synthesis. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 19680-19690, 2024.
[72] W. Zheng, W. Chen, Y. Huang, B. Zhang, Y. Duan, and J. Lu. Occworld: Learning a 3d occupancy world model for autonomous driving. In European conference on computer vision, pages 55-72. Springer, 2024.
[73] W. Zheng, R. Song. X. Guo, C. Zhang, and L. Chen. Genad: Generative end-to-end autonomous driving. In European Conference on Computer Vision, pages 87-104. Springer, 2024.
[74] X. Zhou, Lin, X. Shan, Y. Wang, D. Sun, and M.-H. Yang. Drivinggaussian: Composite gaussian splatting for surrounding dynamic autonomous driving scenes. In Proceedings the JEEE/CVF conference on computer vision and pattern recognition, pages
2024.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly state the claims and contributions in the abstract and the introduction this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We clearly state the limitations of this paper in the conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the experimental settings needed reproduce the experimental results in the main section and supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [No]
Justification: We do not provide the code and dataset at the time of submission, but we will provide open access to the code and dataset after acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /nipss.cc/pulli//uuui guwde//ddee/ee/ Cot//ooodeeesiseeeeeee hwtpbbmmsssiooooooooiooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify all the experimental details, including datasets, and hyperparame-
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive, but we report the results on various settings for validation of generalization ability.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
18

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide information related to the computer resources used for the experi- ments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We confirm the NeurIPS Code of Ethics and follow it in our paper.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the social impacts of our paper in the broader impact section.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We discuss the license of assets in the Dataset Licenses section.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We discuss the new asset of our paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our research does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
21

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [No]
Justification: LLMs were used only for minor editing and translation purposes, which do not affect the core methodology or scientific contributions.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
22


Reviews:
[{"review_id": "7yyvLvUC2H", "reviewer": "Reviewer_RXRc", "summary": "This paper studies viewpoint robustness for end to end autonomous driving. Specifically, the authors proposes a 3DGS-based 3D reconstruction as auxiliary task and a cross-view consistent and distillation module to enhance the viewpoint robustness. Experimental results on the nuScenes open-loop benchmark demonstrate the effectiveness of the proposed method in viewpoint robust end to end planning.", "questions": "1. Studies [1,2] demonstrate that leveraging solely the ego-vehicle's historical states can yield strong performance on the nuScenes open-loop benchmark, exhibiting robustness to viewpoint variations. To validate whether VR-Drive genuinely learns cross-view robustness rather than relying excessively on past ego-state features, could the authors include comparative experimental results for the methods presented in [1,2]?\n2. Given the NAVSIM dataset's greater alignment with real-world driving scenarios and its increased challenge level, would the authors consider evaluating their proposed method on this dataset to further assess its practical applicability and robustness?\n\n\n[1] Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenes\n[2] Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 4, "strengths_and_weaknesses": "### Strength\n1. The authors pioneeringly address viewpoint robustness in end-to-end autonomous driving, a critical step toward enhancing the generalization capability of end-to-end driving systems.\n2. Leveraging a 3D reconstruction pipeline, the authors introduce an auxiliary reconstruction loss alongside cross-view consistency and distillation techniques, which effectively boost the model's robustness.\n\n### Weakness\n1. The paper's clarity, particularly regarding the network pipeline, requires improvement. The authors should elaborate on input/output specifications, the nature of features used in viewpoint distillation and the memory bank, and the integration of camera parameters into the network. These details would significantly aid readers in comprehending the proposed methodology.\n\n2. The current evaluation of viewpoint robustness relies on synthetic images from the nuScenes open-loop benchmark, which may not fully reflect real-world performance. The authors also could consider validating their approach on the NAVSIM test set, which better aligns with real-world driving scenarios.\n\n3. The proposed viewpoint generalization module is only evaluated on the SparseDrive. Given its potential applicability, the authors should evaluate the module across additional end-to-end methods to demonstrate its broader utility and generalization capacity.", "was_revised": true, "final_justification": "The author's responses address most of my concerns.", "rebuttal": "We sincerely thank the reviewer for their insightful and constructive comments. We deeply appreciate the time and effort you have dedicated to our work, and we are pleased that you recognize the quality, clarity, and originality of our approach. In this response, we thoroughly address all the questions raised and hope that our explanations effectively resolve any concerns. We are grateful for the opportunity to refine our work based on your valuable feedback and look forward to your thoughts on the revisions.\n\n---\n\n> **[Comment 1] The authors should provide more details on input/output specifications, viewpoint distillation features, memory bank, and camera parameter integration.**\n\n**Response:** We sincerely appreciate the reviewer’s valuable comments. In response to the feedback, we intend to include additional details to enhance clarity as follows:\n\n- **Input/output specifications:** The inputs consist of the original multi-view images, and the final output is the ego-vehicle's motion planning result. The intermediate outputs are generated from the modules and include novel view images, detection, and mapping results. We will also clarify this part by adding appropriate marks to the figure to make it more explicit.\n\n- **The nature of features used in viewpoint distillation and the memory bank:** The features used in the memory bank are the instance features paired with anchors, based on the anchor-instance feature approach adopted in our pipeline. In Viewpoint-Consistent Distillation, the sampled points are generated from instance features, while the features used for distillation are 2D image feature maps extracted from the camera's multi-view images.\n\n- **The integration of camera parameters into the network:** Camera parameters are used during 3D Gaussian Splatting (3DGS), perception, and viewpoint-consistent distillation. They are primarily used when aligning between 2D and 3D.\n\n\n> **[Comment 2]  The current evaluation of viewpoint robustness uses synthetic images from the nuScenes open-loop benchmark, which may not fully represent real-world performance. The authors may consider validating their approach on the NAVSIM test set, which better reflects real-world scenarios. [Q2] Would the authors consider evaluating their proposed method on this dataset to further assess its practical applicability and robustness?**\n\n**Response:**  We sincerely appreciate the reviewer's insightful comments. While we considered validating our approach on the NAVSIM test set, we found that NAVSIM, like the nuScenes benchmark, does not allow for the flexible manipulation of viewpoints needed for a thorough evaluation of viewpoint robustness. Therefore, in response to the reviewer's suggestion, we decided to use the CARLA simulator, which allows for a broader range of viewpoints and better aligns with real-world driving scenarios, including closed-loop evaluation. \n\nTo facilitate a comprehensive evaluation, including comparisons with several established end-to-end autonomous driving models and analyses across five diverse unseen viewpoints, we construct a novel-view benchmark by sampling 20\\% of sequences from Town05-Short. Training data is collected using the autopilot, as in Transfuser, but only from original viewpoints. \nThe proposed VR-Drive in CARLA shares the same setting as that used in nuScenes, except that we additionally incorporate an MLP-based traffic light classification branch, following the previous work VAD, to recognize traffic signals. \nWe use the ID-1 setting from Table 2 in the main paper (with all proposed modules removed) as the baseline. Table A provides a summary of the closed-loop evaluation, presenting a comparison between our method, the baseline, and prior works such as ST-P3, TCP, AD-MLP, and BEV-Planner. Existing end-to-end autonomous driving models show considerable performance degradation under unseen viewpoints, whereas our method remains consistently robust.\n\n**Table A: Closed-loop test on CARLA Town05 Short dataset. AD-MLP is the method that relies solely on the ego-vehicle’s historical states, showing consistent performance regardless of the viewpoint.**\n\n|**Methods**|**Original DS**|**Original RC**|**pitch +5° DS**|**pitch +5° RC**|**pitch -10° DS**|**pitch -10° RC**|**height +1.0m DS**|**height +1.0m RC**|**height -0.7m DS**|**height -0.7m RC**|**depth +1.0m DS**|**depth +1.0m RC**|**Unseen Avg DS**|**Unseen Avg RC**|\n|--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|\n|ST-P3|44.24|100.00|41.00|100.00|23.85|100.00|25.83|100.00|28.60|100.00|32.06|100.00|30.27|100.00|\n|TCP|92.73|92.73|70.33|80.33|4.65|4.65|88.51|88.51|0.00|0.00|91.11|91.11|50.92|52.92|\n|AD-MLP|13.59|32.83|13.59|32.83|13.59|32.83|13.59|32.83|13.59|32.83|13.59|32.83|13.59|32.83|\n|BEV-Planner|17.25|28.70|7.30|28.89|7.74|28.83|8.51|28.95|7.69|28.70|7.75|28.95|7.80|28.86|\n|Baseline|76.47|99.20|69.41|89.60|45.65|99.38|48.67|100.00|41.59|86.76|35.95|98.06|48.25|94.87|\n|**VR-Drive (Ours)**|84.04|99.04|75.00|100.00|91.26|98.76|98.44|98.99|80.67|97.32|95.88|96.35|88.25|98.28|\n\n> **[Comment 3] The proposed viewpoint generalization module is only evaluated on the SparseDrive. The authors should evaluate the module across additional end-to-end methods.**\n\n**Response:** Our proposed framework was initially evaluated by applying it to the module in SparseDrive, but we agree that demonstrating its broader applicability is important. Therefore, we have applied and validated our framework on additional end-to-end autonomous driving models, VAD and DiffusionDrive, and present the results in Table B below. The proposed approach can be applied to various baselines, demonstrating its scalability.\n\n**Table B: Performance of our method applied to various end-to-end autonomous driving models.**\n|Camera Setting|Methods|L2(1s)|L2(2s)|L2(3s)|L2(Avg)|Col(1s)|Col(2s)|Col(3s)|Col(Avg)|\n|-------------|---------------------------|------|------|------|-------|-------|-------|-------|--------|\n|Original|VAD|0.41|0.70|1.05|0.72|0.07|0.17|0.41|0.22|\n||VAD + VR-Drive|0.36|0.69|1.12|0.72|0.04|0.18|0.38|0.20|\n||DiffusionDrive|0.27|0.54|0.90|0.57|0.03|0.05|0.16|0.08|\n||DiffusionDrive + VR-Drive|0.35|0.67|1.08|0.70|0.03|0.06|0.19|0.09|\n|Unseen Averge|VAD|0.46|0.75|1.11|0.78|0.20|0.37|0.69|0.42|\n||VAD + VR-Drive|0.38|0.72|1.17|0.76|0.04|0.17|0.49|0.24|\n||DiffusionDrive|0.62|1.14|1.76|1.17|0.07|0.36|0.80|0.41|\n||DiffusionDrive + VR-Drive|0.34|0.66|1.09|0.70|0.02|0.11|0.30|0.14|\n\n> **[Q1] Studies [1,2] show that using only the ego-vehicle's historical states yields strong performance on the nuScenes open-loop benchmark, with robustness to viewpoint variations. To confirm if VR-Drive learns cross-view robustness instead of relying on past ego-state features, could the authors include results comparing the methods in [1,2]?**\n\n**Response:** Following the comment, we include the performance of AD-MLP and BEV-Planner in the nuScenes open-loop benchmark (Table C) and the CARLA closed-loop benchmark (Table A). As noted in Table C, AD-MLP, relying solely on historical states, achieves identical performance, while BEV-Planner, which incorporates images, does not guarantee viewpoint robustness, resulting in performance degradation. As the reviewer pointed out, AD-MLP achieves strong performance in open-loop evaluation; however, it shows a limited ability to capture interactions with surrounding agents, as evidenced by its high collision rates. Furthermore, the closed-loop results (Table A) reveal that methods operating without visual context perform significantly worse, highlighting that relying solely on historical states is inadequate for robust planning and viewpoint-aware driving.\n\nIn contrast, VR-Drive does not use the ground truth ego status (e.g., control, speed, etc.) from the previous timestep during inference. Instead, it iteratively estimates the ego status using sensor information. Therefore, our viewpoint-robust performance is entirely derived from the proposed methodology, including joint learning with 3D Gaussian splatting. We would like to highlight that the proposed VR-Drive demonstrates strong performance in both open-loop and closed-loop settings, validating its effectiveness across diverse camera viewpoint scenarios.\n\n**Table C: Open-loop planning performance in the nuScenes dataset. * indicates the results reproduced using the provided code, as the author did not provide the pretrained checkpoints.**\n|Camera Setting|Methods|L2(1s)|L2(2s)|L2(3s)|L2(Avg)|Col(1s)|Col(2s)|Col(3s)|Col(Avg)|\n|--|--|--|--|--|--|--|--|--|--|\n|Original|AD-MLP|0.20|0.26|0.41|0.29|0.17|0.18|0.24|0.19|\n||BEV-Planner*|0.28|0.52|0.84|0.55|0.13|0.17|0.36|0.22|\n||VAD|0.41|0.70|1.05|0.72|0.07|0.17|0.41|0.22|\n||SparseDrive|0.29|0.58|0.96|0.61|0.01|0.05|0.18|0.08|\n||DiffusionDrive|0.27|0.54|0.90|0.57|0.03|0.05|0.16|0.08|\n||VR-Drive (Ours)|0.29|0.57|0.95|0.60|0.01|0.03|0.14|0.06|\n|Unseen Averge|AD-MLP|0.20|0.26|0.41|0.29|0.17|0.18|0.24|0.19|\n||BEV-Planner*|0.28|0.54|0.88|0.57|0.36|0.42|0.63|0.47|\n||VAD|0.46|0.75|1.11|0.78|0.20|0.37|0.69|0.42|\n||SparseDrive|0.47|0.91|1.47|0.95|0.04|0.23|0.65|0.31|\n||DiffusionDrive|0.62|1.14|1.76|1.17|0.07|0.36|0.80|0.41|\n||VR-Drive (Ours)|0.34|0.65|1.06|0.68|0.01|0.07|0.24|0.11|"}, {"review_id": "m2ECfENwOk", "reviewer": "Reviewer_FfNk", "summary": "The authors propose VR-Drive, a method for end-to-end autonomous driving that learns 3D scene reconstructions via Gaussian splatting as an auxiliary task. This enables the method to better generalize to new viewpoints, e.g. when deploying the method on vehicles with moderately different camera setups. Furthermore, the authors plan to publish a new dataset for this task.", "questions": "- Does your method generalize to another dataset?\n- Does your method generalize to other e2e-models, e.g. can you improve VAD [2] with the auxiliary reconstruction target?\n- Does your method improve closed-loop metrics for motion planning as well?\n\n[2] Jiang, B., Chen, S., Xu, Q., Liao, B., Chen, J., Zhou, H., Zhang, Q., Liu, W., Huang, C. and Wang, X., 2023. Vad: Vectorized scene representation for efficient autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 8340-8350).", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The method is well motivated since end-to-end driving models that generalize across different camera setups are important in many applications in the automotive industry.\n- The proposed method significantly improves the generalization to unseen camera setups compared to related methods without this auxiliary objective (cf. Table 1).\n- The authors perform multiple experiments to ablate their design choices.\n- The paper is well structured and well written. \n\nWeaknesses:\n- The proposed method is only evaluated on one dataset, an additional evaluation based-on the Bench2Drive [1] dataset would improve the paper. (it should be possible to change viewpoints in CARLA as well)\n- The evaluation is based on open-loop planning metrics, additional experiments that show the effect on closed-loop metrics would improve the work.\n\n\n[1] Jia, X., Yang, Z., Li, Q., Zhang, Z. and Yan, J., 2024. Bench2drive: Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving. arXiv preprint arXiv:2406.03877.", "was_revised": true, "final_justification": "The authors have resolved all weaknesses, which I identified. Specifically, they evaluated their method on another closed-loop dataset and showed its generalizability to other e2e-driving models. Therefore, I'm increasing my score to accept.", "rebuttal": "Thank you for your thoughtful and constructive feedback. We sincerely appreciate the time and effort you dedicated to reviewing our work. Below, we detail how we have addressed your comments. We are grateful for the opportunity to refine our work based on your insights and look forward to your thoughts on our responses.\n\n\n---\n\n> **[Comment 1] The proposed method is only evaluated on one dataset, an additional evaluation based-on the Bench2Drive [1] dataset would improve the paper. (it should be possible to change viewpoints in CARLA as well)**\n\n**Response:** We sincerely appreciate the reviewer’s insightful comments. In response, we conduct additional evaluations using the CARLA simulator, specifically on the Town05 scenario, which has also been widely adopted in prior works. While the Bench2Drive dataset is a valuable testbed, we chose Town05 because it enables efficient evaluation of multiple baseline models under diverse viewpoint configurations, without incurring the significant computational burden of conducting closed-loop evaluation for all baselines under multiple camera viewpoint settings.\n\nTo this end, we construct a novel-view benchmark by sampling 20\\% of sequences from Town05-Short and applying five viewpoint variants. Training data is collected using the autopilot, following Transfuser, but only from the original camera setting. The proposed VR-Drive in CARLA shares the same setting as that used in nuScenes, except that we additionally incorporate an MLP-based traffic light classification branch, following the previous work VAD, to recognize traffic signals. \nWe use the ID-1 setting from Table 2 in the main paper (with all proposed modules removed) as the baseline. Table A provides a summary of the closed-loop evaluation, presenting a comparison between our method, the baseline, and previous work such as ST-P3, TCP, AD-MLP, and BEV-Planner. We observe that all existing approaches exhibit performance degradation under viewpoint changes. Furthermore, by comparing our full model with its ablated variants where the proposed components are removed, we clearly demonstrate the effectiveness of each component. We believe these results provide strong evidence that our method is robust to viewpoint variations and remains effective even under closed-loop evaluation.\n\n\n**Table A: Closed-loop test on CARLA Town05 Short dataset. AD-MLP is the method that relies solely on the ego-vehicle’s historical states, showing consistent performance regardless of the viewpoint.**\n|**Methods**|**Original DS**|**Original RC**|**pitch +5° DS**|**pitch +5° RC**|**pitch -10° DS**|**pitch -10° RC**|**height +1.0m DS**|**height +1.0m RC**|**height -0.7m DS**|**height -0.7m RC**|**depth +1.0m DS**|**depth +1.0m RC**|**Unseen Avg DS**|**Unseen Avg RC**|\n|--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|\n|ST-P3|44.24|100.00|41.00|100.00|23.85|100.00|25.83|100.00|28.60|100.00|32.06|100.00|30.27|100.00|\n|TCP|92.73|92.73|70.33|80.33|4.65|4.65|88.51|88.51|0.00|0.00|91.11|91.11|50.92|52.92|\n|AD-MLP|13.59|32.83|13.59|32.83|13.59|32.83|13.59|32.83|13.59|32.83|13.59|32.83|13.59|32.83|\n|BEV-Planner|17.25|28.70|7.30|28.89|7.74|28.83|8.51|28.95|7.69|28.70|7.75|28.95|7.80|28.86|\n|Baseline|76.47|99.20|69.41|89.60|45.65|99.38|48.67|100.00|41.59|86.76|35.95|98.06|48.25|94.87|\n|**VR-Drive (Ours)**|84.04|99.04|75.00|100.00|91.26|98.76|98.44|98.99|80.67|97.32|95.88|96.35|88.25|98.28|\n\n> **[Comment 2] The evaluation is based on open-loop planning metrics, additional experiments that show the effect on closed-loop metrics would improve the work. \n[Q1] Does your method improve closed-loop metrics for motion planning as well? \n[Q3] Does your method generalize to another dataset?**\n\n**Response:** We sincerely appreciate the reviewer’s comment. As mentioned in our previous response, we have performed closed-loop evaluation based on the CARLA simulator. We have also observed improvements in closed-loop metrics, specifically Driving Score (DS) and Route Completion (RC), which demonstrate the effectiveness of our method. This further validates that our approach is applicable to other datasets as well.\n\n\n> **[Q2] Does your method generalize to other e2e-models, e.g. can you improve VAD [2] with the auxiliary reconstruction target?**\n\n**Response:** We sincerely appreciate the reviewer’s insightful question. While our proposed framework was initially evaluated by applying it to modules from SparseDrive, we agree that demonstrating its broader applicability is essential. Our proposed approach leverages feedforward 3DGS and can be flexibly applied to other models, as long as Gaussian primitives can be generated from images, since it only requires images captured from cameras.\nTo demonstrate this, we have applied and validated our framework on additional end-to-end autonomous driving models, including VAD and DiffusionDrive, and the results are presented in Table B. The proposed modules can be extended to various baseline models, highlighting their generalization capacity and scalability.\n\n**Table B: Performance of our method applied to various end-to-end autonomous driving models.**\n|Camera Setting|Methods|L2(1s)|L2(2s)|L2(3s)|L2(Avg)|Col(1s)|Col(2s)|Col(3s)|Col(Avg)|\n|-------------|---------------------------|------|------|------|-------|-------|-------|-------|--------|\n|Original|VAD|0.41|0.70|1.05|0.72|0.07|0.17|0.41|0.22|\n||VAD + VR-Drive|0.36|0.69|1.12|0.72|0.04|0.18|0.38|0.20|\n||DiffusionDrive|0.27|0.54|0.90|0.57|0.03|0.05|0.16|0.08|\n||DiffusionDrive + VR-Drive|0.35|0.67|1.08|0.70|0.03|0.06|0.19|0.09|\n|Unseen Averge|VAD|0.46|0.75|1.11|0.78|0.20|0.37|0.69|0.42|\n||VAD + VR-Drive|0.38|0.72|1.17|0.76|0.04|0.17|0.49|0.24|\n||DiffusionDrive|0.62|1.14|1.76|1.17|0.07|0.36|0.80|0.41|\n||DiffusionDrive + VR-Drive|0.34|0.66|1.09|0.70|0.02|0.11|0.30|0.14|"}, {"review_id": "mRtCghGDgh", "reviewer": "Reviewer_Kk38", "summary": "This paper proposes a novel learning pipeline to make end-to-end models be view-robust. The learning consists of three parts, original view learning, novel-view learning and perception-planning learning. In the perception-planning learning, viewpoint-mixed memory bank and viewpoint-consistent distillation are inserted to improve viewpoint consistency.", "questions": "* What is the difference between your perception head and symmetric perception in SparseDrive? \n* As the perception head follows sparse paradigm, would your method be also effective on BEV-based perception?\n* At early training stage, when the render results of feed-forward GS are not good enough, will it make the training of downstream perception task unstable?", "limitations": "The evaluation on NuScenes is not very convincing. If provided with close-loop results on Bench2Drive, you can directly validation the view point robustness with setting new camera extrinsic in Carla, which would be more persuasive.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n* Feed-forward 3D GS are introduced in original view learning and novel view learning to benefit scene understanding, and a cyclic reconstruction loss is adopted to enhance the consistency between original view and novel view. \n* Viewpoint consistent distillation module is proposed to make perceptual feature view-robust with a distillation loss.\n\nWeaknesses:\n* The viewpoint mixed memory bank seems similar to instance bank in Sparse4D, making it less innovative.\n* Open-loop results on NuScenes dataset are not very convincing.", "was_revised": true, "final_justification": "The authors provide extensive experiments in rebuttal period, which solve my main concerns, so I raise my score to \"accept\".", "rebuttal": "We sincerely appreciate your thoughtful and constructive feedback, as well as the time and effort you invested in reviewing our work. Your insights have been invaluable in helping us improve the clarity and quality of our submission. We look forward to your further thoughts on our revised manuscript.\n\n---\n\n> **[Comment 1] The viewpoint mixed memory bank seems similar to instance bank in Sparse4D, making it less innovative.**\n\n**Response:**  We sincerely appreciate the reviewer’s comment. While the viewpoint-mixed memory bank may appear similar to the memory structure found in existing works, we would like to emphasize that the originality of our approach lies not in the memory structure itself, but in the overall VR-Drive framework, which focuses on learning from multiple viewpoints. Specifically, in VR-Drive, the perception and planning pipeline randomly receives features from either the original view or a diverse set of novel views during training.\n\nIt is not sufficient for the perception or planning modules to simply receive different views to construct an optimal feature space. We believe that a structure that allows for interaction between features from different views is necessary. This is why we propose the viewpoint-mixed memory bank. The memory bank stores features from multiple viewpoints, especially as the model gradually mixes features from increasingly diverse views during training. This enables the network to consistently learn viewpoint-invariant representations by combining features from multiple viewpoints into a single, consistent canonical feature. Unlike prior methods, which do not propose combining features from different distributions in this way, our approach actively mixes features from diverse viewpoints, enabling more robust generalization to novel views.\n\n> **[Comment 2] Open-loop results on nuScenes dataset are not very convincing.**\n\n**Response:**  We sincerely appreciate the reviewer’s valuable comments. In response, we carried out additional experiments in the CARLA simulator, specifically using the Town05 scenario, which has also been commonly used in previous studies. Although Bench2Drive serves as a useful evaluation benchmark, we selected Town05 as it allows efficient evaluation of multiple baseline models across various viewpoint configurations, while avoiding the heavy computational cost associated with conducting closed-loop evaluations for all baselines under multiple camera settings.\n\nFor this purpose, we built a novel-view benchmark by selecting 20\\% of sequences from Town05-Short and applying five viewpoint variations, as done in the nuScenes setup. Training data was collected using the autopilot, following the Transfuser setup, but only under the original camera configuration. The proposed VR-Drive in CARLA adopts the same configuration as in the nuScenes setting, except that we additionally include an MLP-based traffic light classification branch, following the previous work VAD to support traffic signal recognition.\n\nWe use the ID-1 setting from Table 2 in the main paper (with all proposed modules removed) as the baseline. Table A summarizes the closed-loop evaluation results, comparing our method against the baseline as well as previous methods such as ST-P3, TCP, AD-MLP, and BEV-Planner. All existing methods show a drop in performance when the viewpoint changes. Furthermore, by comparing our full model with ablated versions in which the proposed components are removed, we demonstrate the contribution of the proposed method. These findings suggest that our method is robust to viewpoint variation and remains effective in closed-loop evaluation.\n\n\n**Table A: Closed-loop test on CARLA Town05 Short dataset. AD-MLP is the method that relies solely on the ego-vehicle’s historical states, showing consistent performance regardless of the viewpoint.**\n|**Methods**|**Original DS**|**Original RC**|**pitch +5° DS**|**pitch +5° RC**|**pitch -10° DS**|**pitch -10° RC**|**height +1.0m DS**|**height +1.0m RC**|**height -0.7m DS**|**height -0.7m RC**|**depth +1.0m DS**|**depth +1.0m RC**|**Unseen Avg DS**|**Unseen Avg RC**|\n|--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|\n|ST-P3|44.24|100.00|41.00|100.00|23.85|100.00|25.83|100.00|28.60|100.00|32.06|100.00|30.27|100.00|\n|TCP|92.73|92.73|70.33|80.33|4.65|4.65|88.51|88.51|0.00|0.00|91.11|91.11|50.92|52.92|\n|AD-MLP|13.59|32.83|13.59|32.83|13.59|32.83|13.59|32.83|13.59|32.83|13.59|32.83|13.59|32.83|\n|BEV-Planner|17.25|28.70|7.30|28.89|7.74|28.83|8.51|28.95|7.69|28.70|7.75|28.95|7.80|28.86|\n|Baseline|76.47|99.20|69.41|89.60|45.65|99.38|48.67|100.00|41.59|86.76|35.95|98.06|48.25|94.87|\n|**VR-Drive (Ours)**|84.04|99.04|75.00|100.00|91.26|98.76|98.44|98.99|80.67|97.32|95.88|96.35|88.25|98.28|\n\n\n> **[Q1] What is the difference between your perception head and symmetric perception in SparseDrive?**\n\n**Response:** As mentioned in the L165 of the main paper, our perception head adopts a well-established approach that has proven to be effective in prior works. By directly using this well-formed module, we ensure a solid foundation for our framework, allowing us to focus on other aspects of the system, such as viewpoint generalization.\n\n> **[Q2] As the perception head follows sparse paradigm, would your method be also effective on BEV-based perception?**\n\n**Response:** In this study, although our framework is designed based on the sparse paradigm, the 3DGS model can be directly applied to other models with minimal modifications. Similarly, although originally designed for anchor-based approaches, the proposed Viewpoint-Mixed Memory Bank and Viewpoint-Consistent Distillation methods can be transferred to BEV-based perception models, such as VAD, with modest adjustments. In this case, the position output from the perception module is used as bounding box proposals, and the corresponding agent queries are applied to the instance features. As shown in the table below, we observe a performance improvement even with BEV-based methods.\n\n**Table B: Performance of our method applied to various end-to-end autonomous driving models.**\n|Camera Setting|Methods|L2(1s)|L2(2s)|L2(3s)|L2(Avg)|Col(1s)|Col(2s)|Col(3s)|Col(Avg)|\n|-------------|---------------------------|------|------|------|-------|-------|-------|-------|--------|\n|Original|VAD|0.41|0.70|1.05|0.72|0.07|0.17|0.41|0.22|\n||VAD + VR-Drive|0.36|0.69|1.12|0.72|0.04|0.18|0.38|0.20|\n||DiffusionDrive|0.27|0.54|0.90|0.57|0.03|0.05|0.16|0.08|\n||DiffusionDrive + VR-Drive|0.35|0.67|1.08|0.70|0.03|0.06|0.19|0.09|\n|Unseen Averge|VAD|0.46|0.75|1.11|0.78|0.20|0.37|0.69|0.42|\n||VAD + VR-Drive|0.38|0.72|1.17|0.76|0.04|0.17|0.49|0.24|\n||DiffusionDrive|0.62|1.14|1.76|1.17|0.07|0.36|0.80|0.41|\n||DiffusionDrive + VR-Drive|0.34|0.66|1.09|0.70|0.02|0.11|0.30|0.14|\n\n> **[Q3] At early training stage, when the render results of feed-forward GS are not good enough, will it make the training of downstream perception task unstable?**\n\n**Response:** We sincerely appreciate the reviewer’s insightful question. To address this, we compare the total loss (sum of detection, map, motion, and planning losses) between the setup where only the 3DGS module is pre-trained and our current setup, where all components are trained jointly from the start. \nAs shown in the table below, we observe that during the early training stage, using the pre-trained 3DGS module results in lower total losses compared to the setup without pretraining, particularly when novel-view inputs are randomly passed to the perception-planning modules.\nHowever, as the number of iterations increases, the difference in total loss continues to decrease. Moreover, we notice that, despite this initial instability, the final performance remains consistent. Notably, the feed-forward GS stabilizes before the first epoch, ensuring that the perception task is not adversely affected in the long term.\n\n**Table C: Comparison of the total loss, defined as the sum of detection, map, motion, and planning losses, when using 3DGS modules with and without pre-training, over training iterations.**\n|Iteration|0|5k|10k|15k|20k|25k|30k|35k|40k|\n|---------|--|---|----|----|----|----|----|----|----|\n|w/o pre-trained 3DGS|54.81|37.22|33.42|31.55|29.22|27.60|26.91|26.24|24.91|\n|w/ pre-trained 3DGS|47.86|30.51|27.93|26.48|26.09|25.55|24.66|23.51|24.22|\n|Δ Total Loss|+6.95|+6.71|+5.49|+5.07|+3.13|+2.05|+2.25|+2.73|+0.69|"}, {"review_id": "N888Cv0fVc", "reviewer": "Reviewer_Cctz", "summary": "Summary\nThis paper proposes VR-Drive, an end-to-end autonomous driving (E2E-AD) framework designed to address viewpoint robustness under varying camera configurations (e.g., pitch, height, depth). The key contributions include:\n1.Joint 3D Scene Reconstruction and Planning: Integrates feed-forward 3D Gaussian Splatting (3DGS) as an auxiliary task to synthesize novel views during training, enabling data augmentation without additional annotations.\n2.Viewpoint-Mixed Memory Bank: Facilitates temporal interaction of features across diverse viewpoints via attention mechanisms, improving consistency.\n3.Viewpoint-Consistent Distillation: Distills knowledge from original views to synthesized views using instance-aware sampling, mitigating synthesis artifacts.\n4.New Benchmark: Introduces a modified NuScenes-based dataset with viewpoint variations for evaluating E2E-AD robustness.\nExperiments show VR-Drive outperforms baselines (e.g., DiffusionDrive) in unseen viewpoint scenarios, reducing collision rates by 35% and ADE by 20% on average. The framework demonstrates scalability and real-time performance (322 FPS on RTX 3090) while maintaining low memory usage (70MB/submap).", "questions": "1.Dynamic Scene Adaptation: The paper assumes static scenes. How does VR-Drive handle dynamic objects (e.g., moving vehicles) during novel view synthesis? If untested, can occlusion handling or motion prediction be integrated?\n2.Parameter Sensitivity: How were critical parameters (e.g., distillation confidence threshold τ, memory bank update frequency) chosen? Provide ablation on parameter ranges to justify robustness.\n3.Calibration Dependency: The conclusion mentions sensitivity to calibration errors. Quantify performance degradation under noisy extrinsics (e.g., ±5cm translation/±2° rotation).\n4.Generalization Beyond NuScenes: The benchmark uses NuScenes-derived data. Does VR-Drive generalize to radically different environments (e.g., off-road, night driving) without retraining?\n5.Computational Overhead: While memory per submap is low (70MB), what is the CPU/GPU load for parallel submap optimization in city-scale scenarios (e.g., 100+ submaps)?", "limitations": "Lighting Robustness: No analysis of performance under extreme illumination (e.g., nighttime, glare), critical for real-world deployment.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1.Comprehensive experiments on NuScenes-derived datasets with diverse viewpoint shifts (pitch, height, depth). Ablation studies validate component contributions (e.g., distillation reduces collision rate by 14%).\n2.Clear pipeline description (Fig. 2-3), modular design (memory bank, distillation), and loss formulations (Eq. 1-5).\n\nWeaknesses:\n1.Ambiguities in key parameters (e.g., confidence threshold τ for distillation) and training details (e.g., cyclic reconstruction loss implementation).\n2.Memory bank and distillation strategies share conceptual similarities with multi-view fusion in DETR3D [57], requiring clearer differentiation.", "was_revised": true, "final_justification": "The author's experimental supplement is very sufficient and addresses my concerns, so I increase the score.", "rebuttal": "Thank you for your constructive and valuable feedback! We greatly appreciate the time you spent reviewing our work. Below, we provide details on how we have responded to your feedback. We appreciate the chance to improve our work using your valuable feedback and eagerly await your thoughts on the rebuttal.\n\n---\n> **[Comment 1] Confidence threshold $\\tau$ for distillation and cyclic reconstruction loss implementation.**\n\n**Response:**  We set the confidence threshold $\\tau$ as 0.1. Regarding the cyclic reconstruction loss, as noted in L242 of the main paper, the loss formulation is identical to the existing rendering loss, with the difference being that it is implemented to reconstruct the original image from the rendered image. We will elaborate further on implementation details in the main paper to provide more clarity.\n\n> **[Comment 2] Memory bank and distillation strategies share conceptual similarities with multi-view fusion in DETR3D [57], requiring clearer differentiation.**\n\n**Response:** While we acknowledge that there are conceptual similarities between our approach and multi-view fusion in DETR3D [57], we would like to emphasize the clear distinctions between the two. The similarity lies in the sampling based on offsets, which, as mentioned in L209 of the main paper, was motivated by the need for effective feature alignment between 3D and 2D representations.\n\nHowever, a key distinction lies in how these aligned features are utilized. Our approach incorporates a viewpoint-mixed memory bank and viewpoint-consistent distillation, which are not present in DETR3D. These two methods actively combine features from different viewpoints, ensuring robust generalization to novel views. Unlike DETR3D, which primarily focuses on multi-view fusion for 3D object detection, our method is specifically designed to enhance viewpoint robustness and improve performance across diverse driving scenarios.\n\nWe hope this clarifies the differences between our approach and DETR3D, and we will ensure that these distinctions are more clearly presented in the main paper.\n\n> **[Q1] How does VR-Drive handle dynamic objects during novel view synthesis?**\n\n**Response:**  The feed-forward 3DGS we used allows for novel view synthesis with just a single multi-view image pair, without the need for temporally continuous video or scene-specific constraints. During the inference process, it can generate images of various views using only the current images, unaffected by dynamic objects. However, since the reconstruction loss in the training stage is computed by rendering through adjacent-time views, it may be influenced by moving objects. To mitigate this, we propose the cyclic reconstruction loss, where the original image is reconstructed from the generated novel view at the same time-step. This process enables stable 3D scene learning, and since the loss is applied between images at the same time-step, it is independent of dynamic objects.\n\n> **[Q2] How were critical parameters chosen? Provide ablation on parameter ranges to justify robustness.**\n\n**Response:** We sincerely appreciate the reviewer’s insightful comment. In response to the reviewer’s suggestion, we provide an analysis of the hyper-parameters, including the distillation confidence threshold and the memory bank update frequency, in Tables A and B below.  \n\nThe confidence threshold determines the selection of anchors that receive our proposed Viewpoint-Consistent Distillation. When the threshold is set to 0, distillation occurs for all anchors, which may include incorrect supervision from anchors that do not represent actual objects, leading to degraded performance. On the other hand, if the threshold is set too high, there will be fewer anchors eligible for cross-viewpoint distillation, making it less effective.\n\nWhen the memory bank update interval is increased, the model fails to robustly handle changing view conditions, resulting in significant performance degradation, especially on the original and unseen views.\n\nWe experimentally selected values that yielded the best performance and used them in our final experiments.  \n\n**Table A: Hyper-parameter analysis of the confidence threshold, $\\tau$. In the main paper, we set the $\\tau$ as 0.1.**\n||Original|||||||| Unseen ||||||||\n|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|\n|$\\tau$|L2(1s)|L2(2s)|L2(3s)|L2(avg)|Col(1s)|Col(2s)|Col(3s)|Col(avg)|L2(1s)|L2(2s)|L2(3s)|L2(avg)|Col(1s)|Col(2s)|Col(3s)|Col(avg)|\n|0.0|0.36|0.68|1.10|0.71|0.04|0.08|0.22|0.11|0.40|0.81|1.28|0.84|0.01|0.07|0.38|0.16|\n|0.1|0.29|0.57|0.95|0.60|0.01|0.03|0.14|0.06|0.34|0.65|1.06|0.68|0.01|0.07|0.24|0.11|\n|0.5|0.29|0.58|0.97|0.62|0.02|0.03|0.14|0.06|0.36|0.69|1.13|0.73|0.02|0.10|0.32|0.15|\n|1.0|0.29|0.56|0.93|0.59|0.04|0.06|0.19|0.09|0.33|0.64|1.06|0.68|0.04|0.13|0.31|0.16|\n\n\n**Table B: Analysis of the viewpoint-mixed memory bank update interval. In the main paper, we set the update interval as 1.**\n||Original|||||||| Unseen ||||||||\n|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|\n|Update Interval|L2(1s)|L2(2s)|L2(3s)|L2(avg)|Col(1s)|Col(2s)|Col(3s)|Col(avg)|L2(1s)|L2(2s)|L2(3s)|L2(avg)|Col(1s)|Col(2s)|Col(3s)|Col(avg)|\n|1| 0.29 |0.57|0.95|0.60|0.01|0.03|0.14|0.06|0.34|0.65|1.06|0.68|0.01|0.07|0.24|0.11| \n|2| 0.32 |0.62|1.02|0.65|0.03|0.02|0.13|0.06|0.38|0.71|1.15|0.74|0.07|0.16|0.37|0.20|\n|5| 0.31|0.61|1.00|0.64|0.01|0.05|0.14|0.07|0.37|0.70|1.14|0.74|0.10|0.15|0.37|0.21|\n\n\n> **[Q3] The conclusion mentions sensitivity to calibration errors. Quantify performance degradation under noisy extrinsics (e.g., ±5cm translation / ±2° rotation).**\n\n**Response:** Following the reviewer's comment, we conducted experiments to quantify the performance degradation under noisy extrinsics. Specifically, we introduced perturbations to the extrinsic parameters, including ±5 cm in translation and ±2° in rotation, and evaluated the model’s performance.\n\nAs shown in Table C below, while the model does exhibit some sensitivity to calibration errors, the performance degradation remains within an acceptable range. We observed that the model maintains robust performance even under moderate extrinsic noise, with only slight reductions in accuracy. However, for larger perturbations, performance degradation becomes more noticeable. As mentioned in the future work section, we plan to further investigate the model's robustness to calibration distortions that may occur during real-world driving.\n\n**Table C: Performance with perturbations applied to calibration parameters.**\n||Original|||||||| Unseen ||||||||\n|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|\n|Calibration Errors|L2(1s)|L2(2s)|L2(3s)|L2(avg)|Col(1s)|Col(2s)|Col(3s)|Col(avg)|L2(1s)|L2(2s)|L2(3s)|L2(avg)|Col(1s)|Col(2s)|Col(3s)|Col(avg)|\n|Original|0.29|0.57|0.95|0.60|0.01|0.03|0.14|0.06|0.34|0.65|1.06|0.68|0.01|0.07|0.24|0.11|\n|±5cm|0.29|0.58|0.95|0.61|0.00|0.04|0.14|0.06|0.34|0.65|1.07|0.69|0.03|0.09|0.26|0.13|\n|±20cm|0.29|0.58|0.95|0.61|0.01|0.04|0.14|0.06|0.35|0.66|1.08|0.70|0.02|0.11|0.40|0.18|\n|±2°|0.31|0.61|1.00|0.64|0.01|0.03|0.18|0.08|0.34|0.66|1.10|0.70|0.02|0.10|0.33|0.15|\n|±7°|0.29|0.58|0.95|0.61|0.03|0.04|0.15|0.07|0.38|0.75|1.22|0.78|0.02|0.21|0.60|0.28|\n\n> **[Q4] Does VR-Drive generalize to radically different environments without retraining? No analysis of performance under extreme illumination (e.g., nighttime, glare), critical for real-world deployment.**\n\n**Response:** We sincerely appreciate the reviewer’s comment. To evaluate performance under extreme illumination conditions, we used the light degradation transform proposed in [A] to generate low-light and over-exposure images from the nuScenes test dataset. While our proposed method is not specifically designed for handling extreme light conditions, the nuScenes dataset contains a range of light conditions, and our model has been trained effectively on this diverse data. As shown in the table below, although there is some performance degradation under extreme lighting, it remains within an acceptable range, which is important for real-world deployment, where such conditions are inevitable.\n\n\n[A] Light the night: A multi-condition diffusion framework for unpaired low-light enhancement in autonomous driving, CVPR 2024\n\n**Table D: Generalization performance under extreme illumination conditions in the nuScenes open-loop planning setting. The model is trained solely on original nuScenes images.**\n||Original|||||||| Unseen ||||||||\n|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|\n|Illumination|L2(1s)|L2(2s)|L2(3s)|L2(avg)|Col(1s)|Col(2s)|Col(3s)|Col(avg)|L2(1s)|L2(2s)|L2(3s)|L2(avg)|Col(1s)|Col(2s)|Col(3s)|Col(avg)|\n|Original|0.29|0.57|0.95|0.60|0.01|0.03|0.14|0.06|0.34|0.65|1.06|0.68|0.01|0.07|0.24|0.11|\n|Low|0.30|0.59|0.98|0.63|0.00|0.05|0.17|0.07|0.36|0.70|1.16|0.74|0.01|0.08|0.28|0.12|\n|High|0.29|0.57|0.93|0.60|0.05|0.06|0.15|0.09|0.35|0.68|1.10|0.71|0.02|0.09|0.31|0.14|\n\n> **[Q5] What is the computational overhead for parallel optimization at city scale?**\n\n**Response:** We trained our model efficiently using PyTorch Distributed Data Parallel on a server equipped with an Intel® Xeon® Gold 6226R CPU and four NVIDIA A6000 GPUs. Our approach jointly trains an end-to-end autonomous driving model with an auxiliary novel view synthesis task implemented using 3D Gaussian Splatting.\n Thanks to the proposed novel cyclic reconstruction loss, the joint training converges efficiently, incurring only a minimal increase in training time. On city-scale benchmarks such as nuScenes and CARLA, the full training process takes approximately 32 hours. Importantly, since the auxiliary view rendering task is only used during training, our method incurs no additional inference cost and matches the baseline model in inference efficiency."}]

Author rebuttal:

Meta-review: This paper enhances the viewpoint robustness of E2E-AD algorithms by introducing a 3DGS-based 3D reconstruction as an auxiliary task. A viewpoint-mixed memory bank and viewpoint-consistent distillation are proposed to improve viewpoint consistency.

The paper initially received mixed scores (3× borderline accept, 1× borderline reject). Reviewers noted several strengths:
- The idea of leveraging 3DGS for viewpoint robustness in E2E-AD is pioneering and effective.
- The experimental evaluation is comprehensive.
- The ablation studies clearly validate the contributions of each component.

The main concerns centered on closed-loop evaluation and generality across other E2E-AD models. However, the authors provided a strong rebuttal that successfully addressed these issues. All reviewers were convinced and raised their ratings to accept. The AC concurs with this recommendation.
Decision: Accept (poster)