RoCA: Robust Cross-Domain End-to-End Autonomous Driving

## Abstract

Abstract End-to-end (E2E) autonomous driving has recently emerged as a new paradigm, offering significant potential. However, few studies have looked into the practical challenge of deployment across domains (e.g., cities).
Although several works have incorporated Large Language Models (LLMs) to leverage their open-world knowledge, LLMs do not guarantee cross-domain driving performance and may incur prohibitive retraining costs during domain adaptation.
In this paper, we propose RoCA, a novel framework for robust cross-domain E2E autonomous driving. RoCA formulates the joint probabilistic distribution over the tokens that encode ego and surrounding vehicle information in the E2E pipeline.
Instantiating with a Gaussian process (GP), RoCA learns a set of basis tokens with corresponding trajectories, which span diverse driving scenarios. Then, given any driving scene, it is able to probabilistically infer the future trajectory.
By using RoCA together with a base E2E model in source-domain training, we improve the generalizability of the base model, without requiring extra inference computation.
In addition, RoCA enables robust adaptation on new target domains, significantly outperforming direct finetuning.
We extensively evaluate RoCA on various cross-domain scenarios and show that it achieves strong domain generalization and adaptation performance.

## 1 Introduction

Moving beyond the traditional modular design, where distinct components like perception li2022bevformer; philion2020lift; wang2024mv2dfusion; yin2021center, motion prediction chai2019multipath; liu2021multimodal; ngiam2021scene, and planning sun2023large are often developed and optimized in isolation, the focus in autonomous driving research has recently shifted towards integrated, end-to-end (E2E) systems casas2021mp3; chen2022learning; chitta2021neat; hu2022st; jiang2023vad; wu2022trajectory.
While E2E approaches can potentially provide enhanced overall driving performance thanks to the joint optimization across components, their robustness can be lacking when encountering less frequent scenarios.
An important factor is the lack of diversity in existing large-scale training datasets *e.g.,* caesar2020nuscenes; dosovitskiy2017carla; ettinger2021large, which often fail to capture the full spectrum of driving scenarios.
For instance, datasets like nuScenes caesar2020nuscenes are dominated by simple events, with limited coverage of rare, safety-critical edge cases.
This imbalance is further amplified by standard training protocols, which tend to prioritize performance on frequent scenarios, causing the optimization to under-weigh long-tail events.
As a result, E2E models trained in such a ways have sub-optimal performance when deployed in different domains, such as different cities, lighting environments, camera characteristics, or weather conditions.

Large language models (LLMs) have recently emerged as a potentially powerful avenue to address these challenges, as their extensive world knowledge, gleaned from vast internet-scale data, may enable more effective generalization
to unseen or rare scenarios wei2022chain. Multi-modal variants (MLLMs) further enhance this by integrating visual inputs li2023blip; liu2024visual, enabling a new wave of E2E driving systems that aim for higher interpretability and improved long-tail robustness hegde2025distilling; hwang2024emma; sima2023drivelm; tian2024tokenize; tian2024drivevlm; wang2024omnidrive; wang2023drivemlm. However, this integration introduces its own set of obstacles.
One concern is that
there is no inherent guarantee that these LLM-infused models will generalize across disparate domains without further adaptation. Moreover, retraining these massive models for new domains is often prohibitively expensive, demanding large quantities of specialized instruction-following data. Critically, even with broad world knowledge, the fundamental issue of data imbalance may persist, limiting model reliability in safety-critical long-tail situations if not explicitly addressed during training.

To address these challenges, we propose RoCA (Robust Cross-domain end-to-end
Autonomous driving).
RoCA is an end-to-end autonomous driving framework designed for enhanced robustness and efficient adaptation using only multi-view images. Instead of relying on pre-trained LLMs, RoCA learns a compact yet comprehensive codebook containing basis token embeddings ($\mathbf{b}$) that represent diverse ego and agent states, spanning both source and potentially target data characteristics. Crucially, RoCA leverages this learned codebook within a Gaussian Process (GP) framework.
During inference, given a new scene’s token embedding, the GP probabilistically predicts future ego waypoints and agent motion trajectories by leveraging the correlation between the current embedding and the learned basis embeddings ($\mathbf{b}$) and their associated known trajectories ($\mathbf{w}=g(\mathbf{b})$) for a learned mapping ($g(.)$)
This probabilistic formulation inherently supports generalization, as predictions for novel scenes are informed by their similarity to known embeddings within the diverse codebook. Furthermore, the variance estimated by the GP provides a principled measure of prediction uncertainty. This variance can be used to dynamically weight the training loss, enabling RoCA to automatically assign greater importance to uncertain or difficult predictions, thereby effectively addressing the training imbalance towards common scenarios and improving performance on critical long-tail events.

The RoCA framework typically involves an initial stage to build the codebook and optimize GP parameters using source data, followed by efficient deployment or adaptation using only multi-view images processed through the learned GP component. This architecture also naturally lends itself to extensions for online streaming adaptation and active learning.

Our main contributions are summarized as follows:

- •
We propose RoCA, a novel framework for robust cross-domain end-to-end autonomous driving. Leveraging a Gaussian process (GP) formulation, RoCA captures the joint distribution over ego and agent tokens, which encode their respective future trajectories, enabling probabilistic prediction.
- •
By utilizing our GP to impose regularization on source-domain training, RoCA leads to more robust end-to-end planning performance across domains.
- •
RoCA enables adaptation of the end-to-end model on a new target domain. Apart from standard finetuning, its uncertainty awareness makes it possible to select more useful data in the active learning setup. Furthermore, RoCA also supports online adaptation.
- •
Through extensive evaluation, RoCA demonstrates robust performance across domains, for instance, driving in different cities and under image degradations. Domain adaptation using RoCA is not only more effective, leading to better planning performance, but also more efficient, by using the predictive uncertainty to select more useful data for finetuning.

## 2 Related Work

##### End-to-End Autonomous Driving.

Autonomous driving (AD) systems have evolved from traditional modular architectures—where perception li2022bevformer; philion2020lift; wang2024mv2dfusion; yin2021center, motion prediction chai2019multipath; liu2021multimodal; ngiam2021scene, and planning codevilla2019exploring are developed and optimized independently—to multi-task frameworks that share a common feature extractor with separate task-specific heads chitta2022transfuser; liang2022effective; zhang2022beverse; zeng2019end. More recently, the field has shifted toward end-to-end (E2E) approaches that directly predict future trajectories casas2021mp3; chen2022learning; chitta2021neat; hu2022st; jiang2023vad; wu2022trajectory. Prominent examples include UniAD hu2023planning, which integrates perception and prediction through a query-based design, and VAD jiang2023vad, which improves planning efficiency by using a lightweight vectorized representation in place of UniAD’s dense maps. These models offer improved computational efficiency, streamlined architecture, and holistic optimization, but still face major challenges—most notably, brittleness in handling rare long-tail events and limited generalization across diverse domains. Domain generalization remains difficult due to varying operational contexts such as city layouts, traffic patterns, and weather conditions ettinger2021large. Recent efforts to improve robustness include data augmentation dosovitskiy2017carla, synthetic data generation li2022bevformer, and specialized training strategies liu2021multimodal. However, these methods often remain limited in their ability to address safety-critical, low-frequency edge cases.

##### Cross-Domain Trajectory Planning.

Recent methods address cross-domain generalization of trajectory planning via architectural innovations, training objectives design, and representation normalization. Loh et al.loh2024cross align attention spaces using graph-based Transformers and adversarial learning. Wang et al.wang2023bridging employ domain-invariant objectives and distillation. Meta-learning approaches like Dual-TT huang2025tailored simulate domain shifts for robust feature learning. Ye et al.ye2023improving reduce geometric variance via Frenet-frame normalization, while AdapTrajqian2024adaptraj disentangles causal and domain-specific features. Vision-based methods, such as Wang et al. wang2022domain, leverage privileged information for staged training. These strategies highlight the importance of combining architecture, training, and data techniques, though challenges in rare event handling and scalability persist.

Gaussian Processes (GPs) offer a principled way to model predictive uncertainty williams2006gaussian, making them well-suited for domain adaptation. Recent work leverages GPs to align source and target distributions ge2023gaussian; kim2019unsupervised and guide cross-domain feature alignment in semantic segmentation liang2025distribution. These methods demonstrate GP’s utility in both uncertainty estimation and domain transfer.
However, the use of GP in end-to-end autonomous driving remains underexplored.
In this work, we incorporate a GP-based module into an E2E driving system to improve trajectory prediction and enable adaptive training for robust generalization and long-tail scenario handling—without large pre-trained models.

## 3 Proposed Approach: RoCA

Figure: Figure 1: RoCA framework overview.(^1^1footnotemark: 1) RoCA consists of two components. (1) A base E2E planner (*e.g.,* jiang2023vad; sun2024sparsedrive) extracts the ego and agent tokens from multi-view images for the motion planner to predict future trajectories. (2) Proposed RoCA module, which leverages Gaussian process (GP). In source-domain training, RoCA learns a set of basis tokens from the source domain via reconstructing ego and agent tokens from the basis, supervised by the tokens from the base model (bottom dashed arrow). Its GP-based trajectory regression model predicts trajectories which are supervised by ground-truth waypoints (top dashed arrow). During adaptation, RoCA generates pseudo ground truth to fine-tune the base model on the target domain (purple arrow).
Refer to caption: extracted/6549955/figures/overview.jpg

We present RoCA, a novel, Gaussian process (GP)-based framework for cross-domain end-to-end autonomous driving. By using a set of basis tokens trained to span diverse driving scenarios, our proposed RoCA module probabilistically infers a trajectory for the current input scene. RoCA not only enhances the robustness of the trained E2E model, but also provides adaptation capability on a new domain.

### 3.1 System Overview

Our proposed E2E pipeline consists of a base E2E model and our proposed RoCA module.
The base model, *e.g.,* jiang2023vad; sun2024sparsedrive, typically has two parts: 1) a scene encoder $st(.;\theta_{st})$, which converts the input images into scene features/tokens, and 2) a motion planner $h(.;\theta_{h})$, which consumes these scene tokens to predict trajectory; $\theta_{st}$ and $\theta_{h}$ are learnable parameters.
The RoCA module contains a Gaussian process model, denoted by $g(.;\theta_{g},\kappa)$, where $\kappa(.)$ is the GP kernel function and $\theta_{g}$ denotes the learnable parameters in th e module.(^1^11See seeger2004gaussian for more details on Gaussian processes.) Figure [1](https://arxiv.org/html/2506.10145v2#footnotex2) shows the system diagram.

Base E2E model. The scene encoder extracts features from the input multi-camera images and cross-attends learnable queries/tokens with these features. Specifically, the scene encoder produces ego tokens $\mathrm{e}$, and agent tokens $\mathrm{a}$ (among other possible tokens), which encode key information for the ego vehicle and of the other surrounding vehicles. The motion planner then takes the ego and agent tokens, and predicts the trajectories for both the ego and other vehicles: $\mathrm{p}_{pred},\mathrm{c}_{pred},\mathrm{p}_{pred,a},\mathrm{c}_{pred,a}=h(
\mathrm{e,a};\theta_{h})$, where $\mathrm{p}$ denotes the waypoints and $\mathrm{c}$ denotes the trajectory class (*e.g.,* total number of classes can be 16 trajectory groups for each driving command of turn left, turn right, and go straight.)

RoCA module. We use a Gaussian process in RoCA. The GP contains learned basis tokens, and for each of the token, there is a matching candidate trajectory.
By computing the correlation between $\mathrm{e}$ and $\mathrm{a}$ and the basis tokens via the kernel function $\kappa$, the GP conditionally infers the future ego and agent trajectories, $\mathrm{p}_{\mathrm{w}},\mathrm{c}_{\mathrm{w}},\mathrm{p}_{\mathrm{w,a}},
\mathrm{c}_{\mathrm{w,a}}=g_{ego}(\mathrm{e,a};\theta_{g},\kappa)$.

Following sun2024sparsedrive, we use an anchor-based method to predict trajectories in both the base motion planner and RoCA module. More specifically, the model first classifies the future trajectory into one of the predefined groups: $N_{ego}$ groups for the ego car and $N_{agent}$ groups for other cars, and then, predicts a residual. The final predicted trajectory is obtained by adding the classified anchor trajectory and the predicted residual.

### 3.2 RoCA Module

In this part, we discuss our proposed RoCA module in details.
This module allows to create an informed and diverse codebook of the plausible trajectories based on prior or pre-existing scenarios.
The key advantage is that it can effectively infer trajectories under uncertainty or in new domains by performing a similarity “lookup” to the basis in the codebook.

#### 3.2.1 Basis Tokens and Trajectories

We construct a “codebook" of learnable basis tokens,
$\mathcal{B}=\{\mathbf{B}_{k}=\{\mathrm{b}_{j,k}\}^{C}_{j=1}\}_{k=1}^{N_{code}}$, where $N_{code}$ is the number of basis groups, each representing a certain trajectory pattern, *e.g.,* turn left, turn right, and $C$ is the group size. These basis tokens should learn to span the space of ego and agent tokens, $\mathrm{e}$ and $\mathrm{a}$, encountered across diverse driving scenes.

These basis tokens are designated to bijectively map to a set of plausible, safe driving trajectories, $\{\mathbf{W}_{k}\}^{N_{code}}_{k=1}$. To construct this set of basis trajectories, we first sample $N_{code}\cdot C$ representative trajectories from ground-truth human driving data, *e.g.,* nuScenes caesar2020nuscenes.
They are then clustered into $N_{code}$ groups, such that each group $\mathbf{W}_{k}$ contains $C$ trajectories with similar driving patterns.

In our Gaussian process formulation, each trajectory $\mathrm{w}_{j,k}\in\mathbf{W}_{k}$ is associated with a unique, learnable basis $\mathrm{b}_{j,k}\in\mathbf{B}_{k}$. In other words, during training, each basis token learns the driving scenario that corresponds to its trajectory. With these basis tokens and trajectories, given a new driving scenario encoded by $\mathrm{e}$ and $\mathrm{a}$, we can infer the ego and agent trajectories based on the correlation between the ego/agent tokens and the basis tokens. Within the $N_{code}$ groups, we designate $N_{ego}$ groups to represent distinct ego-car waypoint patterns and $N_{agent}$ groups for various types of agent trajectories.

#### 3.2.2 Reconstructing Ego and Agent Tokens

The basis tokens $\mathcal{B}=\{\mathbf{B}_{k}=\{\mathrm{b}_{j,k}\}^{C}_{j=1}\}_{k=1}^{N_{code}}$ should
capture the manifold of ego and agent tokens across diverse driving scenarios. In order to train them, we derive the first set of losses via reconstruction of the original ego and agent tokens given by the base model from the basis tokens.

Given a driving scenario with ego and agent tokens from the base model, $\mathrm{e}$ and $\mathrm{a}$, we first classify them into the respective basis groups. Specifically, the ego token is classified into one of the $N_{ego}$ groups and agent token into one of the $N_{agent}$ groups.
Let $\mathrm{c}_{\mathrm{e}}$ denote the index of the group assigned to $\mathrm{e}$. This classification is performed based on the kernel distance metric and an MLP operating on distance, *i.e.,* $\text{MLP}(\kappa(\mathrm{e},\mathbf{B}))$ predicts the classification logits for the ego token (similarly for agent).

Let $\mathbf{B}_{\mathrm{c}_{\mathrm{e}}}$ denote the basis tokens in the classified group $\mathrm{c}_{\mathrm{e}}$. The core mechanism for learning the basis $\mathbf{B}_{\mathrm{c}_{\mathrm{e}}}$ is by reconstructing the original ego token $\mathrm{e}$ using $\mathbf{B}_{\mathrm{c}_{\mathrm{e}}}$ based on Gaussian process. The joint distribution of $\mathrm{e}$ and $\mathbf{B}_{\mathrm{c}_{\mathrm{e}}}$ is given by

$$ $p(\mathrm{e},\mathbf{B}_{\mathrm{c}_{\mathrm{e}}})\sim\mathcal{N}\left(\begin{ bmatrix}{\mathrm{e}}\\ \mathbf{B}_{\mathrm{c}_{\mathrm{e}}}\end{bmatrix},\begin{bmatrix}\kappa( \mathrm{e})&\kappa(\mathrm{e},\mathbf{B}_{\mathrm{c}_{\mathrm{e}}})\\ \kappa(\mathrm{e},\mathbf{B}_{\mathrm{c}_{\mathrm{e}}})^{\top}&\kappa(\mathbf{ B}_{\mathrm{c}_{\mathrm{e}}})\end{bmatrix}\right),$ (1) $$

where $p(.)$ denotes probability density function and $\kappa(.,.)$ is the kernel function evaluating pairwise distances among tokens (specifically, we use the RBF kernel).

The predictive mean $\hat{\mathrm{e}}$ (*i.e.,* the reconstruction of $\mathrm{e}$) and predictive variance $\sigma_{\mathrm{e}}^{2}$ are given by

$$ $\displaystyle\hat{\mathrm{e}}$ $\displaystyle=\mathbf{b}_{anchor,c_{e}}+\kappa(\mathrm{e},\mathbf{B}_{\mathrm{ c}_{\mathrm{e}}})\kappa(\mathbf{B}_{\mathrm{c}_{\mathrm{e}}})^{-1}\bar{\mathbf {B}}_{\mathrm{c}_{\mathrm{e}}},$ (2) $\displaystyle\sigma^{2}_{e}$ $\displaystyle=\kappa(\mathrm{e})-\kappa(\mathrm{e},\mathbf{B}_{\mathrm{c}_{ \mathrm{e}}})\kappa(\mathbf{B}_{\mathrm{c}_{\mathrm{e}}})^{-1}\kappa(\mathrm{e },\mathbf{B}_{\mathrm{c}_{\mathrm{e}}})^{\top}+\sigma_{noise}^{2}\mathbb{I},$ $$

where $\mathbf{b}_{anchor,c_{e}}$ is the mean of the tokens in group $\mathrm{c}_{\mathrm{e}}$, $\bar{\mathbf{B}}_{\mathrm{c}_{\mathrm{e}}}$ is the zero-mean version of ${\mathbf{B}_{\mathrm{c}_{\mathrm{e}}}}$, and $\sigma_{noise}^{2}$ is a small, learnable noise variance.

This prediction $\hat{\mathrm{e}}$ serves as an approximation of the original $\mathrm{e}$, reconstructed with the basis tokens. We supervise this reconstruction with the original ego token. Similarly, we applying the same reconstruction process to obtain $\hat{\mathrm{a}}$ and $\sigma_{\mathrm{a}}$ for each agent token $\mathrm{a}$, using their respective classified group of basis tokens $\mathbf{B}_{\mathrm{c}_{\mathrm{a}}}$. The overall
reconstruction loss for training the basis tokens is given by

$$ $\mathcal{L}_{rec}=\frac{1}{\sigma^{2}_{\mathrm{e}}}|\hat{\mathrm{e}}-\mathrm{e }|^{2}-\log(\sigma_{\mathrm{e}})+\frac{1}{\sigma^{2}_{\mathrm{a}}}|\hat{ \mathrm{a}}-\mathrm{a}|^{2}-\log(\sigma_{\mathrm{a}})+||\mathbf{B}_{\mathrm{c} _{\mathrm{a}}}\mathbf{B}_{\mathrm{c}_{\mathrm{a}}}^{\top}-\mathbb{I}||^{2}+|| \mathbf{B}_{\mathrm{c}_{\mathrm{e}}}\mathbf{B}_{\mathrm{c}_{\mathrm{e}}}^{\top }-\mathbb{I}||^{2},$ (3) $$

where the first four terms are based on maximum likelihood under Gaussian assumption and the last two terms encourages orthogonality of the basis. When learning the basis tokens and the parameters in the GP, we treat the original ego and agent tokens $\mathrm{e}$ and $\mathrm{a}$ as fixed targets, *i.e.,* no gradients flow through them.

#### 3.2.3 Trajectory Prediction via Gaussian Process

Similar to the previous part, given the ego and agent tokens from the base model, $e$ and $a$, we first classify them to their respective basis groups, $\mathrm{c}_{\mathrm{e}}$ and $\mathrm{c}_{\mathrm{a}}$.
A GP-based regression then infers the future trajectory based on the correlation between the ego/agent token and the basis tokens. The predicted mean and variance for the ego trajectory, $\hat{\mathrm{p}}_{\mathrm{e}}$ and $\sigma_{\mathrm{e}}$, is given by

$$ $\displaystyle\hat{\mathrm{p}}_{\mathrm{w}}$ $\displaystyle=\mathbf{w}_{anchor,\mathrm{c}_{\mathrm{e}}}+\kappa(\mathrm{e}, \mathbf{B}_{\mathrm{c}_{\mathrm{e}}})\kappa(\mathbf{B}_{\mathrm{c}_{\mathrm{e} }})^{-1}\bar{\mathbf{W}}_{\mathrm{c_{e}}},$ (4) $\displaystyle\sigma^{2}_{w}$ $\displaystyle=\kappa(\mathrm{e})-\kappa(\mathrm{e},\mathbf{B}_{\mathrm{c}_{ \mathrm{e},\mathrm{w}}})\kappa(\mathbf{B}_{\mathrm{c}_{\mathrm{e}}})^{-1} \kappa(\mathrm{e},\mathbf{B}_{\mathrm{c}_{\mathrm{e}}})^{\top}+\sigma_{noise}^ {2}\mathbb{I},$ $$

where $\mathbf{w}_{anchor,c_{e}}$ is the mean of the trajectories in group $\mathrm{c}_{\mathrm{e}}$, $\bar{\mathbf{W}}_{\mathrm{c}_{\mathrm{e}}}$ is the zero-mean version of ${\mathbf{W}_{\mathrm{c}_{\mathrm{e}}}}$, and $\sigma_{noise}^{2}$ is a small, learnable noise variance. The predicted agent trajectory $\hat{\mathrm{p}}_{w,a}$ and variance $\sigma^{2}_{w,a}$ can be obtained in a similar way.

When training in the source domain, we supervise these GP-based trajectory predictions with the ground truth, as follows:

$$ $\displaystyle\mathcal{L}_{sup}=$ $\displaystyle\frac{1}{\sigma^{2}_{\mathrm{w}}}\mathcal{L}_{planning}(\hat{ \mathrm{p}}_{\mathrm{w}},\mathrm{p}_{gt})-\log(\sigma_{\mathrm{w}})-\frac{1}{ \sigma^{2}_{\mathrm{w,a}}}\mathcal{L}_{motion}(\hat{\mathrm{p}}_{\mathrm{w,a}} ,\mathrm{p}_{gt,a})-\log(\sigma_{\mathrm{w,a}})$ (5) $\displaystyle+\mathcal{L}_{class}(\mathrm{c}_{\mathrm{e}},\mathrm{c}_{gt,e})+ \mathcal{L}_{class}(\mathrm{c}_{\mathrm{a}},\mathrm{c}_{gt,a})+\mathcal{L}_{ triplet}(\mathrm{c}_{\mathrm{e}},\mathrm{c}_{\mathrm{p}},\mathrm{c}_{\mathrm{n }})+\mathcal{L}_{triplet}(\mathrm{c}_{\mathrm{a}},\mathrm{c}_{\mathrm{p,a}}, \mathrm{c}_{\mathrm{n,a}})$ $$

where $\mathrm{p}_{gt}$ and $\mathrm{p}_{gt,a}$ are the ground-truth ego and agent trajectories, $\mathrm{c}_{gt}$ and $\mathrm{c}_{\mathrm{gt,a}}$ are ground-truth ego and agent token categories. The predictive trajectory mean and variance are supervised using variance-weighted losses, similar to those used in jiang2023vad; sun2024sparsedrive). $\mathcal{L}_{planning}$ and $\mathcal{L}_{motion}$ denote the waypoint planning and motion tracking losses, as used in sun2024sparsedrive; jiang2023vad.
To further refine the embedding space, we utilize triplet loss schroff2015facenet. For each ego/agent class prediction, we identify three positive classes ($\mathrm{c}_{\mathrm{p}}$), which exhibit similar driving patterns as the ground-truth class (*e.g.,* turn left with slightly different angles), and three negative classes ($\mathrm{c}_{\mathrm{n}}$), which have driving behaviors different from the true class (*e.g.,* turn left vs. turn right). The triplet loss encourages greater separation between distinct driving categories, and tighter clustering among the same class or similar classes.

### 3.3 Training and Adaptation

#### 3.3.1 Training in Source Domain

Pre-training base E2E model. First, we train the base E2E model on the source domain data following standard training procedure, *e.g.,* jiang2023vad; sun2024sparsedrive.

Learning basis tokens and GP parameters. Secondly, we use both $\mathcal{L}_{rec}$ of Eq. [3](https://arxiv.org/html/2506.10145v2#S3.E3) and $\mathcal{L}_{sup}$ of Eq. [5](https://arxiv.org/html/2506.10145v2#S3.E5) to train RoCA. This includes training the basis tokens and other parameters, *e.g.,* MLP parameters, kernel parameters.

Finetuning base E2E model. Finally, given the trained RoCA module, we utilize it to perform regularized finetuning. More specifically, in addition to the standard supervised loss used to train the base model, we additionally use the following loss by treating RoCA as a teacher:

$$ $\displaystyle\mathcal{L}_{gp}=$ $\displaystyle\mathcal{L}_{class}(\mathrm{c}_{pred},\mathrm{c}_{\mathrm{e}})+ \mathcal{L}_{class}(\mathrm{c}_{pred,a},\mathrm{c}_{\mathrm{a}})+\mathcal{L}_{ triplet}(\mathrm{c}_{pred},\mathrm{c}_{\mathrm{p}},\mathrm{c}_{\mathrm{n}})+ \mathcal{L}_{triplet}(\mathrm{c}_{\mathrm{a}},\mathrm{c}_{\mathrm{p,a}}, \mathrm{c}_{\mathrm{n,a}})$ $\displaystyle+\frac{1}{\sigma^{2}_{\mathrm{w}}}\mathcal{L}_{planning}(\mathrm{ p}_{pred},\hat{\mathrm{p}}_{\mathrm{w}})-\log(\sigma_{\mathrm{w}})+\frac{1}{ \sigma^{2}_{\mathrm{w,a}}}\mathcal{L}_{motion}(\mathrm{p}_{pred,a},\hat{ \mathrm{p}}_{\mathrm{w,a}})-\log(\sigma_{\mathrm{w,a}})$ $\displaystyle+D_{KL}(\mathrm{c}_{pred,e}||\mathrm{c}_{\mathrm{e}})+D_{KL}( \mathrm{c}_{pred,a}||\mathrm{c}_{\mathrm{a}}),$ (6) $$

where $\mathrm{p}_{pred}$, $\mathrm{c}_{pred}$, $\mathrm{c}_{pred,a}$, and $\mathrm{c}_{pred,a}$ are the predicted ego and agent trajectory waypoints and classes from the base E2E model, $D_{KL}$ is the KL-divergence. This loss encourages the base model’s prediction to align with the probabilistic prediction by the trained Gaussian process, which improves prediction robustness and regularizes against noise in training data.

#### 3.3.2 Adaptation in Target Domain

In some cases, ground-truth waypoints are available in the target domain, *e.g.,* based on ego status tracking. In such cases, model adaptation is then the same as the final step in source-domain training, where the standard ground-truth supervision on planning is used together with the GP-based regularization from RoCA: $\mathcal{L}_{gp}$ in Eq. [3.3.1](https://arxiv.org/html/2506.10145v2#S3.Ex1).

There are scenarios where ground-truth trajectories are not available. For instance, it is nontrivial to process large-volume driving logs and thus, ground-truth waypoints may not be available right after data is collected in the target domain (while images are usually readily available). As another example, in an online setting, the E2E driving model streams through video frames as the car drives and does not have access to the ground-truth waypoints on the fly.
For unsupervised domain adaptation, as ground-truth labels are not available, we use $\mathcal{L}_{gp}$ to update the base E2E model.

## 4 Experiments

We conduct extensive experiments to evaluate
RoCA on a standard end-to-end driving benchmark and compare it with the latest state-of-the-art (SOTA) methods. Specifically, we consider challenging cross-domain setups to evaluate the robustness and adaptation performance of our proposed approach.

### 4.1 Experimental Setup

Datasets.
We use nuScenes caesar2020nuscenes to evaluate open-loop planning performance.(^2^22nuScenes is under a CC BY-NC-SA 4.0 license.) This dataset consists of 28k total samples in a 22k/6k training/validation split. The objects in each scene are annotated with 3D bounding box, orientation, and speed. nuScenes contains data collected from two cities, Boston and Singapore, which allows us to evaluate cross-domain generalization and adaptation across cities.
Within the validation set, we also consider a “targeted" subset containing 689 samples where the vehicle must make a turn, as established in weng2024drive.
To further evaluate robustness, we apply degraded versions of the validation set using controlled image corruptions such as motion blur and low light.

Evaluation protocol.
We evaluate the predicted trajectory of the ego vehicle for 3 seconds into the future, with 2 waypoints per second. We use two metrics, the L2 error (in meters) between the predicted and ground-truth waypoints, and the collision rate (in %) between the ego-vehicle and the surrounding vehicles.
Since the evaluation protocols in earlier works hu2023planning; jiang2023vad; zhai2023rethinking were not consistent, *e.g.,* using different BEV grid sizes, not handling invalid/noisy frames, we adopt the standardized evaluation proposed by weng2024drive, which provides a fair comparison across methods.

Model details. We consider two recent, representative methods, SparseDrive sun2024sparsedrive and VAD jiang2023vad as the base E2E models.(^3^33VAD is under an Apache-2.0 license and SparseDrive is under an MIT license.) Specifically, we use VAD-T (tiny configuration) and SparseDrive-S (small configuration). Note that our proposed RoCA can be used with any E2E planning model, as long as it provides a tokenized representation. When there are no explicit ego/agent tokens, we can apply RoCA over the queries or tokens that are fed into the planner.

When we implement the codebook in RoCA, for ego tokens, we use 16 groups for each of the driving commands: turn left, turn right, and go straight, resulting in $N_{ego}=48$. We use $N_{agent}=64$ groups to capture various types of agent trajectories. The total number of groups in the codebook is $N_{code}=N_{ego}+N_{agent}=112$ and we set the group size $C=64$.

Figure: Figure 2: tSNE projection of ego/agent tokens with (top) and without (bottom) RoCA. By using our proposed approach, the model has better separability of different trajectory modes (indicated by different colors). In contrast, the baseline SparseDrive shows poor separability, indicating a sensitivity to any perturbations. The analysis is performed on the full nuScenes val set, the Boston and Singapore subsets(left, middle, right).
Refer to caption: extracted/6549955/figures/tSNE-RoC.png

### 4.2 Learned Tokens in GP

Our proposed GP-based formulation in RoCA provides more robust scene representation and trajectory planning. Specifically, as shown in Figure [2](https://arxiv.org/html/2506.10145v2#S4.F2) (top), the learned basis tokens form clearly distinct clusters (as marked by the different colors), with each cluster representing a different trajectory pattern. As such, even in a new domain, RoCA can still robustly parse the driving scenario in its probabilistic framework and infer a proper trajectory. In contrast, SparseDrive in Figure [2](https://arxiv.org/html/2506.10145v2#S4.F2) (bottom) produces mixed pattern across trajectories types, making it fragile when model is deployed in a new domain. For instance, given a token that corresponds to turning left, a slight perturbation to this token (*e.g.,* due to different camera characteristics, lighting, etc.) can result in drastically different driving behavior in the output of the planner.

### 4.3 Standardized Evaluation on nuScenes Validation

Table [1](https://arxiv.org/html/2506.10145v2#S4.T1) summarizes the results on nuScenes validation set, using the standardized evaluation protocol by weng2024drive. We see that our proposed method RoCA significantly improves the baseline model and achieves competitive performance among the latest state of the art. For instance, when using VAD-T as the base model, by imposing RoCA’s training regularization, we reduce the average L2 error by 16% and collision rate by 38%. Note that this improvement solely results from our improved training and does not require any additional computation at inference time.

When using the trained Gaussian process module in RoCA to predict trajectory, we achieve further improved planning performance, for both base models of VAD-T and SparseDrive-S, reducing the collision rate by 54% and 36%, respectively. This mode of operation, however, requires running the GP module along with the base E2E model and thus, incurs extra inference cost.

While the motivation of RoCA is mainly to enhance cross-domain performance, it is able to boost the planner’s performance even when training and testing are performed in the same domain.

**Table 1: Comparison of L2 trajectory error and collision rate comparison on nuScenes validation set caesar2020nuscenes using the standardized evaluation weng2024drive. *Uses the trajectory inferred by the RoCA Gaussian process module.**
| Model | Using Ego Status | Full Val | Targeted Val |  |  |
| --- | --- | --- | --- | --- | --- |
| Avg. L2 (m) $\downarrow$ | Avg. Col. (%)$\downarrow$ | Avg. L2 (m) $\downarrow$ | Avg. Col. (%)$\downarrow$ |  |  |
| UniAD hu2023planning | ✗ | 0.95 | 0.45 | 1.59 | 0.47 |
| PARA-Drive weng2024drive | ✗ | 0.66 | 0.26 | 1.08 | 0.24 |
| TOKEN tian2024tokenize | ✗ | 0.68 | 0.15 | – | – |
| VAD-T jiang2023vad | ✗ | 0.91 | 0.39 | 1.27 | 0.39 |
| w/  RoCA training regularization | ✗ | 0.76 | 0.24 | 0.99 | 0.28 |
| w/  RoCA trajectory prediction* | ✗ | 0.64 | 0.18 | 0.91 | 0.25 |
| AD-MLP zhai2023rethinking | ✓ | 0.66 | 0.28 | 1.13 | 1.40 |
| TOKEN tian2024tokenize | ✓ | 0.64 | 0.13 | – | – |
| PARA-Drive+ weng2024drive | ✓ | 0.59 | 0.19 | 0.70 | 0.24 |
| SparseDrive-S sun2024sparsedrive | ✓ | 0.65 | 0.14 | 0.85 | 0.31 |
| w/  RoCA training regularization | ✓ | 0.63 | 0.13 | 0.77 | 0.28 |
| w/  RoCA trajectory prediction* | ✓ | 0.55 | 0.09 | 0.65 | 0.25 |

### 4.4 Cross-City Evaluation

In this experiment, we transfer models from a source city to a target city to evaluate cross-domain performance.
The base E2E model and our proposed RoCA module are first trained on the source domain.
In the target domain, we evaluate both zero-shot performance (*i.e.,* no adaption) and performance after adaptation.
Specifically, we compare the baseline model, RoCA with source-only training regularization, as well as adaptation approaches with and without ground-truth labels.

Table [2](https://arxiv.org/html/2506.10145v2#S4.T2) summarizes the performance comparison. When performing zero-shot inference in the target city without any adaptation, RoCA has more robust performance as compared to the baseline. For instance, in the case of using VAD-T as the baseline and running Boston-trained models in Singapore, the collision rate of RoCA is less than half of that of VAD-T (0.211% vs. 0.430%).

In the case of adaptation, when ground-truth is available, RoCA significantly outperforms direct finetuning across all metrics.
For example, on the Boston $\rightarrow$ Singapore transfer using SparseDrive, RoCA reduces the L2 error from 0.544 m to 0.492 m and the collision rate from 0.114% to 0.096%.
This setting provides the fairest comparison, as both approaches utilize ground-truth supervision, and RoCA achieves superior results.
Even in the unsupervised setting, RoCA still outperforms direct finetuning in most cases, highlighting its strong domain adaptation capability and the benefit of our probabilistic GP-based modeling.

**Table 2: Cross-city planning performance.**
| Method | Adapt | Boston $\rightarrow$ Singapore | Singapore $\rightarrow$ Boston |  |  |
| --- | --- | --- | --- | --- | --- |
| Avg. L2 (m) $\downarrow$ | Avg. Col.(%)$\downarrow$ | Avg. L2 (m) $\downarrow$ | Avg. Col.(%)$\downarrow$ |  |  |
| VAD-T | ✗ | 1.245 | 0.430 | 1.157 | 0.224 |
| w/ finetuning using GT | ✓ | 1.193 | 0.263 | 1.105 | 0.190 |
| w/ RoCA training regularization | ✗ | 1.178 | 0.211 | 1.006 | 0.198 |
| w/ RoCA unsupervised adaptation | ✓ | 1.025 | 0.192 | 0.951 | 0.172 |
| w/ RoCA adaptation using GT | ✓ | 0.946 | 0.178 | 0.890 | 0.169 |
| SparseDrive-S | ✗ | 0.904 | 0.174 | 1.014 | 0.241 |
| w/ finetuning using GT | ✓ | 0.544 | 0.114 | 0.671 | 0.122 |
| w/ RoCA training regularization | ✗ | 0.791 | 0.154 | 0.886 | 0.156 |
| w/ RoCA unsupervised adaptation | ✓ | 0.711 | 0.132 | 0.688 | 0.115 |
| w/ RoCA adaptation using GT | ✓ | 0.492 | 0.096 | 0.511 | 0.104 |

### 4.5 Active Learning

In the target domain, our goal is to identify the most informative samples for domain adaptation through active learning, thereby reducing both annotation and adaptation costs. To achieve this, we propose using the GP-based predictive variance as a sampling criterion, selecting samples with the highest uncertainty.
We compare our variance-based selection with the baseline method of random sampling, evaluated at 5%, 10%, and 15% sampling rates of the full target training data.

Table [3](https://arxiv.org/html/2506.10145v2#S4.T3) reports cross-city planning accuracy results for transfers between Singapore and Boston after fine-tuning with ground-truth supervision, using SparseDrive-S as the baseline.
Across all sampling rates, selection based on RoCA consistently results in lower trajectory errors and collision rates compared to random selection, on both the full validation and the targeted subsets.
These results demonstrate the effectiveness of our uncertainty-guided sampling in identifying representative samples for efficient domain adaptation.
Furthermore, RoCA consistently outperforms the baseline under both sampling strategies, underscoring its robustness and adaptability in cross-domain scenarios.

**Table 3: Cross-city active learning performance, using 5%, 10%, and 15% target training samples selected randomly or based on predictive variance by RoCA. The base model is SparseDrive-S in this case.**
| Adapt. Method | Sampling | 5% | 10% | 15% |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Avg. L2 (m) $\downarrow$ | Avg. Col.(%)$\downarrow$ | Avg. L2 (m) $\downarrow$ | Avg. Col.(%)$\downarrow$ | Avg. L2 (m) $\downarrow$ | Avg. Col.(%)$\downarrow$ |  |  |
| Singapore $\rightarrow$ Boston |  |  |  |  |  |  |  |
| Direct finetune | random | 0.767 | 0.215 | 0.753 | 0.199 | 0.711 | 0.175 |
| Direct finetune | RoCA | 0.745 | 0.191 | 0.719 | 0.183 | 0.678 | 0.126 |
| RoCA | random | 0.644 | 0.123 | 0.584 | 0.121 | 0.552 | 0.121 |
| RoCA | RoCA | 0.617 | 0.110 | 0.554 | 0.110 | 0.513 | 0.108 |
| Boston $\rightarrow$ Singapore |  |  |  |  |  |  |  |
| Direct finetune | random | 0.891 | 0.198 | 0.839 | 0.201 | 0.823 | 0.185 |
| Direct finetune | RoCA | 0.828 | 0.192 | 0.815 | 0.172 | 0.793 | 0.166 |
| RoCA | random | 0.707 | 0.148 | 0.656 | 0.133 | 0.633 | 0.126 |
| RoCA | RoCA | 0.673 | 0.135 | 0.604 | 0.113 | 0.561 | 0.102 |

### 4.6 Adaption to Image Degradations

We further assess the generalization and adaptation performance when the image quality is compromised, *e.g.,* due to low light and motion blur. As detailed in Table [4](https://arxiv.org/html/2506.10145v2#S4.T4), without adaptation, the baseline SparseDrive-S model exhibits a significant performance drop when faced with these adverse conditions, whereas the model trained with our RoCA regularization exhibits more robust performance.
When adaptation is performed, our proposed RoCA unsupervised adaptation method achieves significant improvements, and is on par with or better than the baseline adaptation performance which trains the model using ground-truth data with the adverse-conditioned images. When we further utilize the ground truth in adaptation, RoCA achieves significantly better planning performance. These results collectively confirm that our approach enables stronger generalizability and more robust adaptation in diverse conditions, enhancing the reliability of E2E planning.

**Table 4: Planning performance under image degradations including low light and motion blur.**
| Model | Adapt | lowlight | motion blur |  |  |
| --- | --- | --- | --- | --- | --- |
| Avg. L2 (m) $\downarrow$ | Avg. Col.(%)$\downarrow$ | Avg. L2 (m) $\downarrow$ | Avg. Col.(%)$\downarrow$ |  |  |
| Full Val |  |  |  |  |  |
| SparseDrive-S | ✗ | 0.577 | 0.145 | 0.729 | 0.369 |
| w/ supervision using GT | ✓ | 0.547 | 0.228 | 0.573 | 0.118 |
| w/ RoCA training regularization | ✗ | 0.564 | 0.129 | 0.671 | 0.208 |
| w/ RoCA unsupervised adaptation | ✓ | 0.531 | 0.098 | 0.589 | 0.146 |
| w/ RoCA supervised adaptation using GT | ✓ | 0.526 | 0.090 | 0.541 | 0.104 |
| Targeted Val |  |  |  |  |  |
| SparseDrive-S | ✗ | 0.712 | 0.325 | 0.832 | 0.389 |
| w/ supervision using GT | ✓ | 0.707 | 0.291 | 0.721 | 0.282 |
| w/ RoCA training regularization | ✗ | 0.703 | 0.228 | 0.766 | 0.248 |
| w/ RoCA unsupervised adaptation | ✓ | 0.626 | 0.188 | 0.733 | 0.203 |
| w/ RoCA supervised adaptation using GT | ✓ | 0.591 | 0.169 | 0.696 | 0.192 |

### 4.7 Qualitative Results

Figure [3](https://arxiv.org/html/2506.10145v2#S4.F3) shows qualitative planning results on turning scenarios. Our RoCA approach generates trajectories that closely align with the ground-truth trajectories. On the other hand, the baseline SparseDrive model produces trajectories that will lead the vehicle into non-drivable areas.

Figure: Figure 3: Visualization of sample planning results. Left (Right) two scenarios are in Boston (Singapore); note that t is right (left) driving in Boston (Singapore). The red car is the ego vehicle. The color gradient indicates the temporal horizon of the trajectory.
Refer to caption: extracted/6549955/figures/qual4.jpg

## 5 Conclusions and Discussions

We present RoCA, a novel framework for robust cross-domain end-to-end autonomous driving. By leveraging a GP formulation, RoCA models the joint distribution over ego and agent trajectories, enabling probabilistic prediction and uncertainty-aware planning. This GP-based regularization enhances source-domain training and significantly improves generalization to unseen domains. RoCA’s key strength lies in its flexible domain adaptation: it supports standard finetuning, uncertainty-guided active learning, and online adaptation, making it well-suited for real-world deployment. Extensive experiments on the nuScenes benchmark show that RoCA achieves state-of-the-art performance, reducing L2 trajectory error, collision rate, and long-tail errors.

## References

- [1]
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.
nuscenes: A multimodal dataset for autonomous driving.
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020.
- [2]
Sergio Casas, Abbas Sadat, and Raquel Urtasun.
Mp3: A unified model to map, perceive, predict and plan.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14403–14412, 2021.
- [3]
Yuning Chai, Benjamin Sapp, Mayank Bansal, and Dragomir Anguelov.
Multipath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction.
arXiv preprint arXiv:1910.05449, 2019.
- [4]
Dian Chen and Philipp Krähenbühl.
Learning from all vehicles.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17222–17231, 2022.
- [5]
Kashyap Chitta, Aditya Prakash, and Andreas Geiger.
Neat: Neural attention fields for end-to-end autonomous driving.
In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15793–15803, 2021.
- [6]
Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger.
Transfuser: Imitation with transformer-based sensor fusion for autonomous driving.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(11):12878–12895, 2022.
- [7]
Felipe Codevilla, Eder Santana, Antonio M López, and Adrien Gaidon.
Exploring the limitations of behavior cloning for autonomous driving.
In Proceedings of the IEEE/CVF international conference on computer vision, pages 9329–9338, 2019.
- [8]
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun.
Carla: An open urban driving simulator.
In Conference on robot learning, pages 1–16. PMLR, 2017.
- [9]
Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles R Qi, Yin Zhou, et al.
Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset.
In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9710–9719, 2021.
- [10]
Pengfei Ge and Yesen Sun.
Gaussian process-based transfer kernel learning for unsupervised domain adaptation.
Mathematics, 11(22):4695, 2023.
- [11]
Deepti Hegde, Rajeev Yasarla, Hong Cai, Shizhong Han, Apratim Bhattacharyya, Shweta Mahajan, Litian Liu, Risheek Garrepalli, Vishal M Patel, and Fatih Porikli.
Distilling multi-modal large language models for autonomous driving.
arXiv preprint arXiv:2501.09757, 2025.
- [12]
Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao.
St-p3: End-to-end vision-based autonomous driving via spatial-temporal feature learning.
In European Conference on Computer Vision, pages 533–549. Springer, 2022.
- [13]
Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al.
Planning-oriented autonomous driving.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17853–17862, 2023.
- [14]
Feilong Huang, Zide Fan, Xiaohe Li, Wenhui Zhang, Pengfei Li, Ying Geng, and Keqing Zhu.
Tailored meta-learning for dual trajectory transformer: advancing generalized trajectory prediction.
Complex & Intelligent Systems, 11(3):174, 2025.
- [15]
Jyh-Jing Hwang, Runsheng Xu, Hubert Lin, Wei-Chih Hung, Jingwei Ji, Kristy Choi, Di Huang, Tong He, Paul Covington, Benjamin Sapp, et al.
Emma: End-to-end multimodal model for autonomous driving.
arXiv preprint arXiv:2410.23262, 2024.
- [16]
Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang.
Vad: Vectorized scene representation for efficient autonomous driving.
In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340–8350, 2023.
- [17]
Minyoung Kim, Pritish Sahu, Behnam Gholami, and Vladimir Pavlovic.
Unsupervised visual domain adaptation: A deep max-margin gaussian process approach.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4380–4390, 2019.
- [18]
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.
Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.
In International conference on machine learning, pages 19730–19742. PMLR, 2023.
- [19]
Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai.
Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers.
In European conference on computer vision, pages 1–18. Springer, 2022.
- [20]
Chen Liang, Weihua Chen, Xin Zhao, Junyan Wang, Lijun Cao, and Junge Zhang.
Distribution optimization under gaussian hypothesis for domain adaptive semantic segmentation.
In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 9280–9290. IEEE, 2025.
- [21]
Xiwen Liang, Yangxin Wu, Jianhua Han, Hang Xu, Chunjing Xu, and Xiaodan Liang.
Effective adaptation in multi-task co-training for unified autonomous driving.
Advances in Neural Information Processing Systems, 35:19645–19658, 2022.
- [22]
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.
Visual instruction tuning.
Advances in neural information processing systems, 36, 2024.
- [23]
Yicheng Liu, Jinghuai Zhang, Liangji Fang, Qinhong Jiang, and Bolei Zhou.
Multimodal motion prediction with stacked transformers.
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7577–7586, 2021.
- [24]
Jia Quan Loh, Xuewen Luo, Fan Ding, Hwa Hui Tew, Junn Yong Loo, Ze Yang Ding, Susilawati Susilawati, and Chee Pin Tan.
Cross-domain transfer learning using attention latent features for multi-agent trajectory prediction.
In 2024 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 3905–3911. IEEE, 2024.
- [25]
Jiquan Ngiam, Benjamin Caine, Vijay Vasudevan, Zhengdong Zhang, Hao-Tien Lewis Chiang, Jeffrey Ling, Rebecca Roelofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, et al.
Scene transformer: A unified architecture for predicting multiple agent trajectories.
arXiv preprint arXiv:2106.08417, 2021.
- [26]
Chenbin Pan, Burhaneddin Yaman, Tommaso Nesti, Abhirup Mallik, Alessandro G Allievi, Senem Velipasalar, and Liu Ren.
Vlp: Vision language planning for autonomous driving.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14760–14769, 2024.
- [27]
Jonah Philion and Sanja Fidler.
Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d.
In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16, pages 194–210. Springer, 2020.
- [28]
Tangwen Qian, Yile Chen, Gao Cong, Yongjun Xu, and Fei Wang.
Adaptraj: A multi-source domain generalization framework for multi-agent trajectory prediction.
In 2024 IEEE 40th International Conference on Data Engineering (ICDE), pages 5048–5060. IEEE, 2024.
- [29]
Florian Schroff, Dmitry Kalenichenko, and James Philbin.
Facenet: A unified embedding for face recognition and clustering.
In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815–823, 2015.
- [30]
Matthias Seeger.
Gaussian processes for machine learning.
International journal of neural systems, 14(02):69–106, 2004.
- [31]
Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Ping Luo, Andreas Geiger, and Hongyang Li.
Drivelm: Driving with graph visual question answering.
arXiv preprint arXiv:2312.14150, 2023.
- [32]
Qiao Sun, Shiduo Zhang, Danjiao Ma, Jingzhe Shi, Derun Li, Simian Luo, Yu Wang, Ningyi Xu, Guangzhi Cao, and Hang Zhao.
Large trajectory models are scalable motion predictors and planners.
arXiv preprint arXiv:2310.19620, 2023.
- [33]
Wenchao Sun, Xuewu Lin, Yining Shi, Chuang Zhang, Haoran Wu, and Sifa Zheng.
Sparsedrive: End-to-end autonomous driving via sparse scene representation.
arXiv preprint arXiv:2405.19620, 2024.
- [34]
Ran Tian, Boyi Li, Xinshuo Weng, Yuxiao Chen, Edward Schmerling, Yue Wang, Boris Ivanovic, and Marco Pavone.
Tokenize the world into object-level knowledge to address long-tail events in autonomous driving.
arXiv preprint arXiv:2407.00959, 2024.
- [35]
Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Chenxu Hu, Yang Wang, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao.
Drivevlm: The convergence of autonomous driving and large vision-language models.
arXiv preprint arXiv:2402.12289, 2024.
- [36]
Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, and Jose M Alvarez.
Omnidrive: A holistic llm-agent framework for autonomous driving with 3d perception, reasoning and planning.
arXiv preprint arXiv:2405.01533, 2024.
- [37]
Wenhai Wang, Jiangwei Xie, ChuanYang Hu, Haoming Zou, Jianan Fan, Wenwen Tong, Yang Wen, Silei Wu, Hanming Deng, Zhiqi Li, et al.
Drivemlm: Aligning multi-modal large language models with behavioral planning states for autonomous driving.
arXiv preprint arXiv:2312.09245, 2023.
- [38]
Yunkai Wang, Dongkun Zhang, Yuxiang Cui, Zexi Chen, Wei Jing, Junbo Chen, Rong Xiong, and Yue Wang.
Domain generalization for vision-based driving trajectory generation.
In 2022 International Conference on Robotics and Automation (ICRA), pages 8950–8956. IEEE, 2022.
- [39]
Zhibo Wang, Jiayu Guo, Haiqiang Zhang, Ru Wan, Junping Zhang, and Jian Pu.
Bridging the gap: Improving domain generalization in trajectory prediction.
IEEE Transactions on Intelligent Vehicles, 9(1):1780–1791, 2023.
- [40]
Zitian Wang, Zehao Huang, Yulu Gao, Naiyan Wang, and Si Liu.
Mv2dfusion: Leveraging modality-specific object semantics for multi-modal 3d detection.
arXiv preprint arXiv:2408.05945, 2024.
- [41]
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.
Chain-of-thought prompting elicits reasoning in large language models.
Advances in neural information processing systems, 35:24824–24837, 2022.
- [42]
Xinshuo Weng, Boris Ivanovic, Yan Wang, Yue Wang, and Marco Pavone.
Para-drive: Parallelized architecture for real-time autonomous driving.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15449–15458, 2024.
- [43]
Christopher KI Williams and Carl Edward Rasmussen.
Gaussian processes for machine learning, volume 2.
MIT press Cambridge, MA, 2006.
- [44]
Penghao Wu, Xiaosong Jia, Li Chen, Junchi Yan, Hongyang Li, and Yu Qiao.
Trajectory-guided control prediction for end-to-end autonomous driving: A simple yet strong baseline.
Advances in Neural Information Processing Systems, 35:6119–6132, 2022.
- [45]
Luyao Ye, Zikang Zhou, and Jianping Wang.
Improving the generalizability of trajectory prediction models with frenet-based domain normalization.
In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 11562–11568. IEEE, 2023.
- [46]
Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl.
Center-based 3d object detection and tracking.
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11784–11793, 2021.
- [47]
Wenyuan Zeng, Wenjie Luo, Simon Suo, Abbas Sadat, Bin Yang, Sergio Casas, and Raquel Urtasun.
End-to-end interpretable neural motion planner.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8660–8669, 2019.
- [48]
Jiang-Tian Zhai, Ze Feng, Jinhao Du, Yongqiang Mao, Jiang-Jiang Liu, Zichang Tan, Yifu Zhang, Xiaoqing Ye, and Jingdong Wang.
Rethinking the open-loop evaluation of end-to-end autonomous driving in nuscenes.
arXiv preprint arXiv:2305.10430, 2023.
- [49]
Yunpeng Zhang, Zheng Zhu, Wenzhao Zheng, Junjie Huang, Guan Huang, Jie Zhou, and Jiwen Lu.
Beverse: Unified perception and prediction in birds-eye-view for vision-centric autonomous driving.
arXiv preprint arXiv:2205.09743, 2022.

## Appendix A RoCA Training

We trained RoCA using 2 NVIDIA A100 GPUs. As described in Section [3.3](https://arxiv.org/html/2506.10145v2#S3.SS3), in the source domain, we first train the base E2E model following standard supervised procedure, for 48 epochs, requiring approximately 24 hours. Then, we train the Gaussian process in RoCA for 6 epochs, taking approximately 3 hours, to learn the basis tokens, based on the loss in Eqs. [3](https://arxiv.org/html/2506.10145v2#S3.E3) and [5](https://arxiv.org/html/2506.10145v2#S3.E5).
Finally, we finetune the base model using the standard supervised trajectory loss and RoCA regularization of Eq. [3.3.1](https://arxiv.org/html/2506.10145v2#S3.Ex1) for 20 epochs, taking around 10 hours.
When adapting in the target domain, using the GP module as the teacher, we finetune the base model for 10 epochs, which takes approximately 5 hours.

Algorithm [1](https://arxiv.org/html/2506.10145v2#alg1) provides the pseudo code of our source-domain training (showing 1 epoch for each step).

Figure: Algorithm 1 RoCA source-domain training

## Appendix B Computation Analysis

We compare the computational costs without and with using our proposed RoCA module for trajectory prediction. As shown in Table [5](https://arxiv.org/html/2506.10145v2#A2.T5), using RoCA to perform trajectory prediction introduces slightly increased latency and parameters. However, this also considerably improves planning performance, as we have seen in Table [1](https://arxiv.org/html/2506.10145v2#S4.T1) in the main paper.

**Table 5: Latency, inferences per second (IPS), and parameters for VAD-Tiny and SparseDrive-S without and with RoCA Gaussian process-based trajectory prediction. These measurements are conducted on an NVIDIA GeForce RTX 3090 GPU.**
| Metrics | RoCA (VAD-Tiny) | RoCA (SparseDrive-S) |  |  |
| --- | --- | --- | --- | --- |
| Base model | Base model + RoCA | Base model | Base model + RoCA |  |
| Latency (ms) | 59.5 | 75.7 | 133 | 167 |
| IPS | 16.8 | 13.2 | 7.5 | 6 |
| Parameters (M) | 15.4 | 16.8 | 88.5 | 89.9 |

## Appendix C Ablation Study

Table [6](https://arxiv.org/html/2506.10145v2#A3.T6) shows the effect of the different loss terms that we use in training, with SparseDrive-S as the base E2E model, *i.e.,* the last row (ID 5) corresponds to SparseDrive-S w/ RoCA training regularization in Table [1](https://arxiv.org/html/2506.10145v2#S4.T1) in the main paper. We see that these terms are effective, each contributing to reducing the trajectory error and/or collision rate of the trained model.

**Table 6: Effect of different loss terms on source-domain training.**
| ID | $\mathcal{L}_{rec}$ | $\mathcal{L}_{{sup}}$ | Avg. L2 (m) $\downarrow$ | Avg. Col.(%)$\downarrow$ |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| $\mathcal{L}_{planning}$ + $\mathcal{L}_{motion}$ | $\mathcal{L}_{class}$ | $\mathcal{L}_{tripplet}$ |  |  |  |  |
| 1 | ✗ | ✗ | ✗ | ✗ | 0.65 | 0.140 |
| 2 | ✓ | ✗ | ✗ | ✗ | 0.65 | 0.136 |
| 3 | ✓ | ✓ | ✗ | ✗ | 0.64 | 0.130 |
| 4 | ✓ | ✓ | ✓ | ✗ | 0.63 | 0.130 |
| 5 | ✓ | ✓ | ✓ | ✓ | 0.63 | 0.127 |

## Appendix D Comparison with VLP pan2024vlp

VLP is one of few existing works that investigate cross-domain performance for end-to-end autonomous driving.
Since VLP authors have not released their codes/models, we cannot evaluate their method using the standardized nuScenes evaluation protocol. In order to compare with their reported numbers in the paper, here we use the VAD jiang2023vad evaluation protocol.

Table [7](https://arxiv.org/html/2506.10145v2#A4.T7) summarizes the comparison of cross-domain generalization performance on nuScenes validation set. The models are trained on the subset of training data collected in one city (*e.g.,* Boston) and then, zero-shot evaluated on the validation data belonging to the other city (*e.g.,* Singapore). We see that our proposed RoCA provides the best domain generalized planning.

**Table 7: Cross-domain evaluation on Boston and Singapore subsets of nuScenes validation.**
| Model | Boston | Singapore |  |  |
| --- | --- | --- | --- | --- |
| Avg. L2 (m) $\downarrow$ | Avg. Col. (%) | Avg. L2 (m) $\downarrow$ | Avg. Col. (%) |  |
| VAD-T jiang2023vad | 0.86 | 0.27 | 0.78 | 0.39 |
| SparseDrive-S sun2024sparsedrive | 0.84 | 0.23 | 0.70 | 0.15 |
| VLP (VAD) pan2024vlp | 0.73 | 0.22 | 0.63 | 0.20 |
| VLP (UniAD) pan2024vlp | 1.14 | 0.26 | 0.87 | 0.34 |
| RoCA (VAD-T) | 0.69 | 0.14 | 0.56 | 0.21 |
| RoCA (SparseDrive-S) | 0.52 | 0.09 | 0.50 | 0.12 |

## Appendix E Cross-City Evaluation on Targeted Validation Split

As an extension of Table [2](https://arxiv.org/html/2506.10145v2#S4.T2) in the main paper, Table [8](https://arxiv.org/html/2506.10145v2#A5.T8) here provides the cross-domain performance on targeted scenarios (where the ego vehicle has to make a turn) on nuScenes.

We see that in these more challenging scenarios, our proposed RoCA consistently outperforms the baseline model in terms of domain generalization (no adaptation), and the direct finetuning in terms of domain adaptation (model weights are updated). It it noteworthy that even without using the ground-truth trajectory annotations, models adapted using RoCA perform better than those directly finetuned with ground truth in the target city. When using ground-truth labels, RoCA further improves the performance of the adapted models.

**Table 8: Cross-city planning performance on targeted scenarios.**
| Method | Adapt | Boston $\rightarrow$ Singapore | Singapore $\rightarrow$ Boston |  |  |
| --- | --- | --- | --- | --- | --- |
| Avg. L2 (m) $\downarrow$ | Avg. Col.(%)$\downarrow$ | Avg. L2 (m) $\downarrow$ | Avg. Col.(%)$\downarrow$ |  |  |
| VAD-T | ✗ | 1.629 | 0.482 | 1.157 | 0.224 |
| w/ finetuning using GT | ✓ | 1.402 | 0.270 | 1.105 | 0.190 |
| w/ RoCA training regularization | ✗ | 1.241 | 0.256 | 1.006 | 0.198 |
| w/ RoCA unsupervised adaptation | ✓ | 1.035 | 0.231 | 0.951 | 0.172 |
| w/ RoCA adaptation using GT | ✓ | 0.897 | 0.200 | 0.890 | 0.169 |
| SparseDrive-S | ✗ | 1.061 | 0.281 | 1.014 | 0.241 |
| w/ finetuning using GT | ✓ | 0.785 | 0.199 | 0.671 | 0.122 |
| w/ RoCA training regularization | ✗ | 0.942 | 0.192 | 0.886 | 0.156 |
| w/ RoCA unsupervised adaptation | ✓ | 0.800 | 0.104 | 0.688 | 0.115 |
| w/ RoCA adaptation using GT | ✓ | 0.701 | 0.094 | 0.511 | 0.104 |

## Appendix F Long-Tail Evaluation on nuScenes Validation

Table [9](https://arxiv.org/html/2506.10145v2#A6.T9) is an extension of Table [1](https://arxiv.org/html/2506.10145v2#S4.T1) in the main paper, where we evaluate the long-tail scenarios, such as resume from step, overtake, and 3-point turn, in the nuScenes validation set, using the standardized evaluation protocol. We use the average L2 trajectory error as the metric and do not use the collision rate here, as it is less statistically stable due to the small number of long-tail samples.

We see that RoCA enables better performance in these long-tail scenarios, confirming its effectiveness in making the model more generalizable. In particular, while a base model like VAD-T under-performs other existing methods, by leveraging RoCA, we are able to significantly boost its performance and make it work better in these challenging long-tail cases.

**Table 9: Comparison of L2 trajectory error and collision rate comparison on long-tail scenarios of nuScenes validation set caesar2020nuscenes using the standardized evaluation weng2024drive. *Uses the trajectory inferred by the RoCA Gaussian process module.**
| Model | Using Ego Status | Resume from Stop | Overtake | 3-Point Turn |
| --- | --- | --- | --- | --- |
| PARA-Drive weng2024drive | ✗ | 1.08 | 1.03 | 1.55 |
| TOKEN tian2024tokenize | ✗ | 0.80 | 0.90 | 1.43 |
| VAD-T jiang2023vad | ✗ | 1.75 | 1.32 | 1.83 |
| w/  RoCA training regularization | ✗ | 1.13 | 1.21 | 1.72 |
| w/  RoCA trajectory prediction* | ✗ | 0.95 | 1.00 | 1.41 |
| TOKEN tian2024tokenize | ✓ | 0.65 | 0.74 | 0.73 |
| SparseDrive-S sun2024sparsedrive | ✓ | 0.67 | 0.69 | 0.79 |
| w/  RoCA training regularization | ✓ | 0.41 | 0.63 | 0.71 |
| w/  RoCA trajectory prediction* | ✓ | 0.34 | 0.53 | 0.61 |

## Appendix G Additional Qualitative Results

We show additional qualitative results on challenging scenes in the nuScenes validation set (best viewed in color).(^4^44Images from nuScenes, licensed under [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode).)

Fig. [4](https://arxiv.org/html/2506.10145v2#A7.F4) shows a night-time scenario where it is very dark. RoCA generates the correct trajectory while the baseline leads the car outside the drivable area.

Fig. [5](https://arxiv.org/html/2506.10145v2#A7.F5) shows a sample at an intersection with several other cars and multiple pedestrians. RoCA provides the right action. The baseline generates a trajectory that goes out of the road. The predicted trajectories of other agents are also shown in the figure.

Figure: Figure 4: Qualitative result on a night-time scenario in nuScenes validation set.
Refer to caption: extracted/6549955/figures/2324_1.jpg

Figure: Figure 5: Qualitative result on an intersection scenario in nuScenes validation set.
Refer to caption: extracted/6549955/figures/0394_1.jpg

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 4
presentation: 2
questions: 1. How do you define the problem? What are the precise inputs and outputs for the entire system?
2. How are the initial token embeddings (e and a from the base model) learned, and what information do they carry? How does RoCA handle cases where the base model fails to recognize a new driving scenario and produces a problematic or out-of-distribution token embedding?
3. Why is a set of tokens needed for each group? What are the special characteristics of the learned basis tokens within a single group? How were the group size and the number of groups chosen?
4. How is the group classification performed? The paper mentions an MLP and ground-truth labels. How are these ground-truth group labels obtained?
soundness: 2
strengths: 1. The paper tackles the critical and practical problem of cross-domain generalization and adaptation for E2E models, which is a significant barrier to real-world deployment.
2. The approach of using a Gaussian Process over a learned codebook of tokens. This probabilistic formulation allows for principled uncertainty estimation.
3. The framework demonstrates consistent performance improvements across multiple base E2E models, and across various challenging settings, including closed-loop simulation, sim-to-real, cross-city transfer, and robustness to image degradations.
summary: This paper proposes RoCA, a novel framework for robust cross-domain end-to-end autonomous driving. The core idea is to learn a codebook of basis tokens that represent diverse ego and agent states, with each token mapping to a corresponding trajectory. RoCA uses a Gaussian Process (GP) to model the joint probabilistic distribution over these tokens. This GP-based module is then used in two ways: 1) as a regularizer during source-domain training to improve the generalization of a base E2E model, and 2) as an adaptation module to improve performance when transferring to new target domains.
weaknesses: 1. The paper is hard to follow, and the writing needs to be greatly improved. This is a major barrier to understanding the contribution and evaluating the method's soundness.
    - Key terminologies (e.g., 'codebook', 'basis token', 'ego and agent states') are used extensively in the abstract and introduction before being formally defined in the methodology. The author is expected to define or explain implementations of terminologies when they first appear.
    - The paper lacks a clear, formal problem definition. It jumps into the system overview without first defining the precise inputs (e.g., multi-view images, ego status?) and the target outputs (e.g., ego waypoints, agent trajectories?).
    - The entire method is based on Gaussian Processes, but there is no preliminary section or background provided. A brief mathematical overview of GP regression in the main text would be helpful for readers.
    - The experiment section is dense and difficult to parse. It would be much clearer if it started with a summary of the experimental goals, the base models used, and explained more about the different settings being tested (e.g., source-domain generalization, unsupervised adaptation, active learning) before presenting the tables. Also, it will be clearer to bold the highest scores in tables.

2. The paper is missing critical analysis to justify why the proposed formulation is superior to simpler alternatives.
    - The core claim is that the GP formulation adds robustness. The t-SNE plot (Fig 2) shows better cluster separation, but the paper does not deeply analyze why this probabilistic approach leads to better generalization compared to a deterministic one.
    - The "codebook" of "basis trajectories" sounds very similar to a set of anchors. The paper is missing a crucial comparison to a strong anchor-based method. What is the explicit advantage of this complex GP formulation over a simpler, non-probabilistic method that also learns a codebook of "anchors" (e.g., via k-means or simple end-to-end learning) and predicts a residual? An ablation is needed to isolate the benefit of the GP itself.

I am willing to increase my score if the authors can substantially improve the paper's clarity and address the key questions regarding the method's justification.

---

confidence: 3
contribution: 3
initial_rating: 4
presentation: 3
questions: 1. In cross-city experiments, is the set of basis trajectories constructed from the data of a single city, or from the whole nuScenes dataset?
soundness: 2
strengths: 1. The proposed framework is general and works with several end-to-end models.
2. The motivation of integrating Gaussian process (GP) is clear.
summary: The paper introduces RoCA, a robust cross-domain end-to-end autonomous driving framework. RoCA learns the joint probabilistic distribution over the tokens in the end-to-end pipeline with a set of basis tokens via Gaussian process. Then it infers the future trajectories with GP-based regression. The framework shows strong domain generalization and adaptation performance.
weaknesses: 1. The closed-loop evaluation in Table 1 should be on the whole validation set (220 routes) of Bench2Drive rather than Dev10 subset. The Dev10 is proposed for quick development and ablation studies. For main results, it is better to evaluate on the whole validation set for comprehensive evaluation and convenient comparation.
2. Although the Table 1 reports the closed-loop results, the experiments of domain adaptation and robustness are all open-loop. It is recommended to measure these significant abilities of RoCA in closed-loop manner too. 
3. The set of basis trajectories are constructed by sampling from dataset and clustering. Thus, the set can only represent the data distribution of the selected dataset rather than the general distribution. It may be hard to formulate the distribution in other domain with the set.
4. The citiation format does not meet the requirements of ICLR that "citations within the text should include the authors’ last names and year".

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Learning the Basis Tokens B is central to ROCA. How can you ensure that B doesn't collapse during training to represent only a few common scenarios? Could you provide a t-SNE visualization of B at different training stages (similar to Figure 2, but showing the evolution) to further demonstrate its coverage of diverse driving modes?

2. Figure 2 shows clear token clustering. Could the authors provide more analysis explaining which specific driving behaviors or scenarios (e.g., "sharp turns," "lane changes," etc.) these learned basis token B clusters represent to enhance the interpretability of the method?

3. How does RoCA behave when encountering a completely unseen scenario that far exceeds the coverage of the source domain basis tokens B? In this corner case, will the prediction variance σ² reliably spike, thus correctly indicating a lack of model confidence?

4. Formatting issue: The citation style in the paper does not comply with the conference requirements. Please revise it.
soundness: 3
strengths: - For the first time, Gaussian processes are introduced as an uncertainty-aware module into the token/feature space of E2E autonomous driving for probabilistic trajectory modeling. This approach provides a theoretically complete Bayesian framework for autonomous driving decision-making, naturally quantifying the uncertainty of predictions.

- Utilizing an active learning strategy guided by Gaussian process prediction variance, performance comparable to or even better than randomly selected 10% or 15% of the target domain data is achieved using only 5% of the target domain data, significantly improving the efficiency of target domain data labeling and adaptation.

- Compelling closed-loop experiments were conducted on the Bench2Drive dataset, achieving promising results. Extensive robustness testing, active learning testing, and ample ablation studies strongly demonstrate the contributions of each component within the framework.
summary: This paper proposes a novel end-to-end autonomous driving framework, RoCA, which aims to address the insufficient robustness of E2E models in cross-domain generalization and long-tail scenarios. The core innovation of ROCA lies in the introduction of a Gaussian Process-based module for jointly modeling the probability distribution of Ego and Agent Tokens extracted from the E2E model and their corresponding future trajectories. By learning a set of "basis tokens" representing diverse driving scenarios and their corresponding trajectories, RoCA can perform probabilistic inference for new scenarios and estimate prediction uncertainty.

Extensive experiments demonstrate that RoCA significantly outperforms existing baseline methods and some LLM-based methods in terms of domain generalization and adaptation.
weaknesses: ROCA is an add-on module that relies on the quality of the Ego/Agent Tokens output by the Scene Encoder of the baseline E2E model. If the base model's tokens exhibit significant instability and domain shift across domains, ROCA's performance will be limited.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Corresponding to W1, the authors are suggested to clearly define the scope of “domain” in this context or add some clarification.
2. Corresponding to W2, the authors are suggested to compare with more LLM/VLM-based methods.
soundness: 3
strengths: 1. Cross-domain robustness: Effectively improves generalization across different cities, lighting, and weather conditions.
2. Uncertainty modeling: Provides principled uncertainty estimates that aid in active learning and online adaptation.
3. Plug-and-play design: Can be attached to existing E2E models (e.g., VAD, SparseDrive) without changing their architectures.
4. Strong empirical results: Consistently outperforms baselines and several methods on multiple benchmarks.
summary: This paper proposes RoCA (Robust Cross-domain end-to-end Autonomous driving), a Gaussian Process–based framework that enhances the robustness and adaptability of end-to-end driving models across different domains. Key Contributions: (1) Introduces a probabilistic Gaussian Process module that models joint distributions of ego and agent tokens for trajectory prediction. (2) Builds a learnable codebook of basis tokens and trajectories to improve cross-domain generalization. (3) Enables uncertainty-aware domain adaptation and active learning without heavy retraining. (4)Demonstrates state-of-the-art cross-domain performance on Bench2Drive and nuScenes benchmarks.
weaknesses: 1. The paper’s “cross-domain” claim is somewhat overstated. While the title suggests broad domain generalization (e.g., across cities, weather, or sensors), the experiments mainly cover cross-city and sim-to-real settings, which represent limited domain shifts.
2. The paper argues that recent LLM/VLM-based E2E driving models do not guarantee cross-domain generalization and may incur high retraining costs. However, this claim is not experimentally supported, the authors do not provide many quantitative comparisons with such models (except for DiMA) under domain-shift scenarios. Given that LLMs and VLMs are often credited with strong open-world or cross-domain capabilities, the lack of empirical evidence weakens this argument and makes the motivation for replacing them with RoCA less convincing.


Decision: Reject