UniDrive: Towards Universal Driving Perception Across Camera Configurations

## Abstract

Abstract Vision-centric autonomous driving has demonstrated excellent performance with economical sensors. As the fundamental step, 3D perception aims to infer 3D information from 2D images based on 3D-2D projection. This makes driving perception models susceptible to sensor configuration (e.g., camera intrinsics and extrinsics) variations. However, generalizing across camera configurations is important for deploying autonomous driving models on different car models. In this paper, we present UniDrive , a novel framework for vision-centric autonomous driving to achieve universal perception across camera configurations. We deploy a set of unified virtual cameras and propose a ground-aware projection method to effectively transform the original images into these unified virtual views. We further propose a virtual configuration optimization method by minimizing the expected projection error between original cameras and virtual cameras. The proposed virtual camera projection can be applied to existing 3D perception methods as a plug-and-play module to mitigate the challenges posed by camera parameter variability, resulting in more adaptable and reliable driving perception models. To evaluate the effectiveness of our framework, we collect a dataset on Carla by driving the same routes while only modifying the camera configurations. Experimental results demonstrate that our method trained on one specific camera configuration can generalize to varying configurations with minor performance degradation.
Code and benchmark: https://github.com/ywyeli/UniDrive .

## 1 Introduction

Vision-centric autonomous driving has gained significant traction (Wang et al., ) due to its ability to deliver high-performance perception using economical sensors like cameras. At the core of this approach lies 3D perception (Liu et al., ), which reconstructs 3D spatial information from 2D images via 2D-3D lift transform (Philion & Fidler, ). This transform is critical for enabling vehicles to understand their environment, detect objects, and navigate safely. Previous works (Huang et al., ; Xie et al., ; Reading et al., ; Li et al., ; Zhou & Krähenbühl, ; Zeng et al., ; Lu et al., ; Huang & Huang, ; Liu et al., ) have achieved remarkable 3D perception ability by utilizing Bird’s Eye View (BEV) representations to process 2D-3D lift. Recently, many vision-based 3D occupancy prediction methods (Huang et al., ; Wei et al., ; Huang et al., ; ; Zhao et al., ) further improved the understanding of dynamic and cluttered driving scenes, pushing the boundaries of the research domain. As a result, vision-based systems have become one of the primary solutions for scalable autonomous driving.

Despite the exciting development of the state-of-the-art vision-based autonomous driving (Liu et al., ; Zong et al., ; Zheng et al., ; ; Hu et al., ; Jiang et al., ), a critical limitation still remains: the sensitivity of these models to variations in camera configurations, including intrinsics and extrinsics. Autonomous driving models typically rely on well-calibrated sensor setups, and even slight deviations in camera parameters across different vehicles or platforms can significantly degrade performance (Wang et al., ). As illustrated in Figure 1, this lack of robustness to sensor variability makes it challenging to transfer perception models between different vehicle platforms without extensive retraining or manual adjustment.
This variation necessitates training separate models for each vehicle, which consumes a significant amount of computational resources. Thus, achieving generalization across camera configurations is essential for the practical deployment of vision-centric autonomous driving.

In this paper, we address two key questions surrounding generalizable driving perception: 1) How can we construct a unified framework that enables perception models to generalize across different multi-camera parameters? 2) How can we further optimize the generalization of perception models to ensure robust performance across varying multi-camera configurations?

To achieve this, we introduce UniDrive, a novel framework designed to address the challenge of generalizing perception models across multi-camera configurations. This framework deploys a set of unified virtual camera spaces and leverages a ground-aware projection method to transform original camera images into these unified virtual views. Additionally, we propose a virtual configuration optimization strategy that minimizes the expected projection error between the original and virtual cameras, enabling consistent 3D perception across diverse setups. Our framework serves as a plug-and-play module for existing 3D perception methods, improving their robustness to camera parameter variability. We validate our framework in CARLA by training and testing models on different camera configurations, demonstrating that our approach significantly reduces performance degradation while maintaining adaptability across diverse sensor setups.
To summarize, we make the following key contributions in this paper:

- •
To the best of our knowledge, UniDrive presents the first comprehensive framework designed to generalize vision-centric 3D perception models across diverse camera configurations.
- •
We introduce a novel strategy that transforms images into a unified virtual camera space, enhancing robustness to camera parameter variations.
- •
We propose a virtual configuration optimization strategy that minimizes projection error, improving model generalization with minimal performance degradation.
- •
We contribute a systematic data generation platform along with a 160,000 frames multi-camera dataset, and benchmark evaluating perception models across varying camera configurations.

Figure: (a) Deploy on *same* camera configurations: Succeed!
Refer to caption: extracted/5935521/figure/2_succeed.jpg

## 2 Related Work

Vision-based 3D Detection.
The development of camera-only 3D perception has gained great momentum recently. Early works such as FCOS3D (Wang et al., ), which extended the 2D FCOS detector (Tian et al., ) by adding 3D object regression branches, paved the way for improvements in depth estimation via probabilistic modeling (Wang et al., ; Chen et al., ).
Later methods like DETR3D (Wang et al., ), PETR (Liu et al., ), and Graph-DETR3D (Chen et al., ) applied transformer-based architectures with learnable object queries in 3D space, drawing from the foundations of DETR (Zhu et al., ; Wang et al., ), bypassing the limitations of perspective-based detection.
Recent works utilize bird’s-eye view (BEV) for better 3D understanding. BEVDet (Huang et al., ) and M²BEV (Xie et al., ) effectively extended the Lift-Splat-Shoot (LSS) framework (Philion & Fidler, ) for 3D object detection. CaDDN (Reading et al., ) introduced explicit depth supervision in the BEV transformation to improve depth estimation.
In addition, BEVFormer (Li et al., ), CVT (Zhou & Krähenbühl, ), and Ego3RT (Lu et al., ) explored multi-head attention mechanisms for view transformation, demonstrating further improvements in consistency.
To further enhance accuracy, BEVDet4D (Huang & Huang, ), BEVFormer (Li et al., ), and PETRv2 (Liu et al., ) leveraged temporal cues in multi-camera object detection, showing significant improvements over single-frame methods.

Cross Domain Perception.
The cross-camera configurations problem proposed in this paper lies in the area of cross-domain perception. Domain generalization or adaptation is to enhance model performance on varying domains without re-training. For 2D perception, numerous cross-domain methods, such as feature distribution alignment and pseudo-labeling (Muandet et al., ; Li et al., ; Dou et al., ; Facil et al., ; Chen et al., ; Xu et al., ; He & Zhang, ; Zhao et al., ), have primarily addressed domain shifts caused by environmental factors like rain or low light. Recent 3D driving perception works (Hao et al., ; Peng et al., ) focus on transfering the models trained on clean environment or perfect sensor situations to corrupted sensor and noisy environments, leading to several benchmarks and methods. Cross camera configuration is a relatively new topic in this area. While some works (Wang et al., ) find that the model’s overfitting to camera parameters can lead to degrade performance because the models learn the fixed observation perspectives, the driving perception across camera parameters has seldom been systematically investigated.

Sensor Configuration.
Sensor configurations has been proven important in the design of perception systems. performance (Joshi & Boyd, ; Xu et al., ). Despite being relatively new in autonomous driving research (Liu et al., ), sensor placement has gained significant attention. For instance, Hu et al. () were the first to explore multi-LiDAR setups for improving 3D object detection, and Li et al. () studied how combining LiDAR and cameras impacts multi-modal detection systems. Several other studies (Jin et al., ; Kim et al., ; Cai et al., ; Jiang et al., ) focused on the strategic positioning of roadside LiDAR sensors for vehicle-to-everything (V2X) communication, shifting away from in-vehicle sensor placements. Although many efforts have aimed to refine sensor configurations for better performance, the challenge of adapting perception models to different sensor setups has been largely overlooked. Our research is the first to explore the generalization of driving perception models across diverse camera configurations.

## 3 UniDrive

### 3.1 Problem Formulation

In real-world multi-camera driving systems, perception models are typically trained on a specific camera configuration with fixed intrinsic and extrinsic parameters. However, the performance of these models often deteriorates when applied to new camera configurations, where the cameras may have different placements, orientations, or intrinsic properties.

Perception Across Multi-camera Configurations. Given a set of cameras $\mathcal{C}=\{C_{1},C_{2},\dots,C_{J}\}$, each characterized by its intrinsic matrix $\mathbf{K}^{C_{j}}\in\mathbb{R}^{3\times 3}$ and extrinsic matrix $\mathbf{E}^{C_{j}}\in\mathbb{R}^{4\times 4}$, where $j\in\{1,2,\dots,J\}$ and $J$ is the number of cameras. The images captured by these cameras are denoted as $\mathbf{I}^{C_{j}}\in\mathbb{R}^{H^{C_{j}}\times W^{C_{j}}\times 3}$, where $H^{C_{j}}$ and $W^{C_{j}}$ are the height and width of the image $\mathbf{I}^{C_{j}}$. When deploying the model trained on $\{C_{1},C_{2},\dots,C_{J}\}$ to a new set of cameras $\{C^{\prime}_{1},C^{\prime}_{2},\dots,C^{\prime}_{J^{\prime}}\}$ with different camera numbers and intrinsic and extrinsic parameters, the model may no longer effectively understand the 3D scene due to the differences between the training and testing configurations.

Universal Multi-camera Representation. To address the transferability of learned models across camera configurations, we attempt to design a universal representation, which transforms images from different camera configurations to a unified space before input to the deep learning network.
To achieve this, we propose a Virtual Camera Projection approach, which re-projects the views $\mathbf{I}^{C_{j}}$ from the original cameras $\mathcal{C}=\{C_{1},C_{2},\dots,C_{J}\}$ into a unified set of virtual camera configurations $\mathcal{V}=\{V_{1},V_{2},...,V_{K}\}$, where $K$ is the number of virtual cameras. The image is represented as $\mathbf{I}^{V_{k}}\in\mathbb{R}^{H^{V_{k}}\times W^{V_{k}}\times 3}$, where $H^{V_{k}}$ and $W^{V_{K}}$ are the image sizes, and $k\in\{1,2,\dots,K\}$ indexes the virtual camera views. We denote $\mathbf{K}^{V_{k}}$ and $\mathbf{E}^{V_{k}}$ as the intrinsic and extrinsic matrices for the virtual camera $V_{k}$. This virtual configuration serves as a standardized coordinate system for both training and inference, allowing the model to operate consistently across different physical camera setups.

Figure: Figure 2: Overview of UniDrive framework. We transform the input images into a unified virtual camera space to achieve universal driving perception. To estimate the depth of pixels in the virtual view for projection, we propose a ground-aware depth assumption strategy. To obtain the most effective virtual camera space for multiple real camera configurations, we propose a data-driven CMA-ES (Hansen, ) based optimization strategy. To evaluate the efficacy of our framework, we propose an automatic data generation platform in CARLA (Dosovitskiy et al., ).
Refer to caption: x1.png

### 3.2 Virtual Camera Projection

In this subsection, we explain the Virtual Camera Projection method to project points from multiple camera views onto virtual camera views using a combination of ground and cylindrical surface assumptions, as shown in Figure 2. The goal is to learn a transformation function $\mathcal{T}_{V\leftarrow C}$ that maps the images from the original cameras $\mathcal{C}=\{C_{1},C_{2},\dots,C_{J}\}$ to the virtual cameras $\mathcal{V}=\{V_{1},V_{2},\dots,V_{K}\}$ with minimum errors.

Ground-aware Assumption. For each pixel at coordinates $(u^{V_{k}},v^{V_{k}})$ in the virtual view, its 3D coordinates in the virtual camera frame $(X^{V_{k}}_{\text{c}},Y^{V_{k}}_{\text{c}},Z^{V_{k}}_{\text{c}})$ are calculated based on the pixel’s position in the image and the depth assumptions. Let the camera height be $h_{\text{c}}$, the focal lengths of the camera be $f_{x}^{V_{k}}$ and $f_{y}^{V_{k}}$, and the principal point (image center) be $(c_{x}^{V_{k}},c_{y}^{V_{k}})$.
We first project all pixels to the ground plane to compute the initial assumption of 3D coordinates in virtual camera frame as,

$$ $\displaystyle\hat{X}^{V_{k}}_{\text{c}}=\frac{(u^{V_{k}}-c_{x}^{V_{k}})\cdot h _{\text{c}}}{(v^{V_{k}}-c_{y}^{V_{k}})/f_{y}^{V_{k}}}~{},\ \hat{Y}^{V_{k}}_{ \text{c}}=h_{\text{c}}~{},\ \hat{Z}^{V_{k}}_{\text{c}}=\frac{h_{\text{c}}}{(v^ {V_{k}}-c_{y}^{V_{k}})/f_{y}^{V_{k}}}~{}.$ (1) $$

The Euclidean distance to optical center is computed as $\hat{D}_{\text{c}}^{V_{k}}=\left\|\begin{pmatrix}\hat{X}^{V_{k}}_{\text{c}},
\hat{Y}^{V_{k}}_{\text{c}},\hat{Z}^{V_{k}}_{\text{c}}\end{pmatrix}\right\|$.
Then we compare the distance $\hat{D}_{\text{c}}^{V_{k}}$ with threshold $D_{0}$, if $\hat{D}_{\text{c}}^{V_{k}}<D_{0}$, the points connected to corresponding pixels in the images are assumed on the ground, $(X^{V_{k}}_{\text{c}},Y^{V_{k}}_{\text{c}},Z^{V_{k}}_{\text{c}})=(\hat{X}^{V_{
k}}_{\text{c}},\hat{Y}^{V_{k}}_{\text{c}},\hat{Z}^{V_{k}}_{\text{c}})$.
If $\hat{D}_{\text{c}}^{V_{k}}\geq D_{0}$, we assume that the points lie on a cylindrical-like surface at a fixed distance $D_{0}$ from the camera’s optical center. In this case, the 3D coordinates are computed as:

$$ $\displaystyle X^{V_{k}}_{\text{c}}=\frac{(u^{V_{k}}-c_{x}^{V_{k}})\cdot D_{0}} {d^{V_{k}}}~{},\ Y^{V_{k}}_{\text{c}}=\frac{(v^{V_{k}}-c_{y}^{V_{k}})\cdot D_{ 0}}{d^{V_{k}}}~{},\ Z^{V_{k}}_{\text{c}}=\frac{f_{y}^{V_{k}}\cdot D_{0}}{d^{V_ {k}}}~{},$ (2) $$

where $d^{V_{k}}=\left\|\begin{pmatrix}u^{V_{k}}-c_{x}^{V_{k}},v^{V_{k}}-c_{y}^{V_{k}
},f_{y}^{V_{k}}\end{pmatrix}\right\|$.

Figure: Algorithm 1 Virtual Camera Projection

Point-wise Projection.
Once the 3D coordinates $(X^{V_{k}}_{\text{c}},Y^{V_{k}}_{\text{c}},Z^{V_{k}}_{\text{c}})$ in the virtual camera frame are calculated, we transform the point into the world coordinate system with extrinsic matrix $\mathbf{E}^{V_{k}}$, $\mathbf{p}_{\text{w}}=\mathbf{E}^{V_{k}}\cdot\mathbf{p}^{V_{k}}_{\text{c}}$,
where $\mathbf{p}^{V_{k}}_{\text{c}}=(X^{V_{k}}_{\text{c}},Y^{V_{k}}_{\text{c}},Z^{V_
{k}}_{\text{c}},1)^{\top}$ is the homogeneous coordinate of the point in the virtual camera’s frame, and $\mathbf{p}_{\text{w}}\in\mathbb{R}^{4}$ is the 3D point in the world coordinate system.
Next, we transform the point from the world coordinate system into the original camera’s coordinate system using the inverse of the original camera’s extrinsic matrix $\mathbf{p}^{C_{j}}_{\text{c}}={\mathbf{E}^{C_{j}}}^{-1}\cdot\mathbf{p}_{\text{
w}}$.
Finally, we project the point back onto the original camera’s 2D image plane using its intrinsic matrix, $(u^{C_{j}},v^{C_{j}},1)^{\top}=\mathbf{K}^{C_{j}}\cdot\mathbf{p}^{C_{j}}_{c}=
\mathbf{K}^{C_{j}}\cdot{\mathbf{E}^{C_{j}}}^{-1}\cdot\mathbf{p}_{\text{w}}$.
This provides the pixel coordinates $(u^{C_{j}},v^{C_{j}})$ in the original view that correspond to the pixel $(u^{V_{k}},v^{V_{k}})$ in the virtual view. We denote $\mathbf{P}_{V_{k}\leftarrow C_{j}}(\hat{D}_{\text{c}}^{V_{k}})$ as the projection transform matrix from $(u^{C_{j}},v^{C_{j}})$ in the $i$-th original view to $(u^{V_{k}},v^{V_{k}})$ based on the Euclidean distance to virtual camera optical center $\hat{D}_{\text{c}}^{V_{k}}$.

Image-level Transformation.
The point-wise projection is extended to the entire image view. For each pixel $(u^{V_{k}},v^{V_{k}})$ in the $k$-th virtual view, we compute the corresponding pixel $(u^{C_{j}},v^{C_{j}})$ in the $i$-th original view based on the projection matrix $\mathbf{P}_{V_{k}\leftarrow C_{j}}(\hat{D}_{\text{c}}^{V_{k}})$. The entire image $\mathbf{I}^{C_{j}}$ of the $i$-th original view is warped into the virtual view $\mathbf{I}^{V_{k}\leftarrow C_{j}}$ as follows, $\mathbf{I}^{V_{k}\leftarrow C_{j}}=\mathcal{T}(\mathbf{I}^{C_{j}},\mathbf{P}_{
V_{k}\leftarrow C_{j}}(\hat{D}_{\text{c}}^{V_{k}}))$, where $\mathcal{T}(\mathbf{I},\mathbf{P})$ represents the warping function applied to the image $\mathbf{I}^{C_{j}}$ using the projection matrix $\mathbf{P}_{V_{k}\leftarrow C_{j}}(\hat{D}_{\text{c}}^{V_{k}})$ based on the $\hat{D}_{\text{c}}^{V_{k}}$.

Blending Multiple Views.
Since each pixel in a single virtual view may have corresponding pixels from various original view, after transforming each original view into the virtual view, we merge all the transformed images $\mathbf{I}^{V_{k}\leftarrow C_{j}}$ to form the final output image $\mathbf{I}^{V_{k}}$. This blending is performed by computing a weighted sum of all the projected views:

$$ $\displaystyle\mathbf{I}^{V_{k}}=\frac{1}{\mathbf{W}}\sum_{j=1}^{J}w_{j}\cdot \mathbf{I}^{V_{k}\leftarrow C_{j}}=\frac{1}{\mathbf{W}}\sum_{i=1}^{J}w_{j} \cdot\mathcal{T}(\mathbf{I}^{C_{j}},\mathbf{P}_{V_{k}\leftarrow C_{j}}(\hat{D} _{\text{c}}^{V_{k}})),$ (3) $$

where $\mathbf{W}=\sum_{j=1}^{J}w_{j}$ is the total weight, and $w_{j}$ is the blending weight for the $j$-th original view. The weights can be based on factors such as the angular distance between the original and virtual views, or the proximity of the cameras.
We presented the detailed computation process in Algorithm 1.

### 3.3 Virtual Projection Error

To evaluate the accuracy of the Virtual Camera Projection method in the context of a 3D object detection task, we propose a weighted projection error metric based on angular discrepancies between the virtual and original camera views. This method accounts for both angular deviations and the distance from the camera’s optical center to provide a more robust error evaluation.

Angle Computation.
Given a driving scenario of 3D bounding box information, for each 3D bounding box $b_{n}=\{(x_{n,m},y_{n,m},z_{n,m})^{\top}\}_{m=1}^{8}$, we first project its corner points onto the original camera $C_{j}$ as the pixel $(u^{C_{j}}_{n,m},v^{C_{j}}_{n,m})$, using the intrinsic matrix $\mathbf{K}^{C_{j}}$ and extrinsic matrix $\mathbf{E}^{C_{j}}$.
Then, we use the inverse of the warping process $\mathbf{P}_{V_{k}\leftarrow C_{j}}$ to find the corresponding pixel $(u^{V_{k}}_{n,m},v^{V_{k}}_{n,m})$ in the virtual camera view $V_{k}$ for each corner point.
We compute the pitch angle $\theta^{V_{k}}_{n,m}$ and yaw angle $\phi^{V_{k}}_{n,m}$ relative to the virtual camera’s optical center:

$$ $\displaystyle\theta^{V_{k}}_{n,m}=\arctan(\frac{v^{V_{k}}_{n,m}-c_{y}^{V_{k}}} {f_{y}^{V_{k}}}),\phi^{V_{k}}_{n,m}=\arctan(\frac{u^{V_{k}}_{n,m}-c_{x}^{V_{k} }}{f_{x}^{V_{k}}}).$ (4) $$

Next, for the same corner points, we directly project to the virtual view using $\mathbf{K}^{V_{k}}$ and $\mathbf{E}^{V_{k}}$ as $(u^{V_{k}\prime}_{n,m},v^{V_{k}\prime}_{n,m})$. Then the pitch angle $\theta^{V_{k}\prime}_{n,m}$ and yaw angle $\phi^{V_{k}\prime}_{n,m}$ are

$$ $\displaystyle\theta^{V_{k}\prime}_{n,m}=\arctan(\frac{v^{V_{k}\prime}_{n,m}-c_ {y}^{V_{k}}}{f_{y}^{V_{k}}}),\phi^{V_{k}\prime}_{n,m}=\arctan(\frac{u^{V_{k} \prime}_{n,m}-c_{x}^{V_{k}}}{f_{x}^{V_{k}}}).$ (5) $$

Angle Error Calculation. For each corner point, we compute the angular error between the original camera projection and the corresponding point in the virtual camera. The absolute errors in pitch and yaw are
$\Delta\theta_{n,m}^{V_{k}}=\left|\theta^{V_{k}}_{n,m}-\theta^{V_{k}\prime}_{n,
m}\right|,$
$\Delta\phi_{n,m}^{V_{k}}=\left|\phi^{V_{k}}_{n,m}-\phi^{V_{k}\prime}_{n,m}
\right|.$
We use the distance $D_{n,m}^{V_{k}}$ of each corner point from the original camera’s optical center as a weight. The distance is computed as
$D_{n,m}^{V_{k}}=\left\|\begin{pmatrix}x^{V_{k}}_{n,m},y^{V_{k}}_{n,m},z^{V_{k}
}_{n,m}\end{pmatrix}\right\|$.
The weighted error for each corner point is then calculated as
$\mathcal{E}_{n,m}^{V_{k}}=D_{n,m}^{V_{k}}\cdot(\Delta\theta_{n,m}^{V_{k}}+
\Delta\phi_{n,m}^{V_{k}}).$
The overall error for a 3D bounding box $b_{n}$ is obtained by summing the weighted errors of its eight corner points
$\mathcal{E}_{b_{n}}^{V_{k}}=\sum_{m=1}^{8}\mathcal{E}_{n,m}^{V_{k}}.$
We sum the projection errors across all 3D bounding boxes $b_{n}\in\mathcal{B}$ to compute the total projection error

$$ $\displaystyle\mathcal{E}=\sum_{n=1}^{N}\mathcal{E}_{b_{n}}^{V_{k}}=\sum_{n=1}^ {N}\sum_{m=1}^{8}\mathcal{E}_{n,m}^{V_{k}}.$ (6) $$

### 3.4 Optimizing Virtual Camera Configurations

Figure: Algorithm 2 Virtual Camera Configuration Optimization

Given a set of multi-camera systems, we aim to design a unified virtual camera configuration that minimizes the reprojection error across all original camera configurations. To achieve this, we adopt the heuristic optimization based on the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) (Hansen, ) to find an optimized set of virtual camera configurations.

Objective Function. Given multiple driving perception systems with varying multi-camera confgirations indexed by $s$, the total error across all systems is expressed as $\mathcal{E}_{total}=\sum_{s=1}^{S}\mathcal{E}^{(s)}(\mathbf{u})$, where $\mathbf{u}=\{V_{k},\mathbf{K}^{V_{k}},\mathbf{E}^{V_{k}}\}_{k=1}^{K}$ includes both the intrinsic and extrinsic camera parameters of virtual multi-camera framework, $K$ is the total quantity of virtual cameras and $S$ is the total quantity of multi-camera driving systems that share the same perception model. We aim to minimize this error by sampling and updating the virtual camera parameters iteratively through a CMA-ES based optimization method.

Optimization Method.
Our Optimization strategy begins by defining a multivariate normal distribution $\mathcal{N}(\mathbf{m}^{(t)},(\sigma^{(t)})^{2}\mathbf{C}^{(t)})$, where $\mathbf{m}^{(t)}$ represents the mean vector, $\sigma^{(t)}$ denotes the step size, and $\mathbf{C}^{(t)}$ is the covariance matrix at iteration $t$. The configuration space $\mathcal{U}$ is discretized with a density $\delta$, and $N_{t}$ candidate configurations $\mathbf{u}_{i}^{(t)}\sim\mathcal{N}(\mathbf{m}^{(t)},(\sigma^{(t)})^{2}\mathbf
{C}^{(t)})$ are sampled at each iteration $t$. Initialization begins with the initial mean $\mathbf{m}^{(0)}$, step size $\sigma^{(0)}$, and covariance matrix $\mathbf{C}^{(0)}=\mathbf{I}$. The updated mean vector $\mathbf{m}^{(t+1)}$ is calculated in the subsequent iteration to serve as the new center for the search distribution concerning the virtual camera configuration. The process can be mathematically expressed as:

$$ $\mathbf{m}^{(t+1)}=\sum_{i=1}^{M_{t}}w_{i}\hat{\mathbf{u}}_{i}^{(t)},\; \mathcal{E}(\hat{\mathbf{u}}_{1}^{(t)})\geq\mathcal{E}(\hat{\mathbf{u}}_{2}^{( t)})\geq\dots\geq\mathcal{E}(\hat{\mathbf{u}}_{M_{t}}^{(t)})~{},$ (7) $$

where $M_{t}$ is the number of top solutions selected to update $\mathbf{m}^{(t+1)}$, and $w_{i}$ are weights determined by solution performance. The evolution path $\mathbf{p}_{\mathbf{C}}^{(t+1)}$, which tracks the direction of successful optimization steps, is updated as:

$$ $\displaystyle\mathbf{p}_{\mathbf{C}}^{(t+1)}=(1-c_{\mathbf{C}})\cdot\mathbf{p} _{\mathbf{C}}^{(t)}+\sqrt{1-(1-c_{\mathbf{C}})^{2}}\cdot\sqrt{\frac{1}{\sum_{i =1}^{M_{t}}w_{i}^{2}}}\cdot\frac{\mathbf{m}^{(t+1)}-\mathbf{m}^{(t)}}{\sigma^{ (t)}}~{},$ (8) $$

where $c_{\mathbf{C}}$ is the learning rate for updating the covariance matrix. The covariance matrix $\mathbf{C}$, which defines the distribution’s shape for camera configurations, is adjusted at each iteration as follows:

$$ $\mathbf{C}^{(t+1)}=(1-c_{\mathbf{C}})\mathbf{C}^{(t)}+c_{\mathbf{C}}\mathbf{p} _{\mathbf{C}}^{(t+1)}{\mathbf{p}_{\mathbf{C}}^{(t+1)}}^{T}~{}.$ (9) $$

Similarly, the evolution path for the step size, $\mathbf{p}_{\sigma}$, is updated, and the global step size $\sigma$ is then adjusted to balance exploration and exploitation:

$$ $\mathbf{p}_{\sigma}^{(t+1)}=(1-c_{\sigma})\mathbf{p}_{\sigma}^{(t)}+\sqrt{1-(1 -c_{\sigma})^{2}}\cdot\sqrt{\frac{1}{\sum_{i=1}^{M_{t}}w_{i}^{2}}}\cdot\frac{ \mathbf{m}^{(t+1)}-\mathbf{m}^{(t)}}{\sigma^{(t)}}~{},$ (10) $$

$$ $\sigma^{(t+1)}=\sigma^{(t)}\exp\left(\frac{c_{\sigma}}{d_{\sigma}}\left(\frac{ \|\mathbf{p}_{\sigma}^{(t+1)}\|}{E\|\mathcal{N}(0,\mathbf{I})\|}-1\right) \right)~{},$ (11) $$

where $c_{\sigma}$ is the learning rate for updating $\mathbf{p}_{\sigma}$, and $d_{\sigma}$ is a normalization factor controlling the adjustment rate of the global step size.
We presented the detailed optimization process in Algorithm 2.

Figure: (a) $4\times 95^{\circ}$
Refer to caption: extracted/5935521/figure/4x.png

## 4 Experiments

### 4.1 Benchmark Setups

Data Generation. We generate multi-view image data and 3D objects ground truth in CARLA simulator (Dosovitskiy et al., ). We use the maps of Towns 1-6 to collect data. We incorporate 6 classes for 3D object detection, including Car, Bus, Truck, Motorcycle, Bicycle, and Pedestrian. The dataset consists of 500 scenes (20,000 frames) for each camera configuration. We split 250 scenes for training and 250 scenes for validation. Our dataset is organized as the format of nuScenes (Caesar et al., ) and compatible to the nuscenes-devkit python package for convenient processing.

Camera Configurations. We adopt several commonly used camera configurations in automotive practice with various camera quantities, placements and field of views. These configurations are represented in Figure 3. We set all camera resolutions to 1600$\times$900 as nuScenes. Our camera configurations include camera numbers from 4 to 8. For the field of view (FOV) for cameras, we conduct study mainly on 6 cameras with FOV = 60, 70, 80. For the placement, we design two different types of placement as shown in Figure 3 (c), (d). We also include the original configurations of nuScenes (Caesar et al., ) dataset with five $70^{\circ}$ cameras and a $110^{\circ}$ camera.

Deteciton Method. Due to the extensive computation resource needed to benchmark the multi-camera configurations, we only compare our method with the camera variant of BEVFusion (Liu et al., ) (abbreviated as BEVFusion-C). BEVFusion is one of the most state-of-the-art approaches in many leaderboards, such as nuScenes (Caesar et al., ) and Waymo (Sun et al., ). Thus, we believe our comparisons with BEVFusion are representative and meaningful.

### 4.2 Comparative Study

We conduct comparative studies to evaluate the performance of camera perception across configurations. Through our analysis, we are able to demonstrate the effectiveness of UniDrive framework.

Effectiveness of UniDrive. In Table 1 and 2, we present the 3D object detection results of BEVFusion-C (Liu et al., ) and UniDrive. The models are trained on one configuration and tested on other varying camera configurations. Table 1 demonstrates that the performance of BEVFusion-C degrades a lot when deployed on cross-camera configuration tasks, nearly unusable on other configurations. As shown in Table 2, we train the models using our plug-and-play UniDrive framework. The detection performance significantly improves compared to BEVFusion-C (Liu et al., ). Our method only experiences little performance degradation on cross-camera configuration tasks. We present more results in Figure 4, which comprehensively shows the effectiveness of our framework.

Optimization via UniDrive. To demonstrate the importance of optimization in UniDrive, we compare the perception performance between optimized virtual camera configurations and intuitive one in Figure 4. The intuitive virtual camera configuration places all cameras in the center of the vehicle roof. As shown in Figure 4 (b), although the intuitive setup (without optimizing) also significantly improved cross-camera configuration perception performance compared to BEVFusion-C (in Figure 4 (a)), it exhibited a clear preference for certain configurations while performing poorly on others. In contrast, the optimized virtual camera parameters (in Figure 4 (c)) demonstrated greater adaptability, showing relatively consistent performance across various configurations. This is crucial for the concurrent development of multiple multi-camera perception systems in autonomous driving.

**Table 1: Quantitative results of BEVFusion-C for 3D detection across camera configurations. The detector is trained on the blue configurations and tested on all configurations directly. We report the mAP ($\uparrow$) and class-level AP ($\uparrow$) scores in percentage ($\%$).**
| Configurations | mAP | Car | Bus | Truck | Ped. | Motor | Bic. | Configurations | mAP | Car | Bus | Truck | Ped. | Motor | Bic. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $5\times 70^{\circ}+110^{\circ}$ | 63.9 | 62.4 | 58.0 | 66.5 | 54.7 | 68.7 | 72.9 | $6\times 60$ | 69.3 | 68.7 | 67.4 | 66.3 | 62.2 | 78.4 | 72.8 |
| $4\times 95^{\circ}$ | 4.9 | 4.6 | 5.1 | 3.8 | 3.9 | 3.1 | 4.2 | $4\times 95^{\circ}$ | 3.8 | 4.1 | 4.3 | 3.2 | 4.1 | 3.3 | 3.6 |
| $5\times 75^{\circ}$ | 7.2 | 9.5 | 4.8 | 6.2 | 5.1 | 9.0 | 8.3 | $5\times 75^{\circ}$ | 3.4 | 3.7 | 2.5 | 3.1 | 2.7 | 4.3 | 4.2 |
| $6\times 80^{\circ}a$ | 8.5 | 11.7 | 8.8 | 8.4 | 6.1 | 8.2 | 7.7 | $6\times 80^{\circ}a$ | 0.6 | 1.7 | 0.4 | 0.5 | 0.1 | 0.4 | 0.6 |
| $6\times 80^{\circ}b$ | 6.9 | 10.0 | 7.2 | 7.8 | 5.2 | 6.1 | 5.6 | $6\times 80^{\circ}b$ | 0.4 | 1.4 | 0.0 | 0.7 | 0.0 | 0.2 | 0.1 |
| $6\times 70^{\circ}$ | 67.5 | 65.2 | 61.2 | 69.3 | 57.9 | 79.5 | 72.1 | $6\times 70^{\circ}$ | 8.1 | 9.6 | 4.3 | 6.8 | 7.1 | 11.0 | 10.0 |
| $6\times 60^{\circ}$ | 9.2 | 12.4 | 7.0 | 8.0 | 6.5 | 11.9 | 9.4 | $5\times 70^{\circ}+110^{\circ}$ | 4.6 | 4.9 | 3.0 | 3.5 | 3.4 | 5.4 | 7.4 |
| $8\times 50^{\circ}$ | 0.5 | 0.6 | 0.1 | 0.9 | 0.2 | 0.3 | 0.6 | $8\times 50^{\circ}$ | 17.3 | 18.5 | 9.9 | 14.1 | 16.7 | 21.2 | 23.4 |
| $6\times 80^{\circ}a$ | 66.7 | 65.4 | 66.2 | 63.7 | 55.8 | 75.9 | 72.9 | $6\times 80^{\circ}b$ | 69.1 | 66.0 | 65.1 | 72.1 | 58.3 | 78.6 | 74.2 |
| $4\times 95^{\circ}$ | 3.8 | 4.3 | 5.0 | 3.6 | 3.2 | 2.8 | 3.9 | $4\times 95^{\circ}$ | 3.5 | 3.9 | 4.1 | 3.3 | 3.2 | 2.6 | 3.7 |
| $5\times 75^{\circ}$ | 30.4 | 31.2 | 23.4 | 27.8 | 28.6 | 36.9 | 34.2 | $5\times 75^{\circ}$ | 29.6 | 30.3 | 22.6 | 27.1 | 27.9 | 36.3 | 33.2 |
| $5\times 70^{\circ}+110^{\circ}$ | 9.2 | 10.5 | 6.5 | 8.6 | 7.1 | 8.8 | 13.3 | $6\times 80^{\circ}a$ | 63.2 | 65.5 | 67.0 | 66.4 | 46.7 | 68.2 | 65.1 |
| $6\times 80^{\circ}b$ | 63.3 | 65.4 | 63.8 | 70.9 | 46.3 | 68.1 | 65.4 | $6\times 60^{\circ}$ | 1.7 | 2.9 | 0.5 | 1.1 | 0.7 | 2.3 | 2.6 |
| $6\times 70^{\circ}$ | 16.4 | 18.0 | 9.4 | 13.3 | 14.7 | 22.2 | 20.6 | $6\times 70^{\circ}$ | 16.1 | 17.7 | 8.3 | 12.3 | 14.6 | 23.7 | 19.9 |
| $6\times 60^{\circ}$ | 1.8 | 3.3 | 0.8 | 1.5 | 0.6 | 2.3 | 2.4 | $5\times 70^{\circ}+110^{\circ}$ | 8.9 | 10.3 | 5.6 | 7.5 | 7.1 | 9.6 | 13.4 |
| $8\times 50^{\circ}$ | 0.4 | 0.5 | 0.0 | 0.8 | 0.1 | 0.1 | 0.6 | $8\times 50^{\circ}$ | 0.2 | 0.4 | 0.0 | 0.9 | 0.3 | 0.3 | 0.4 |

### 4.3 Ablation Study

In this section, we analyze the interplay between our proposed virtual projection strategy and perception performance to address these questions: 1) What’s the impact of camera extrinsic and intrinsic for cross-configuration perception? 2) How UniDrive works towards these parameters separately?

Camera Intrinsics. Changes in camera intrinsics pose the greatest challenge for cross-camera parameter perception. In Figure 4 (a), BEVFusion-C almost entirely fails when tasked on distinct camera intrinsics with the detection accuracy mostly under 20%. For instance, BEVFusion only gets 1.8% when deploying models trained on $6\times 80^{\circ}a$ to $6\times 60^{\circ}$. In contrast, in Figure 5, our UniDrive framework demonstrates substantial robustness, with performance dropping by at most 9.8% under the largest intrinsic differences, which highlights the effectiveness of our approach.

Camera Height. The variation in the vertical position of cameras can significantly impact perception performance, as cameras at varying heights capture images with distinct geometric features.
We perform experiments specifically for varying camera heights at 1.6 meters, 1.4 meters, 1.8 meters, and 2.5 meters. We train the model on 1.6 meters and test on other configuraitons. As shown in Fig. 5b. BEVFusion-C experiences a substantial performance drop for more than 10%, when faced with varying camera heights. In contrast, UniDrive significantly improves performance across different camera heights, demonstrating enhanced robustness with only 3.0% performance decreasing.

Camera Placement. Changing the camera’s horizontal position and orientation on presents a relatively smaller challenge for cross-camera parameter perception. As shown in Figure 4 (a), BEVFusion-C experiences a performance drop of 5.9% when deploying the model trained on the $6\times 80^{\circ}b$ configuration to the $6\times 80^{\circ}a$ configuration. Nonetheless, our UniDrive framework further enhances cross-camera parameter perception performance. In Figure 4 (c), we train the model on the $6\times 80^{c}irca$ configuration and test on other configurations, UniDrive only experiences a 4.6% when deploying the model trained on the $6\times 80^{\circ}b$ configuration to the $6\times 80^{\circ}a$ configuration.

**Table 2: Quantitative results of UniDrive for 3D detection across camera configurations. The detector is trained on the blue configurations and tested on all configurations directly. We report the mAP ($\uparrow$) and class-level AP ($\uparrow$) scores in percentage ($\%$).**
| Configurations | mAP | Car | Bus | Truck | Ped. | Motor | Bic. | Configurations | mAP | Car | Bus | Truck | Ped. | Motor | Bic. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $5\times 70^{\circ}+110^{\circ}$ | 68.8 | 67.5 | 64.8 | 71.9 | 59.1 | 73.6 | 75.9 | $6\times 60^{\circ}$ | 64.6 | 63.4 | 58.2 | 59.7 | 59.2 | 76.7 | 70.0 |
| $4\times 95^{\circ}$ | 60.1 | 59.1 | 57.2 | 58.4 | 59.2 | 68.6 | 67.8 | $4\times 95^{\circ}$ | 58.9 | 57.7 | 54.1 | 56.9 | 53.1 | 65.2 | 67.4 |
| $5\times 75^{\circ}$ | 66.7 | 64.5 | 65.9 | 67.8 | 59.6 | 72.3 | 70.1 | $5\times 75^{\circ}$ | 62.2 | 59.5 | 60.6 | 65.8 | 53.6 | 70.3 | 63.1 |
| $6\times 80^{\circ}a$ | 69.4 | 68.4 | 68.1 | 67.5 | 57.8 | 78.2 | 76.1 | $6\times 80^{\circ}a$ | 64.4 | 65.1 | 64.9 | 65.5 | 53.3 | 70.2 | 67.1 |
| $6\times 80^{\circ}b$ | 65.8 | 64.7 | 62.0 | 63.9 | 55.8 | 76.7 | 71.7 | $6\times 80^{\circ}b$ | 65.7 | 63.2 | 63.1 | 63.9 | 55.8 | 76.7 | 71.7 |
| $6\times 70^{\circ}$ | 68.4 | 66.8 | 64.3 | 69.8 | 57.6 | 79.1 | 72.8 | $6\times 70^{\circ}$ | 65.0 | 62.9 | 60.8 | 65.4 | 55.1 | 73.0 | 72.8 |
| $6\times 60^{\circ}$ | 63.1 | 60.6 | 57.4 | 58.8 | 59.2 | 73.0 | 69.6 | $5\times 70^{\circ}+110^{\circ}$ | 63.6 | 61.2 | 61.4 | 64.8 | 55.2 | 71.0 | 68.1 |
| $8\times 50^{\circ}$ | 58.9 | 57.1 | 55.4 | 56.1 | 51.1 | 69.7 | 64.1 | $8\times 50^{\circ}$ | 63.8 | 60.2 | 58.3 | 62.6 | 56.8 | 74.2 | 70.5 |
| $6\times 80^{\circ}a$ | 69.4 | 69.0 | 67.7 | 66.6 | 58.6 | 78.4 | 76.1 | $6\times 80^{\circ}b$ | 63.1 | 63.2 | 61.9 | 59.8 | 53.4 | 71.1 | 69.4 |
| $4\times 95^{\circ}$ | 55.9 | 56.4 | 58.4 | 52.7 | 48.3 | 59.5 | 60.1 | $4\times 95^{\circ}$ | 53.6 | 51.2 | 48.0 | 49.7 | 51.0 | 61.5 | 60.1 |
| $5\times 75^{\circ}$ | 65.2 | 63.6 | 64.8 | 66.8 | 57.0 | 70.9 | 68.3 | $5\times 75^{\circ}$ | 62.7 | 62.1 | 60.6 | 62.2 | 56.3 | 69.9 | 67.0 |
| $5\times 70^{\circ}+110^{\circ}$ | 63.7 | 61.3 | 58.9 | 60.5 | 59.9 | 72.8 | 68.8 | $6\times 80^{\circ}a$ | 64.5 | 62.6 | 60.3 | 62.4 | 58.6 | 71.4 | 71.6 |
| $6\times 80^{\circ}b$ | 66.2 | 65.2 | 61.1 | 65.1 | 56.1 | 76.3 | 73.2 | $6\times 60^{\circ}$ | 62.6 | 60.3 | 59.1 | 62.6 | 53.4 | 70.2 | 70.2 |
| $6\times 70^{\circ}$ | 68.9 | 67.9 | 63.5 | 70.7 | 58.3 | 79.5 | 73.8 | $6\times 70^{\circ}$ | 62.5 | 59.4 | 55.6 | 62.9 | 52.6 | 73.4 | 70.8 |
| $6\times 60^{\circ}$ | 59.6 | 57.2 | 54.0 | 55.7 | 57.0 | 67.5 | 66.1 | $5\times 70^{\circ}+110^{\circ}$ | 57.9 | 56.1 | 52.5 | 53.7 | 56.9 | 64.3 | 63.7 |
| $8\times 50^{\circ}$ | 61.2 | 60.3 | 58.1 | 59.9 | 54.5 | 68.2 | 65.9 | $8\times 50^{\circ}$ | 58.4 | 60.3 | 57.0 | 54.3 | 53.9 | 64.4 | 60.2 |

### 4.4 Analysis

Figure: (a) BEVFusion-C (Liu et al., )
Refer to caption: x2.png

Figure: (a) Camera intrinsic
Refer to caption: x5.png

In this section, we further investigate some useful insight points found in the benchmark experiments: 1) What’s the impact of inconsistency in multi-camera intrinsics for perception? 2) How UniDrive works towards this inconsistency?

Degradation with Inconsistent Intrinsics. In our experiments, we observed that for multi-camera systems, models perform better when camera intrinsics are consistent compared to when they vary. However, due to design aesthetics and other constraints, many autonomous driving companies use multiple cameras with different intrinsic parameters to achieve 360-degree perception. For instance, the nuScenes (Caesar et al., ) uses five $70^{\circ}$ cameras and one $110^{\circ}$ camera. As shown in Fig 1, BEVFusion-C performs a lot better in $6\times 80^{\circ}a$ and $6\times 60^{\circ}$ compared to configuration $5\times 70^{\circ}+110^{\circ}$. Thus, inconsistency in camera intrinsics can potentially hinder perception improvement.

Improvement via UniDrive. Our framework significantly enhances the perception performance of multi-camera systems with varying intrinsics by leveraging a virtual camera system with consistent intrinsics. For training and testing on the same configurations, as demonstrated in Figure 2, UniDrvie achieves 68.8% accuracy in $5\times 70^{\circ}+110^{\circ}$ configuration, which surpasses 4.9% than BEVFusion-C (63.9%). For testing across camera configurations, UniDrive experiences little accuracy reduction only in rare situations. This demonstrates that UniDrive has substantial potential to push advancements in driving perception technology.

## 5 Conclusion

In this paper, we introduce the UniDrive framework, a robust solution for enhancing the generalization of vision-centric autonomous driving models across varying camera configurations. By leveraging a unified set of virtual cameras and a ground-aware projection method, our approach effectively mitigates the challenges posed by camera intrinsics and extrinsics. The proposed virtual configuration optimization ensures minimal projection error, enabling adaptable and reliable performance across diverse sensor setups. Extensive experiments in CARLA validate the effectiveness of UniDrive, demonstrating strong generalization capabilities with minimal performance loss. Our framework not only serves as a plug-and-play module for existing 3D perception models but also paves the way for more versatile and scalable autonomous driving solutions.

Limitation. The camera configurations analyzed in this paper can not cover all real-world setups, more comprehensive experiments may be required. In addition, our research are fully conducted on simulation data, as real-world experiments are time-consuming and need extensive resource.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please check weaknesses.
soundness: 3
strengths: - This paper addresses a valuable real-world demand. As autonomous driving sensor configurations (e.g., camera intrinsics, extrinsics, and the number of cameras) evolve, it becomes crucial to ensure the perception algorithms are robust to such changes.
- The authors propose a re-mapping approach that transforms pixels from the original view into a unified virtual view, enabling training and inference to be conducted on this virtual view. This approach enhances robustness to cross-configuration shifts in camera systems. Furthermore, the method is modular, making it straightforward to integrate into existing perception pipelines.
- A virtual camera configuration optimization method is also introduced, which can quantitatively find a better virtual camera configuration based on a set of real-world multi-camera system to enhance the cross-configuration robustness and perception performance. 
- Experimental results demonstrate that the proposed approach significantly improves robustness across different configurations. Interestingly, performance also improves on configurations identical to the training setup. This is surprising, as one might assume additional transformations could lead to information loss rather than enhancement. This raises questions about whether these improvements are dataset-dependent (I posed in Weaknesses).
- The paper presents a well-formulated problem and demonstrates effective solutions under simulated data conditions.

Despite some limitations, especially those related to simulated data, this paper introduces compelling ideas, offering a well-defined problem formulation that could benefit the autonomous driving industry and potentially embodied AI.
summary: The paper addresses the challenge of maintaining robustness in perception algorithms for autonomous driving systems as camera configurations—such as intrinsics, extrinsics, and the number of cameras—evolve. The authors introduce a novel re-mapping approach that transforms pixels from the original camera view to a unified virtual view, allowing training and inference to be conducted in this virtual space. This approach improves robustness across different configurations and integrates seamlessly into existing perception pipelines. 
In addition, the paper proposes a virtual camera configuration optimization method, enabling the identification of an optimal virtual configuration based on real-world multi-camera systems to further enhance robustness across configurations. 
Experimental results on synthesized data show that the method not only enhances cross-configuration performance but also unexpectedly boosts performance on the same configurations.
weaknesses: - As the authors noted, one main limitation is that all analyses were conducted in a simulated environment. While this controlled setting is useful for evaluating the method’s potential, it raises concerns about real-world applicability, as training-based detectors are often sensitive to dataset-specific factors. Given the paper’s real-world motivation, validation on actual datasets would be valuable.
- As I posed in Strengths, it is a little counterintuitive that the transformation from the original view to a virtual view may cause information loss but can bring improvements. The authors should consider proposing hypotheses to explain why this transformation improves performance.

Apart from the issues I posed above, I also have some suggestions that will not prevent this paper to be accepted if unresolved. In real-world applications, there are several datasets of different camera configurations. As the proposed method can leverage the virtual view to unify these different configurations, I am curious whether the proposed method can leverage these datasets to train a better-to-all detector on each dataset.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: To what extent does the reliance on simulated data affect the generalizability of the results to real-world settings, particularly in urban driving scenarios?

Does the framework introduce any latency or computational overhead that could impact its feasibility for real-time applications in autonomous driving?
soundness: 2
strengths: The paper addresses a valuable and relevant problem in autonomous driving, focusing on camera configuration generalization.

The proposed approach sounds generally reasonable, employing a unified virtual camera space and ground-aware projection to help manage variability in camera setups.

Experimental results show clear performance gains, with UniDrive maintaining high perception accuracy across diverse camera configurations and outperforming baseline models.
summary: The paper addresses the challenge of achieving consistent 3D perception in autonomous driving across different camera configurations. It proposes the UniDrive framework, which creates a unified virtual camera space using ground-aware projections and configuration optimization to reduce projection errors. Experiments in simulation demonstrate that UniDrive improves perception robustness across varied camera setups with minimal performance loss.
weaknesses: W1: The innovation is somewhat limited, as the framework mainly leverages established virtual projection and optimization techniques without presenting substantially novel concepts.

W2: The approach relies heavily on simulated environments (e.g., CARLA), raising concerns about its applicability and robustness in real-world conditions.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Does the virtual configuration optimization strategy actually work effectively? （Core concern: I am highly willing to increase the score if this issue is explained thoroughly.）

   1.1 In certain camera configurations, the results from a single dataset in the paper are not as good as those of the original BEVFusion-C (e.g., 6x60 69.3 vs. 64.6, 6x80b 69.1 vs. 63.1). Why can't the virtual configuration optimization improve to achieve a virtual camera space as good as or better than the original BEVFusion-C?

   1.2 In some camera configurations, the results of individual optimization are not as good as those from transfer learning, such as 60x80b (train on 60x80b 63.1 vs. train on 60x70 67.9). Why can't the virtual configuration optimization achieve a virtual camera space as good as or better than the transfer setting?

   1.3 For different camera configurations, what do the optimized Virtual Cameras look like? Can you provide the optimized intrinsics and extrinsics?

   1.4 How well does the original view warp to the virtual view? Are there any areas that are not properly warped? Can you provide visual results?

2. While BEVFusion is popular, the paper focuses on vision-centric autonomous driving. It would be beneficial to include experiments with popular detectors in the camera domain, such as BEVFormer and PETR. These differ from methods like BEVFusion-C and BEVDet, as they are based on query-based transformer solutions.

3. Were the number of virtual cameras optimized together? If not, could you provide ablation experiments?

4. The paper discusses that Inconsistent Intrinsics have a significant impact. Since real autonomous vehicles also have inconsistent heights, could you include an experimental analysis?

   4.1 Were the heights of virtual cameras optimized together?

5. Given the various camera configurations, would training on datasets that mix various configurations to optimize a unified virtual camera space lead to improved results?

6. All experiments are conducted on simulation data. Incorporating transfer experiments with real datasets would be more convincing, like the real dataset transfer experiments conducted in 3D Domain Generalization/Adaptation.

Note: I have provided several questions. The authors can prioritize addressing the core issues based on their time constraints, and it is not necessary to conduct all experiments.
soundness: 4
strengths: 1. The paper proposes to transform images into a unified virtual camera space, improving robustness across various camera configurations.
2. Additionally, the paper proposes a virtual configuration optimization strategy that minimizes projection errors.
3. The paper also presents a systematic data generation platform and a benchmark for evaluating perception models under different camera configurations.
summary: I greatly appreciate this paper; in fact, I had similar ideas that I hadn't implemented. Seeing the author perfectly execute this idea and achieve excellent results is truly exciting. The paper proposes transforming images into a unified virtual camera space with a virtual configuration optimization strategy, thereby enhancing robustness to different camera configurations.
weaknesses: See the Questions section.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - As mentioned in weaknesses, the effectiveness of the proposed method should be testified on multiple camera-only 3D detection methods.
- It is mentioned in line 302 that the virtual camera configuration is obtained through an optimization process based on a given set of camera systems; it is curious to know if the proposal method also works toward a randomly picked virtual camera setting. This would testify to the effectiveness of the proposed metric and optimization process in Sections 3.3 and 3.4. It is recommanded to compare the optimized virtual camera configuration to randomly selected configurations and evaluate performance across different camera settings.
soundness: 3
strengths: - The article is clear and easy to follow.
- The motivation of the article is a long-standing problem in the 3D detection domain, and the authors propose a feasible way for solving the problem.
- Mathematical details are given for the projection from virtual cameras to real cameras.
- In the experiments, the proposed method is testified on seven different camera settings, showing its generalization to different camera settings.
summary: Camera-only 3D object detection is sensitive to camera settings. It has been proved that a detection model trained on a camera setting fails to provide satisfactory performance in another camera setting. The authors propose a method, UniDrive, to unify this. The proposed UniDrive comprises a set of virtual cameras, which wrap real cameras into virtual space and thus bridge the gap between different camera settings. It is worth noticing that the number of real and virtual cameras does not need to be the same. The authors also proposed a virtual projection error, which enables an optimization process to find the optimal virtual camera setting. Experiments are conveyed on the CARLA simulator and the BEVFusion-C model. It is shown that, with the proposed method, model performance on various camera settings is significantly improved.
weaknesses: - Only BEVFusion-C is used in the experiments. In line 407, the authors mentioned that BEVFusion-C is one of the SOTA methods in popular public leaderboards, which is not currently the case. The effectiveness of the proposed method should show its universal capability on multiple methods with at least 2-3 different camera settings, such as some methods spercifically designed camera-only detection like BEVFormer [1] and Sparse4D [2].
- It is not spercifically described that how to adopt the proposed method to exsiting network. It would be helpful as the proposed method serves as a plug-and-play component.


[1] Li, Zhiqi, et al. "Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers." European conference on computer vision. Cham: Springer Nature Switzerland, 2022.

[2] Sun, Wenchao, et al. "SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representation." arXiv preprint arXiv:2405.19620 (2024).

Meta-review: The paper presents a method to improve 3D perception across sensor configurations. The problem is a well-recognized one and the proposed solution, while not significantly novel, is technically sound. The paper is well-written with convincing experiments. Overall, the AC agrees with the reviewers that the paper is ready for acceptance at ICLR and recommends that the authors include rebuttal clarifications in the final version of the paper.

Decision: Accept (Poster)