UniDrive: Towards Universal Driving Perception Across Camera Configurations
Authors: Ye Li
Wenzhao Zheng
Xiaonan Huang
Kurt Keutzer
Venue: ICLR 2025 Poster (2025)
Keywords: Autonomous Driving
3D Detection
Sensor Configuration
Primary area: applications to robotics, autonomy, planning

Abstract: Vision-centric autonomous driving has demonstrated excellent performance with economical sensors. As the fundamental step, 3D perception aims to infer 3D information from 2D images based on 3D-2D projection. This makes driving perception models susceptible to sensor configuration (e.g., camera intrinsics and extrinsics) variations. However, generalizing across camera configurations is important for deploying autonomous driving models on different car models. In this paper, we present UniDrive, a novel framework for vision-centric autonomous driving to achieve universal perception across camera configurations. We deploy a set of unified virtual cameras and propose a ground-aware projection method to effectively transform the original images into these unified virtual views. We further propose a virtual configuration optimization method by minimizing the expected projection error between original and virtual cameras. The proposed virtual camera projection can be applied to existing 3D perception methods as a plug-and-play module to mitigate the challenges posed by camera parameter variability, resulting in more adaptable and reliable driving perception models. To evaluate the effectiveness of our framework, we collect a dataset on CARLA by driving the same routes while only modifying the camera configurations. Experimental results demonstrate that our method trained on one specific camera configuration can generalize to varying configurations with minor performance degradation.

Published as a conference paper at ICLR 2025
UNIDRIVE: TOWARDS UNIVERSAL DRIVING PERCEPTION ACROSS CAMERA CONFIGURATIONS
Ye Li
Wenzhao Zhenga1
Xiaonan Huang
Kurt Keutzer²
University of Michigan, Ann Arbor University of California, Berkeley
https: /ttp///www/ggg//e/eeee/ weeeinng.ee//ttte /UniDrive
ABSTRACT
Vision-centric autonomous driving has demonstrated excellent performance with economical sensors. As the fundamental step, 3D perception aims to infer 3D information from 2D images based on 3D-2D projection. This makes driving perception models susceptible to sensor configuration (c.g., camera intrinsics and extrinsics) variations. However, generalizing across camera configurations is im- portant for deploying autonomous driving models different car models. In this paper, we present UniDrive, a novel framework for vision-centric autonomous driving achieve universal perception across camera configurations. We deploy a set of unified virtual cameras and propose a ground-aware projection method effectively transform the original images into these unified virtual views. We further propose a virtual configuration optimization method by minimizing the expected projection error between original and virtual cameras. The proposed vir- tual camera projection can be applied to existing 3D perception methods as plug- and-play module mitigate the challenges posed by camera parameter variability, resulting in more adaptable and reliable driving perception models. To evaluate the effectiveness of our framework, we collect a dataset on CARLA by driving the same routes while only modifying the camera configurations. Experimental results demonstrate that our method trained on one specific camera configuration generalize varying configurations with minor performance degradation.
INTRODUCTION
Vision-centric autonomous driving has gained significant traction (Wang et al., 2023a) due to its ability to deliver nigh-performance perception using economical sensors like cameras. At the core of this approach lies 3D perception (Liu et al., 2022), which reconstructs 3D spatial information from 2D images via 2D-3D lift transform (Philion & Fidler, 2020). This transform critical for enabling vehicles to understand their environment, detect objects, and navigate safely. Previous works (Huang et al., 2021; Xie al., 2022; Reading et al., 2021: Li al., 2022; Zhou Krähenbühl, 2022; Zeng et al., 2024; Lu et al., 2022: Huang & Huang, 2022; Liu et al., 2023a) have achieved remarkable 3D perception ability by utilizing Bird's Eye View (BEV) representations to process 2D-3D lift. Recently, many vasion-based 3D occupancy prediction methods (Huang et al., 2023; Wei et al., 2023; Huang et al., 2024a;b; Zhao et al., 2024) further improved the understanding of dynamic and cluttered driving scenes, pushing the boundaries of the research domain. As result, vision-based systems have become one of the primary solutions for scalable autonomous driving.
Despite the exciting development of the ffite-of-the-eee vision-based autonomous driving (Liu al., 2024: Zong et al., 2023; Zheng et al., 2024b;a; Hu et al., 2023; Jiang et al., 2023a), a critical limi- tation still remains: the sensitivity of these models variations in camera configurations, including intrinsics and extrinsics. Autonomous driving models typically rely on well-calibrated sensor se- tups, and even slight deviations in camera parameters across different vehicles or platforms can significantly degrade performance (Wang et al., 2023b). As illustrated in Figure 1a, this lack of robustness to sensor variability makes it challenging to transfer perception models between differ- ent vehicle platforms without extensive retraining or manual adjustment. This variation necessitates
Work done while visiting UC Berkeley.
1 "oorrsponniig author.

---

Published as a conference paper at ICLR 2025
(a) Deploy on same camera configurations: Succeed!
(b) Deploy on different camera configurations: Fail!
Figure I: Comparison of deploying perception models on the same and distinct configurations.
training separate models for each vehicle, which consumes significant amount of computational resources. Thus, achieving generalization across camera configurations is essential for the practical deployment of vision-centric autonomous driving.
In this paper, we address two key questions surrounding generalizable driving perception: 1) How can we construct a unified framework that enables perception models to generalize across different multi-camera parameters? ? 2) How can further optimize the generalization of perception models to ensure robust performance across varying multi-camera configurations?
To achieve this, we introduce UniDrive, novel framework designed to address the challenge of generalizing perception models across multi-camera configurations. This framework deploys a set of unified virtual camera spaces and leverages ground-aware projection method to transform origi- nal camera images into these unified virtual views. Additionally, we propose virtual configuration optimization strategy that minimizes the expected projection error between the original and vir- tual cameras, enabling consistent 3D perception across diverse setups. Our framework serves as plug-and-play module for existing 3D perception methods, improving their robustness to camera parameter variability. We validate our framework in CARLA by training and testing models on different camera configurations, demonstrating that our approach significantly reduces performance degradation while maintaining adaptability across diverse sensor setups. To summarize, we make the following key contributions in this paper:
To the best of our knowledge, UniDrive presents the first comprchensive framework designed to generalize vision-centric 3D perception models across diverse camera configurations.
We introduce a novel strategy that transforms images into unified virtual camera space, enhanc- ing robustness to camera parameter variations.
We propose virtual configuration optimization strategy that minimizes projection error, improv- ing model generalization with minimal performance degradation.
We contribute a systematic data generation platform along with 160,000 frames multi-camera dataset, and benchmark evaluating perception models across varying camera configurations.
2 RELATED WORK
Vision-based 3D Detection. The development of camera-only 3D perception has gained great momentum recently. Early works such FCOS3D (Wang et al., 2021), which extended the 2D FCOS detector (Tian et al., 2020) by adding 3D object regression branches, paved the way for improvements in depth estimation via probabilistic modeling (Wang et al., 2022a; Chen et al., 2022a). Later methods like DETR3D (Wang et al., 2022c), PETR (Liu et al., 2022). and Graph- DETR3D (Chen et al., 2022b) applied transformer-bast based architectures with learnable object queries in 3D space, drawing from the foundations of DETR (Zhu et al., 2021; Wang et al., 2022b), by- passing the limitations of perspective-basee detection. Recent works utilize bird's-eye view (BEV)
2

---

Published as a conference paper at ICLR 2025
for better 3D understanding. BEVDet (Huang et al., 2021) and MBBEV (Xie et al., 2022) effec- tively extended the Lift-Splat-Shoot (LSS) framework (Philion & Fidler, 2020) for 3D object de- tection. CaDDN (Reading et al., 2021) introduced explicit depth supervision in the BEV trans- formation to improve depth estimation. In addition, BEVFormer (Li et al., 2022), CVT (Zhou & Krähenbühl, 2022), and Ego3RT (Lu et al., 2022) explored multi-head attention mechanisms for view transformation, demonstrating further improvements in consistency. To further enhance accu- racy, BEVDet4D (Huang & Huang, 2022), BEVFormer (Li al., 2022), and PETRv2 (Liu et al., 2023a) leveraged temporal cues in multi-camera object detection, showing significant improvements over single-frams, methods.
Cross Domain Perception. The cross-camera configurations problem proposed in this paper lies in the area of cross-domain perception. Domain generalization or adaptation is to enhance model performance on varying domains without re-training. For 2D) perception, numerous cross-domain methods, such as feature distribution alignment and -aaelling (Muandet et al., 2013; Li al., 2018; Dou et al., 2019; Facil et al., 2019; Chen et al., 2018; Xu et al., 2020; He & Zhang, 2020; Zhao et al., 2020), have primarily addressed domain shifts caused by environmental factors like rain or low light. Recent 3D driving perception works (Hao et al., 2024; Peng et al., 2023) focus on transfering the models trained on clean environment or perfect sensor situations to corrupted sensor and noisy environments, leading to several benchmarks and methods. Cross camera configuration is a relatively new topic in this area. While some works (Wang et al., 2023b) find that the model's overfitting to camera parameters can lead to degrade performance because the models learn the fixed observation perspectives, the driving perception across camera parameters has seldom been systematically investigated.
Sensor Configuration. Sensor configurations has been proven important in the design of perception systems (Joshi & Boyd, 2008; Xu et al., 2022). Despite being relatively new in autonomous driving research (Liu et al., 2019). sensor placement has gained significant attention. For instance, Hu et al. (2022) and Li et al. (2024b) were the first to explore multi-LiDDAR setups for improving 3D> object detection, and Li et al. (2024a) studied how combining LiDAR and cameras impacts multi- modal detection systems. Several other studies (Jin et al., 2022; Kim et al., 2023; Cai et al., 2023; Jiang et al., 2023b) focused on the strategic positioning of roadside LiDAR sensors for vehicle-to- everything (V2X) communication, shifting away from in-velicle sensor placements. Although many efforts have aimed to refine sensor configurations for better performance, the challenge of adapting perception models to different sensor setups has been largely overlooked. Our research is the first to explore the generalization of driving perception models across diverse camera configurations.
3 UNIDRIVE
3.1 PROBLEM FORMULATION
In real-world multi-camera driving systems, perception models are typically trained on a specific camera configuration with fixed intrinsic and extrinsic parameters. However, the performance of these models often deteriorates when applied to new camera configurations, where the cameras may have different placements, orientations, or intrinsic properties.
Perception Across Multi-camera Configurations. Given set of cameras C = (Cr,C2..... Cy), each characterized by its intrinsic matrix KS E 3x3 and extrinsic matrix EC₁ E R4x4, where e (I,2...., JJ and J is the number of cameras. The images captured by these cameras are denoted as I E " x F where H' and WC, are the height and width of the image 19, When deploying the model trained on C,...,,CCrr to a new set of cameras {C,, Cg.... Cr.} with different camera numbers and intrinsic and extrinsic parameters, the model may longer effectively understand the 3D scene due to the differences between the training and testing configurations.
Universal Multi-camera Representation. To address the transferability of learned models across camera configurations, we attempt to design a universal representation, which transforms images unified space before input the deep learning network.
from different camera configurations To achieve this, we propose a Virtual Camera Projection approach, which re-projects the views Io, from the original cameras
{C1, C2,.... Cr} into unified set of virtual camera configurations V {V1, V2, ..., VxJ. where K is the number of virtual cameras. The image is represented as Iv2 E R XV where H and VVx are the image sizes, and k E (1,2..... K) indexes the
3

---

Published as a conference paper at ICLR 2025
Plug-and-Play Virtual Camera Projection Module
Optimization
Optimizing Virtual Configuretions
Ground-aware
True 3D Bounding Box
Original View
Depth Assumption
Virtual 3D
Bounding Box
Project
Vision-based
b]
Virtual Camera
Vision-based Perception Model
Iterative Optimization
Driving System
Projection
Crass-conigguration
Paramaters
Deploy
Multi-camera
f] Vision Data
Minimizing Projection Error
Configuration
Synthesis
Figure 2: Overview of UniDrive framework. We transform the input images into unified virtual camera space to achieve universal driving perception. To estimate the depth of pixels in the virtual view for projection, we propose a ground-aware depth assumption strategy. To obtain the most effective virtual camera space for multiple real camera configurations, we propose a data-driven CMA-ES (Hansen, 2016) based optimization strategy. To evaluate the efficacy of our framework, propose automatic data generation platform in CARLA (Dosovitskiy et al., 2017).
virtual camera views. We denote Vv and V the intrinsic and extrinsic matrices for the virtual camera Vi. This virtual configuration serves as standardized coordinate system for both training and inference, allowing the model to operate consistently across different physical camera setups.
3.2 VIRTUAL CAMERA PROJECTION
In this subsection, we explain the Virtual Camera Projection method to project points from mul- tiple camera views onto virtual camera views using a combination of ground and cylindrical sur- face assumptions, as shown in Figure 2. The goal is to learn transformation function Tv-c that maps the images from the original cameras == (V1. .2...., Vx} with minimum errors.
(C1, CC,,,,,..,,CJ to the virtual cameras
Ground-aware Assumption. For each pixel at coordinates (u Vv) in the virtual view, its 3D coordinates in the virtual camera frame (X.+, Y.V., V. are calculated based on the pixel's position the image and the depth assumptions. Let the camera height he, the focal lengths the camera be fV and fV,, and the principal point (image center) be (c)² c. cy²). We first project all pixels to the ground plane to compute the initial assumption of 3D coordinates in virtual camera frame as,
S Vu
SV V.Ve
(xVe, V.v. 2
Cig V/ hos he:
he
1 (v.Va
Ve
Cj
The Euclidean distance optical center is computed D. compare the distance D.== with threshold Do. if D.V. < Do, the points connected to corresponding pixels in the images assumed the ground, SrV,,Y.M. VV (X.², y.V. 2V4.. If D 2 Da. we assume that the points lie on cylindrical-like surface at a fixed distance Do from the camera's optical center. In this case, the 3D coordinates are computed as:
|1 p.V. 2/ Then
(u cV²) Do (0 -cy) Do
Do du
(XV+, V.V. 2.
f
dv
oV dVe
where d%4 =
Vo 2
Point-wise Projection. Once the 3D coordinates (X.,, Y.V, V.v) in the virtual camera frame calculated, we transform the point into the world coordinate system with extrinsic matrix E.V,
4

---

Published as a conference paper at ICLR 2025
Pw =E " where A = (X Y.V. - _)T is the homogeneous coordinate of the point in the virtual camera's 's frame, and Pw E IR is the 3D point in the world coordinate system. Next, we transform the point from the world coordinate system into the original camera's coordinate system
using the inverse of the original camera's s extrinsic matrix Pe E 'Dw. Finally, we project the point back onto the original camera's s 2D image plane using its intrinsic matrix, (u ,,,,)" _1)
KC, - = K EC; Pw- This provides the pixel coordinates (u ()) in the original view that correspond to the pixel (u v"k) in the virtual view. We denote Pvcec, (D as the projection transform matrix from (u0,,00) 0)) in the i-th original view to (n'e, vV)) based on the Euclidean distance to virtual camera optical center D.
Image-level Transformation. The point-wise projection is extended to the entire image view. For each pixel , in the k-th virtual view, we compute the corresponding
Algorithm 1 Virtual Camera Projection
1: Input: {C,, kC EE 1 19 (V2, K',, 3
2: Output: [[
]k=
for k= 1.2, .... K do for (u Vi, Vk in 1 do
pixel (u 00)) in the i-th original view based on the projection matrix Pv.+c, (...). The entire image I of the i-th original view warped into the virtual view Iv--c as follows, I = T(IS, Pv (D.)). where T(I, P) represents the warping
Compute (8. Y: 2. D: using equation
6:
if D. Dg then (X Ye 2 (X y.v, 2²²²)
else
9:
Compute (X V/ Ze" using equation 2
10:
end if
function applied to the image I using
11:
Pw E p p. (X Y:V,, 2e
the projection matrix Pveec, (D.)
12:
G -Es -1 Pw
based on the D.
13:
(u 01))+ +0)+Ks Pe
14:
1v (ulkko"" e- 1 (u
Blending Multiple Views. Since each pixel in single virtual view may have corresponding pixels from various orig- inal view, after transforming each origi- nal view into the virtual view, we merge all the transformed images IV++C, to form the final output image Iv. This blending is performed by computing a weighted sum of all the projected views:
15: end for
16: end for
17:
% t w プラ= Wy 1 using equation
v
Σ wy Pn+c, W Σ wy P = W 1=1
(3)
where W = Σÿー my is the total weight and wy is the blending weight for the j-th original view. The weights can be based on factors such as the angular distance between the original and virtual views, or the proximity of the cameras. We presented the detailed computation process in Algorithm 1.
3.3 VIRTUAL PROJECTION ERROR
To evaluate the accuracy the Virtual Camera Projection method in the context of a 3D object de- tection task, we propose a weighted projection error metric based angular discrepancies between the virtual and original camera views. This method accounts for both angular deviations and the distance from the camera's 's optical center to provide more robust error evaluation.
Angle Computation. Given a driving scenario of 3D bounding box information, for each 3D bound- ing box b, [I Un, =n,m) Jm==1 first project its corner points onto the original camera Cj as the pixel (un.m E m, using the intrinsic matrix KC and extrinsic matrix EC, Then, we use the inverse of the warping process Pv,ec, to find the corresponding pixel Va V. in the virtual camera view Vi for each corner point. We compute the pitch angle 0%, and yaw angle n,m relative to the virtual camera's optical center:
V,
Vi n,m Cy
arctan
fy arctan
u m,m
C
(4)
f
Ve
5

---

Published as a conference paper at ICLR 2025
Preprocess
netwnnk
Virtual Cemere
20 Encoder
2D Feeture
t0 BEV
BeV Encedar
Images
Spoce
Transform
Festure
Figure 3: Integration with Existing Methods. Our virtual camera projection can be integrated into the pipeline as vrepproesssing stage before feeding the multi-view images into the network.
Algorithm Virtual Camera Configuration Optimization
I: Initialize: f + 0, m (0)
(0) C No. Mr vt E 20.
2: for t= 0, 1, 2, ..... T do
for pari= 1 N. do
4: Sample u ~ (m (o(0))(()) from ô-density gird-level candidates
5:
Calculate ((,,
6: end for
7: Update ((+11) based on the top Mr best solutions ti via equation
8: Update (f++) and ((++) via equation 8, equation 9, equation 10, and equation 11
end for
Next, for the same corner points, directly project to the virtual view using K and E as Vul VVVe) Then the pitch angle gVi and yaw angle oiy are n,m
n.m)-
Ver y
Ve
Ve
VoVe n, @
= arctan
arctan
Una
(5)
f
Angle Error Calculation. For each corner point, we compute the angular error between the original camera projection and the corresponding point in the virtual camera. The absolute errors in pitch Vee We use the distance DV/ of nmm each corner point from the original camera's s optical center as weight. The distance is computed >
and yaw are A n.m -0%
8%/. n.m A =
n.m.
as DV22 n.m = (N.V, n,m: y 2V4 )|. The weighted error for each corner point is then calculated as
cVe. = DV.. (A0% n,m + AdkE)) n. The overall error for a 3D bounding box bn is obtained by C,mm summing the weighted errors its eight corner points EV Em= EVe We sum the projection emors across all 3D bounding boxes bn E B to compute the total projection error
n,m
N
N
eV m
(6)
Um
n=1
n=1
m=1
3.4 OPTIMIZING VIRTUAL CAMERA CONHGURATIONS
Given a set of multi-camera systems, we aim to design a unified virtual camera configuration that minimizes the reprojection enror across all original camera configurations. To achieve this, we adopt the heuristic optimization based on the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) (Hansen, 2016) to find an optimized set of virtual camera configurations.
Objective Function. Given multiple driving perception systems with varying multi-camera conf- girations indexed by 8, the total error across all systems is expressed as Exotal Σ 15-,, g(4) (u), where u = (Ve, KV,, E } includes both the intrinsic and extrinsic camera parameters vir- tual multi-camera framework, K is the total quantity of virtual cameras and S is the total quantity of multi-camera driving systems that share the same perception model. We aim to minimize this error by sampling and updating the virtual camera parameters iteratively through CMA-ES based optimization method.
Optimization Method. Our Optimization strategy begins by defining a multivariate normal distri- bution A (m) ((()), where (!)) represents the mean vector, (4) denotes the step size, and ( is the covariance matrix at iteration f. The configuration space 2/ is discretized with density

---

Published as a conference paper at ICLR 2025
DE
(a) 4 95
75
80%a
(d) 6 80%
70"
(f) 6 60
50
(h) 5 x 70 + 110
(i w/o optimization
w/ optimization
Figure 4: Visualized multi-camera configurations. We illustrate the multi-view camera configu- rations used in our study. There configurations are inspired by practical applications in the industry.
and N, candidate configurations u, (4) (m (o(1))((())) are sampled each iteration Initialization begins with the initial mean m (0), step size o0) and covariance matrix C I. The updated mean vector (t-1) is calculated in the subsequent iteration serve the center for the search distribution concerning the virtual camera contiguration. The process can be mathemati- cally expressed as:
Mc
Σ w.u
E(u 2 E(ul)) 2. ... ((
where M, is the number of top solutions selected to update ((+1), and w; weights determined by solution performance. The evolution path PC (++1) which tracks the direction of successful opti- mization steps, is updated as:
m (++1) m (C)
(++1) -1 PC Pc
(8)
(())
where ec is the learning rate for updating the covariance matrix. The covariance matrix C, which defines the distribution's ' shape for camera configurations, is adjusted at each iteration as follows:
((++1) (c) + ccPe
(++1) (++1)T
Pe
Similarly, the evolution path for the step size, Pa. is updated, and the global step size is then adjusted to balance exploration and exploitation:
m (1+1) (()
De (++1) (1 Crr (p + V
(1 c2)
(10)
i=
M.
wp
(())
(++1) (1)) exp
Ca
l|po
(e+1)
)
(11)
dr
where 6a is the learning rate for updating Pa, and d normalization factor controlling the ad- justment rate of the global step size. We presented the detailed optimization process in Algorithm 2.
4 EXPERIMENTS
4.1 BENCHMARK SETUPS
Data Generation. We generate multi-view image data and 3D objects ground truth in CARLA simulator (Dosovitskiy et al., 2017). We use the maps of Towns 1-6 to collect data. We incorporate 6 classes for 3D object detection, including Car, Bus, Truck, Motorcycle, Bicycle, and Pedestrian. The dataset consists of 500 scenes (20,000 frames) for each camera configuration. We split 250 scenes

---

Published as a conference paper at ICLR 2025
Table I: Quantitative results of BEVFusion-C for 3D detection across camera contigurations. The detector is trained on the blue configurations and tested on configurations directly. We report the mAP (1) and class-level AP (1) scores in percentage (%).
Configurations
mAP
Car
Bus
Truck
Ped.
Motor
Bic.
Configurations mAP
Car
Bax
Truck
Ped.
Motor
Bic.
5 x 70" 110
63.9
62.4
58.0
66.5
54.7
68.7
72.9
6 x
69.3
68.7
67.4
66.3
62.2
78.4
72.8
4 x 95
4,9
4.6
5,1
3.8
3,9
3.1
4.2
95
3.8
4.1
4.3
3.2
4.1
3.3
3.6
5 75
4.8
6.2
5.1
8.3
75
3.4
3.7
3.1
2.7
80
11.7
8.8
8.4
8.2
7.7
x 80%a
0.6
1.7
0.4
0.5
0.1
x 80"b
6.9
10.0
7.2
7.8
5.2
6.1
5.6
x 80%b
0.4
144
0.0
0.7
0.0
0.2
0.1
6 70"
67.5
65.2
61.2
69.3
57.9
79.5
72.1
x 70"
8.1
9.6
4.3
6.8
7.1
11.0
10.0
x 60"
12.4
7.0
8.0
6.5
11.9
9.4
70"
110"
4.6
4.9
3.5
3.4
5,4
50"
0.1
0.3
0.6
17.3
18.5
14.1
16.7
66.2
63.7
75.9
72.9
80%
69.1
66.0
72.1
58.3
78.6
4 x 95
3.8
4.3
5.0
3.6
3.2
2.8
3.9
4 95
3.5
3.9
4.1
3.3
3.2
2.6
3.7
5 x 75
30.4
31.2
23.4
27.8
28.6
36.9
34.2
5 x 75
29.6
30,3
22.6
27.1
27.9
36.3
33.2
5 709 110"
6.5
8.6
8.8
13.3
80%a
63.2
65.5
67.0
66.4
46.7
68.2
65.1
80"b
63.8
70.9
46.3
68.1
65.4
60
1.7
2.9
1.1
0.7
6 x 70"
16.4
18.0
9.4
13.3
14.7
22.2
20.6
x 70"
16.1
17.7
8.3
12.3
14.6
23.7
19.9
6 x 60"
1.8
3,3
0.8
1.5
0.6
2.3
2.4
70" + 110"
8.9
10,3
5.6
7.5
7.1
9.6
1344
x 50"
0.0
0.8
0.1
0.1
0.6
50
0.2
0.4
0.9
0.3
for training and 250 scenes for validation. Our dataset is organized as the format of nuScenes (Caesar al., 2020) and compatible to the nuscenes-devki t. python package for convenient processing.
Camera Configurations. We adopt several commonly used camera configurations in automotive practice with various camera quantities, placements and field of views. These configurations are represented in Figure 4. We set all camera resolutions to 1 600 x 900 as nuScenes. Our camera configurations include camera numbers from we conduct study mainly on 6 cameras with FOV = 60, 70, 80. For the placement, we design two different types of placement as shown in Figure 4 (c), (d). We also include the original configurations of nuScenes (Caesar et al., 2020) dataset with five 70% cameras and 110" camera.
For the field of view (FOV) for cameras,
Deteciton Method. Due to the extensive computation resource needed benchmark the multi- camera configurations, we only compare our method with the camera variant of BEVFusion (Liu et al., 2023b) (abbreviated as BEVFusion-C) BEVFusion is one of the representative methods in many leaderboards, such nuScenes (Caesar al., 2020) and Waymo (Sun al., 2020).
4.2 COMPARATIVE STUDY
We conduct comparative studies to evaluate the performance of camera perception across configu- rations. Through our analysis, we able demonstrate the effectiveness of UniDrive framework.
Effectiveness of UniDrive. In Table 1 and 2, we present the 3D object detection results of BEVFusion-C (Liu al., 2023b) and UniDrive. The models are trained on one configuration and tested on other varying camera configurations. The performance of BEVFusion-C degrades a lot when deployed on cross-camera configuration tasks, nearly unusable shown in Table 2, we train the models using our plug-and and-play UniDrive framework. The detection performance significantly improves compared to BEVFusion-C C (Liu al., 2023b). Our method only experiences little performance degradation cross-camera configuration tasks. We present more results in Figure 5, which comprehensively shows the effectiveness of our framework.
other configurations. As
Optimization via UniDrive. To demonstrate the importance of optimization in UniDrive, we com- pare the perception performance between optimized virtual camera configurations and intuitive one in Figure 5. The intuitive virtual camera configuration places all cameras in the center of the vehicle roof. As shown in Figure 5 (b), although the intuitive setup (without optimizing) also significantly improved cross-camera configuration perception performance compared to BEVFusion-C (in Fig- ure (a)), exhibited a clear preference for certain configurations while performing poorly on others. In contrast, the optimized virtual camera parameters (in Figure 5 (c)) demonstrated greater
8

---

Published as a conference paper at ICLR 2025
Table 2: Quantitative results of UniDrive for 3D detection across camera configurations. The detector is trained on the blue configurations and tested on configurations directly. We report the mAP (1) and class-level AP (f) scores in percentage (%).
Configurations mAP
Car
Bus
Truck
Ped.
Motor
Bic.
Configurations mAP
Car
Bax
Truck
Ped.
Motor
Bic.
5 x 70" 110
68.8
67.5
64.8
71.9
59.1
73.6
75.9
x 60"
64.6
63.4
58.2
59.7
59.2
76.7
70.0
4 x 95
60.1
59.1
57.2
58.4
59.2
68.6
67.8
95
58.9
57.7
54.1
56.9
53.1
66.2
67.4
5 x 75
66.7
64.5
65.9
67.8
59.6
72.3
70.1
x 75
62.2
59.5
60.6
65.8
53.6
70.3
63.1
80"
69.4
68.4
68.1
67.5
57.8
78.2
76.1
x 80%a
64.4
65.1
64.9
65.5
53.3
70.2
67.1
6 x 80"b
65.8
64.7
62.0
63.9
55.8
76.7
71.7
x 80%b
65.7
63.2
63.1
63.9
55.8
76.7
71.7
6 x 70"
68.4
66.8
64.3
69.8
57.6
79.1
72.8
6 x 70P
65.0
62,9
60.8
65.4
55.1
73.0
72.8
x 60"
63.1
60.6
57.4
58.8
59.2
73.0
69.6
70" + 110"
63.6
61.2
61.4
64.8
55.2
71.0
68.1
x 50"
55.4
56.1
69.7
64.1
63.8
60.2
62.6
56.8
74.2
70.5
67.7
66.6
76.1
80% b
63.1
63.2
53.4
71.1
4 x 95
55.9
56.4
58.4
52.7
48.3
59.5
60.1
4 95
53.6
51.2
48.0
49.7
$1.0
61.5
60.1
5 x 75
65.2
63.6
64.8
66.8
57.0
70.9
68.3
5 75
62.7
62.1
60.6
62.2
56,3
69.9
67.0
5 709 110"
63.7
61.3
58.9
60.5
72.8
68.8
80%a
64.5
62.6
62.4
58.6
714
71.6
80"6
66.2
65.2
61.1
65.1
56.1
76.3
73.2
60
62.6
60.3
59.1
62.6
53.4
70.2
70.2
6 x 70"
68.9
67.9
63.5
70.7
58.3
79.5
73.8
x 70"
62.5
59.4
55.6
62,9
52.6
73.4
70.8
6 x 60"
59.6
57.2
54.0
55.7
57.0
67.5
66.1
70" + 110"
57.9
56.1
52.5
53.7
56,9
64.3
63.7
8 x 50"
61.2
60.3
58.1
39.9
54.5
68.2
65.9
50
58.4
60.3
57.0
54.3
53.9
64.4
60.2
adaptability, showing relatively consistent performance across various configurations. This is crucial for the concurrent development of multiple multi-camera perception systems in autonomous driving.
4.3 ABLATION STUDY
In this section, we analyze the interplay between our proposed virtual projection strategy and percep- tion performance address these questions: 1) What's 's the impact of camera extrinsic and intrinsic for cross-configuration perception? 2) How UniDrive works towards these parameters separately?
Camera Intrinsies. Changes in camera intrinsics pose the greatest challenge for cross-camera pa- rameter perception. In Figure 5 (a), BEVFusion-C almost entirely fails when tasked on distinct camera intrinsies with the detection accuracy mostly under 20%. For instance, BEVFusion only gets 1.8% when deploying models trained on 6 x 80% a to 6 x 60%. In contrast, in Figure 6, our UniDrive framework demonstrates substantial robustness, with performance dropping by at most 9.8% under the largest intrinsic differences, which highlights the effectiveness of our approach.
Camera Height. The variation in the vertical position of cameras can significantly impact percep- tion performance, cameras at varying heights capture images with distinct geometric features. We perform experiments specifically for varying camera heights at 1.6 meters, 1.4 meters, 1.8 meters, and 2.5 meters. We train the model 1.6 meters and test other configuraitons. As shown in Fig. 6b. BEVFusion-C experiences a substantial performance drop for more than 10%, when faced with varying camera heights. In contrast, UniDrive significantly improves performance across dif- ferent camera heights, demonstrating enhanced robustness with only 3.0% performance decreasing.
Camera Placement. Changing the camera's horizontal position and orientation on presents relatively smaller challenge for cross-camera parameter perception. As shown in Figure 5 (a), BEVFusion-C experiences a performance drop of 5.9% when deploying the model trained on the 6 x 8 configuration to the 6 x 80%a configuration. Nonetheless, our UniDrive framework further enhances cross-cament parameter perception performance. In Figure (c), we train the model on the 6 x 80% a configuration and test on other configurations, UniDrive only experiences 4.6% when deploying the model trained on the 6 x 80% configuration to the 80% configuration.
4.4 ANALYSIS
In this section, we further investigate some useful insight points found in the benchmark experi- ments: 1) What's 's the impact of inconsistency in intrinsics for perception? 2) How UniDrive works fowards this inconsistency?
9

---

Published as a conference paper at ICLR 2025
Dowt
Lxst
4x30
4.93
4:50
5675
47.5
16.1
16.0
6633
66.
6644
02.2
14.
0.6
24.0
62.
654
S4C
6:80a
R41
221
6e8Db
1639
8x80b
40.81
42.0
46
122
04
6622
62.
40.
bomor/10
0
0.7
6x50
4
Erain
Satber
0.50
0.70
4,5
Batza
8.7
(a) DEVFusion-C (Liu et al., 2023b)
(b) UniDrive (w/o optimization)
(c) UniDrive (w/ optimization)
Figure 5: Performance evaluations of BEVFusion-C and UniDrive on object detection across camera configurations. We report the mAP (1) scores in percentage (%).
BEVFusion-C
UniDrive
75
BEVFusionC
UniiDrive
BEVFueionC
UniDrive
100
80
08.7
68.8
50.6
70
68.7
69.4
64.2
694
6313
67.2
68.4
70
60.7
894
63.3
662
4
04.88
65
65
10.4
56.2
1.0
55
Ex80s
6:75
6:1.8
8:25
Ex80s
ExBCb
6:90:
(a) Camera intrinsic
(b) Camera height
(c) Camera placement
Figure 6: Ablation Study of BEVFusion-C and UniDrive on 3D object detection across camera configurations. We report the mAP (f) scores in percentage (%).
Degradation with Inconsistent Intrinsics. In our experiments, we observed that for multi-camera systems, models perform better when camera intrinsies consistent compared to when they vary. However, due to design aesthetics and other constraints, many autonomous driving companies use multiple cameras with different intrinsic parameters to achieve 360-degree perception. For instance, the nuScenes (Caesar et al., 2020) uses five 70" cameras and one 110e camera. As shown in Fig 1, BEVFusion- performs lot better in 6 80% a and 6600 compared to configuration 5 70%+710+10 +110%. Thus, inconsistency in camera intrinsics can potentially hinder perception improvement.
Improvement via UniDrive. Our framework significantly enhances the perception performance of multi-camera systems with varying intrinsics by leveraging virtual camera system with consis- tent intrinsics. For training and testing on the same configurations, as demonstrated in Figure 2, UniDrvie achieves 68.8% accuracy in 5 x 70 + 110" configuration, which surpasses 4.9% than BEVFusion-C C (63.9%). For testing across camera configurations, UniDrive experiences little accu- racy reduction only in rare situations. This demonstrates that UniDrive has substantial potential to push advancements driving perception technology.
5 CONCLUSION
In this paper, we introduce the UniDrive framework, a robust solution for enhancing the gener- alization of vision-centric autonomous driving models across varying camera configurations. By leveraging unified set of virtual cameras and ground-aware projection method, our approach ef- fectively mitigates the challenges posed by camera intrinsics and extrinsics. The proposed virtual configuration optimization ensures minimal projection error, enabling adaptable and reliable perfor- mance across diverse sensor setups. Extensive experiments in CARLA validate the effectiveness of UniDrive, demonstrating strong generalization capabilities with minimal performance loss. Our framework not only serves as a plug-and-plaa module for existing 3D perception models but also paves the way for more versatile and scalable autonomous driving solutions.
The camera configurations analyzed this paper can not cover all real-world setups, more comprehensive experiments may be required. In addition, our research fully conducted on experiments are time-consuming and need extensive resource.
Limitation.
simulation data, as
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving. In CVPR, pp. 11621-11631, 2020.
Xinyu Cai, Wentao Jiang, Runsheng Xu, Wenquan Zhao, Jiaqi Ma. Si Liu, and Yikang Li. Analyzing infrastructure lidar placement with realistic lidar simulation library. In ICRA, pp. 5581-5587, 2023.
Hansheng Chen, Pichao Wang, Fan Wang, Wei Tian, Lu Xiong, and Hao Li. Epro-pnp: Generalized end-to-end probabilistic perspective-n-poing for monocular object pose estimation. In CVPR, pp. 2781-2790 2022a.
Yuhua Chen, Wen Li, Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Domain adaptive faster r-cnn for object detection in the wild. In CVPR, pp. 3339-3348, 2018.
Zehui Chen, Zhenyu Li, Shiquan Zhang, Liangji Fang, Qinhong Jiang, and Feng Zhao. Graph- detr3d: rethinking overlapping regions for multi-view 3d object detection. ACM MM, pp- 5999-6008, 2022b.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In CoRL, pp. 1-16, 2017.
Qi Dou, Daniel Coelho de Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain generaliza- tion via model-agnostic learning of semantic features. NeurlPS, 32, 2019.
Jose Facil, Benjamin Ummenhofer, Huizhong Zhou, Luis Montesano, Thomas Brox, and Javier Civera. Cam-convs: Camera-aware multi-scale convolutions for single-view depth. In CVPR, pp- 11826-11835, 2019.
Nikolaus Hansen. The ema evolution strategy: A tutorial. arXiv preprint arXiv: 2016.
Xiaoshuai Hao, Mengchuan Wei, Yifan Yang, Haimei Zhao, Hui Zhang, Yi Zhou, Qiang Wang. Weiming Li, Lingdong Kong, and Jing Zhang. Is your hd map constructor reliable under sensor corruptions? arXiv preprint 2xXv:2446.122144 2024.
Zhenwei He and Lei Zhang. Domain adaptive object detection via asymmetric tri-way faster-renn. In ECCV, pp. 309-324, 2020.
Hanjiang Hu, Zuxin Liu, Sharad Chitlangia, Akhil Agnihotri, and Ding Zhao. Investigating the impact of multi-lida placement object detection for autonomous driving. In CVPR, pp. 2550- 2559, 2022.
Yihan Hu, Jiazhi Yang. Li Chen, Keyu Li, Chonghao Sima. Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oricnte autonomous driving. In CVPR, pp. 17853- 17862, 2023.
Junjie Huang and Guan Huang. Bevdet4d: Exploit temporal cues multi-camera 3d object detec- tion. arXiv preprint arXiv:2203.1703. 2022.
Junjie Huang. Guan Huang, Zheng Zhu, Yun Ye, and Dalong Du. Bevdet: High-performance multi- camera 3d object detection in bird-eye-view. arXiv preprint 12. 2021.
Yuanhui Huang, Wenzhao Zheng, Yumpeng Zhang, Jie Zhou, and Jiwen Lu. Tripperspective view for vision-based 3d semantic occupancy prediction. In CVPR, pp. 9223-9232, 2023.
Yuanhui Huang, Wenzhao Zheng, Borui Zhang, Jie Zhou, and Jiwen Lu. Selfoce: Self-supervised vision-based 3d occupancy prediction. In CVPR, pp. 19946-19956, 2024a.
Yuanhui Huang, Wenzhao Zheng, Yunpeng Zhang, Jie Zhou, and Jiwen Lu. Gaussianformer: Scene as gaussians for vision-based 3d semantic occupancy prediction. arXiv preprint arXiv:2405. 17429, 2024b.
11

---

Published as a conference paper at ICLR 2025
Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vectorized scene representation for efficient au- tonomous driving. arXiv preprint 1.XXxv:2333.207777 12077, 2023a.
Wentao Jiang, Hao Xiang, Xinyu Cai, Runsheng Xu, Jiaqi Ma, Yikang Li, Gim Hee Lee, and Si Liu. Optimizing the placement of roadside lidars for autonomous driving. In ICCV, pp. 18381-18390, 2023b.
Shaojie Jin, Ying Gao, Fei Hui, Xiangmo Zhao, Cheng Wei, Tao Ma, and Weihao Gan. A novel information theory-based metrie for evaluating roadside lidar placement. IEEE Sensors Journal, (2(21))21009-2102) 2022.
Siddharth Joshi and Stephen Boyd. Sensor selection via convex optimization. IEEE Transactions on Signal Processing, 57(2):451 2008.
Tae-Hyeong Kim, Gi-Hwan Jo, Hyeong-Seok Yun, Kyung-Su Yun, and Tae-Hyoung Park. Place- ment method of multiple lidars for roadside infrastructure in urban environments. Sensors, 23 (21):8808, 2023.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex Kot. Domain generalization with adversarial feature learning. In CVPR, pp. 5400-5409, 2018.
Ye Li, Hanjiang Hu, Zuxin Liu, Xiaohao Xu, Xiaonan Huang, and Ding Zhao. Influence of camera- lidar configuration on 3d object detection for autonomous driving. In ICRA, pp. 9018-9025, 2024a.
Ye Li, Lingdong Kong, Hanjiang Hu, Xiaohao Xu, and Xiaonan Huang. Is your lidar placement optimized for 3d scene understanding? In NeurlPS, 2024b.
Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird s-eye-view representation from multi-camera images via spa- tiotemporal transformers. In ECCV, pp. 1-18, 2022.
Feng Liu, Tengteng Huang, Qianjing Zhang, Haotian Yao, Chi Zhang, Fang Wan, Qixiang Ye, and Yanzhao Zhou. Ray denoising: Depth-awan hard negative sampling for multi-view 3d object detection. In ECCV, 2024.
Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position embedding transformation for multi-view 3d object detection. In ECCV, pp. 531-548, 2022.
Yingfei Liu, Junjie Yan, Fan Jia, Shuailin Li, Aqi Gao, Tiancai Wang, and Xiangyu Zhang. Petrv2: A unified framework for 3d perception from multi-camera images. In ICCV, pp. 3262-3272,
2023a.
Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela L. Rus, and Song Han. Bevfusion: Multi-task multi-sensor fusion with unified bird's-eye view representation. In ICRA, pp. 2774-2781, 2023b.
Zuxin Liu, Mansur Arief, and Ding Zhao. Where should we place lidars on the autonomous vehicle?-
an optimal design approach. ICRA, pp. 2793-2799, 2019.
Jiachen Lu, Zheyuan Zhou, Xiatian Zhu, Hang Xu, and Li Zhang. Learning ego 3d representation as ray tracing. In ECCV, pp. 129-144, 2022.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In ICMI,, pp. 10-18, 2013.
Xidong Peng, Runnan Chen, Feng Qiao, Lingdong Kong, Youquan Liu, T Wang. X Zhu, and Y Ma. Learning to adapt sam for segmenting cross-domain point clouds. arXiv preprint arXiv:2310.08820, 2023.
Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting 3d. In ECCV, 2020.
12

---

Published as a conference paper at ICLR 2025
Cody Reading, Ali Harakeh, Julia Chae, and Steven L Waslander. Categorical depth distribution 2021.
network for monocular 3d object detection. In CVPR, Pp.
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR, pp. 2446-2454, 2020.
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos: simple and strong anchor-free object detector. TPAMI, 44(4):1922-1923 1933, 2020.
Shihao Wang, Yingfei Liu, Tiancai Wang, Ying Li, and Xiangyu Zhang. Exploring object-centric temporal modeling for efficient multi-view 3d object detection. In ICCV, pp. 3621-3631, 2023a.
Shuo Wang, Xinhai Zhao, Hai-Ming Xu, Zehui Chen, Dameng Yu, Jiahao Chang, Zhen Yang, and Feng Zhao. Towards domain generalization for multi-view 3d object detection in bird-eye-view. In CVPR. pp. 13333-13342 2023b.
Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin. Feos3d: Fully convolutional one-stage monocular 3d object detection. In ICCV, pp. 913-922, 2021.
Tai Wang, ZHU Xinge, Jiangmiao Pang, and Dahua Lin. Probabilistic and geometric depth: Detect- ing objects in perspective. In CoRL, pp. 1475-1485, 2022a.
Yingming Wang, Xiangyu Zhang, Tong Yang, and Jian Sun. Anchor detr: Query design for transformer-batt based detector. In Proceedings of the AAAI conference on artificial intelligence, 2022b.
Yue Wang. Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In CoRL, pp. 180-191, 2022c.
Yi Wei, Linging Zhao, Wenzhao Zheng, Zheng Zhu, Jie Zhou, and Jiwen Lu. Surroundoce: Multi- camera 3d occupancy prediction for autonomous driving. In ICCV, pp. 21729 -17400, 2023.
Xie, Z Yu, D Zhou, J Philion, A Anandkumar, S Fidler, Luo, and JM Alvarez. M2bev: Multi- camera joint 3d detection and segmentation with unified birds-eye view representation. arXiv preprint arXiv:2204.0500
2022.
Chang-Dong Xu, Xing- Ran Zhao, Xin Jin, and Xiu-Shen Wei. Exploring categorical regularization for domain adaptive object detection. In CVPR, pp. 11724-11733, 2020.
Xiaohao Xu, Zihao Du, Huaxin Zhang, Ruichao Zhang, Zihan Hong, Qin Huang, and Bin Han. Optimization of forcemyography sensor placement for arm movement recognition. IROS, pp- 9845-98500 2022.
Shuai Zeng, Wenzhao Zheng, Jiwen Lu, and Haibin Yan. Hardness-aware scene synthesis for semi- supervised 3d object detection. TMM, 2024.
Ganlong Zhao, Guanbin Li, Ruijia Xu, and Liang Lin. Collaborative training between region pro- posal localization and classification for domain adaptive object detection. In ECCV, pp. 86-102, 2020.
Linqing Zhao, Xiuwei Xu, Ziwei Wang, Yunpeng Zhang, Borui Zhang, Wenzhao Zheng, Dalong Du, Jie Zhou, and Jíwen Lu. Lowrankoce: Tensor decomposition and low-rank recovery for vision-based 3d semantic occupancy prediction. In CVPR, pp. 9806-9815, 2024.
Wenzhao Zheng, Weiliang Chen, Yuanhui Huang, Borui Zhang, Yueqi Duan, and Jiwen Lu. Occ- world: Learning a 3d occupancy world model for autonomous driving. In ECCV, 2024a.
Wenzhao Zheng, Ruiqi Song, Xianda Guo, and Long Chen. tonomous driving. In ECCV, 2024b.
Genad: Generative end-to-end au-
13

---

Published as a conference paper at ICLR 2025
Brady Zhou and Philipp Krähenbühl. Cross- view transformers for real-time map-view semantic segmentation. In CVPR, pp. 13760-13769, 2022.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2021.
Zhuofan Zong, Dongzhi Jiang, Guanglu Song, Zeyue Xue, Jingyong Su, Hongsheng Li, and Yu Liu. Temporal enhanced training of multi-view 3d object detector via historical object prediction. In ICCV, pp. 3781-3790, 2023.
14

---

Published as a conference paper at ICLR 2025
A VISUALIZATION
We present the visualization results of the virtual camera projection in Figure 7. Overall, the warping from the original view the virtual view is highly accurate. Only few areas are warped because the original cameras lack coverage of those regions.
Original
Wrapped Multi-view Images
6x80a
6x60
E
5x70+110
5x
Figure Wrapped Multi-view Images.
15


Reviews:
[{"review_id": "v3N9QIbYcH", "reviewer": "Reviewer_sBiy", "summary": "The paper addresses the challenge of maintaining robustness in perception algorithms for autonomous driving systems as camera configurations—such as intrinsics, extrinsics, and the number of cameras—evolve. The authors introduce a novel re-mapping approach that transforms pixels from the original camera view to a unified virtual view, allowing training and inference to be conducted in this virtual space. This approach improves robustness across different configurations and integrates seamlessly into existing perception pipelines. \nIn addition, the paper proposes a virtual camera configuration optimization method, enabling the identification of an optimal virtual configuration based on real-world multi-camera systems to further enhance robustness across configurations. \nExperimental results on synthesized data show that the method not only enhances cross-configuration performance but also unexpectedly boosts performance on the same configurations.", "questions": "Please check weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper addresses a valuable real-world demand. As autonomous driving sensor configurations (e.g., camera intrinsics, extrinsics, and the number of cameras) evolve, it becomes crucial to ensure the perception algorithms are robust to such changes.\n- The authors propose a re-mapping approach that transforms pixels from the original view into a unified virtual view, enabling training and inference to be conducted on this virtual view. This approach enhances robustness to cross-configuration shifts in camera systems. Furthermore, the method is modular, making it straightforward to integrate into existing perception pipelines.\n- A virtual camera configuration optimization method is also introduced, which can quantitatively find a better virtual camera configuration based on a set of real-world multi-camera system to enhance the cross-configuration robustness and perception performance. \n- Experimental results demonstrate that the proposed approach significantly improves robustness across different configurations. Interestingly, performance also improves on configurations identical to the training setup. This is surprising, as one might assume additional transformations could lead to information loss rather than enhancement. This raises questions about whether these improvements are dataset-dependent (I posed in Weaknesses).\n- The paper presents a well-formulated problem and demonstrates effective solutions under simulated data conditions.\n\nDespite some limitations, especially those related to simulated data, this paper introduces compelling ideas, offering a well-defined problem formulation that could benefit the autonomous driving industry and potentially embodied AI.", "weaknesses": "- As the authors noted, one main limitation is that all analyses were conducted in a simulated environment. While this controlled setting is useful for evaluating the method’s potential, it raises concerns about real-world applicability, as training-based detectors are often sensitive to dataset-specific factors. Given the paper’s real-world motivation, validation on actual datasets would be valuable.\n- As I posed in Strengths, it is a little counterintuitive that the transformation from the original view to a virtual view may cause information loss but can bring improvements. The authors should consider proposing hypotheses to explain why this transformation improves performance.\n\nApart from the issues I posed above, I also have some suggestions that will not prevent this paper to be accepted if unresolved. In real-world applications, there are several datasets of different camera configurations. As the proposed method can leverage the virtual view to unify these different configurations, I am curious whether the proposed method can leverage these datasets to train a better-to-all detector on each dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xhxYsbRgEu", "reviewer": "Reviewer_rjYz", "summary": "The paper addresses the challenge of achieving consistent 3D perception in autonomous driving across different camera configurations. It proposes the UniDrive framework, which creates a unified virtual camera space using ground-aware projections and configuration optimization to reduce projection errors. Experiments in simulation demonstrate that UniDrive improves perception robustness across varied camera setups with minimal performance loss.", "questions": "To what extent does the reliance on simulated data affect the generalizability of the results to real-world settings, particularly in urban driving scenarios?\n\nDoes the framework introduce any latency or computational overhead that could impact its feasibility for real-time applications in autonomous driving?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper addresses a valuable and relevant problem in autonomous driving, focusing on camera configuration generalization.\n\nThe proposed approach sounds generally reasonable, employing a unified virtual camera space and ground-aware projection to help manage variability in camera setups.\n\nExperimental results show clear performance gains, with UniDrive maintaining high perception accuracy across diverse camera configurations and outperforming baseline models.", "weaknesses": "W1: The innovation is somewhat limited, as the framework mainly leverages established virtual projection and optimization techniques without presenting substantially novel concepts.\n\nW2: The approach relies heavily on simulated environments (e.g., CARLA), raising concerns about its applicability and robustness in real-world conditions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x0hoCukh9k", "reviewer": "Reviewer_3zcR", "summary": "I greatly appreciate this paper; in fact, I had similar ideas that I hadn't implemented. Seeing the author perfectly execute this idea and achieve excellent results is truly exciting. The paper proposes transforming images into a unified virtual camera space with a virtual configuration optimization strategy, thereby enhancing robustness to different camera configurations.", "questions": "1. Does the virtual configuration optimization strategy actually work effectively? （Core concern: I am highly willing to increase the score if this issue is explained thoroughly.）\n\n   1.1 In certain camera configurations, the results from a single dataset in the paper are not as good as those of the original BEVFusion-C (e.g., 6x60 69.3 vs. 64.6, 6x80b 69.1 vs. 63.1). Why can't the virtual configuration optimization improve to achieve a virtual camera space as good as or better than the original BEVFusion-C?\n\n   1.2 In some camera configurations, the results of individual optimization are not as good as those from transfer learning, such as 60x80b (train on 60x80b 63.1 vs. train on 60x70 67.9). Why can't the virtual configuration optimization achieve a virtual camera space as good as or better than the transfer setting?\n\n   1.3 For different camera configurations, what do the optimized Virtual Cameras look like? Can you provide the optimized intrinsics and extrinsics?\n\n   1.4 How well does the original view warp to the virtual view? Are there any areas that are not properly warped? Can you provide visual results?\n\n2. While BEVFusion is popular, the paper focuses on vision-centric autonomous driving. It would be beneficial to include experiments with popular detectors in the camera domain, such as BEVFormer and PETR. These differ from methods like BEVFusion-C and BEVDet, as they are based on query-based transformer solutions.\n\n3. Were the number of virtual cameras optimized together? If not, could you provide ablation experiments?\n\n4. The paper discusses that Inconsistent Intrinsics have a significant impact. Since real autonomous vehicles also have inconsistent heights, could you include an experimental analysis?\n\n   4.1 Were the heights of virtual cameras optimized together?\n\n5. Given the various camera configurations, would training on datasets that mix various configurations to optimize a unified virtual camera space lead to improved results?\n\n6. All experiments are conducted on simulation data. Incorporating transfer experiments with real datasets would be more convincing, like the real dataset transfer experiments conducted in 3D Domain Generalization/Adaptation.\n\nNote: I have provided several questions. The authors can prioritize addressing the core issues based on their time constraints, and it is not necessary to conduct all experiments.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes to transform images into a unified virtual camera space, improving robustness across various camera configurations.\n2. Additionally, the paper proposes a virtual configuration optimization strategy that minimizes projection errors.\n3. The paper also presents a systematic data generation platform and a benchmark for evaluating perception models under different camera configurations.", "weaknesses": "See the Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FGrXYWydz0", "reviewer": "Reviewer_DD4T", "summary": "Camera-only 3D object detection is sensitive to camera settings. It has been proved that a detection model trained on a camera setting fails to provide satisfactory performance in another camera setting. The authors propose a method, UniDrive, to unify this. The proposed UniDrive comprises a set of virtual cameras, which wrap real cameras into virtual space and thus bridge the gap between different camera settings. It is worth noticing that the number of real and virtual cameras does not need to be the same. The authors also proposed a virtual projection error, which enables an optimization process to find the optimal virtual camera setting. Experiments are conveyed on the CARLA simulator and the BEVFusion-C model. It is shown that, with the proposed method, model performance on various camera settings is significantly improved.", "questions": "- As mentioned in weaknesses, the effectiveness of the proposed method should be testified on multiple camera-only 3D detection methods.\n- It is mentioned in line 302 that the virtual camera configuration is obtained through an optimization process based on a given set of camera systems; it is curious to know if the proposal method also works toward a randomly picked virtual camera setting. This would testify to the effectiveness of the proposed metric and optimization process in Sections 3.3 and 3.4. It is recommanded to compare the optimized virtual camera configuration to randomly selected configurations and evaluate performance across different camera settings.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The article is clear and easy to follow.\n- The motivation of the article is a long-standing problem in the 3D detection domain, and the authors propose a feasible way for solving the problem.\n- Mathematical details are given for the projection from virtual cameras to real cameras.\n- In the experiments, the proposed method is testified on seven different camera settings, showing its generalization to different camera settings.", "weaknesses": "- Only BEVFusion-C is used in the experiments. In line 407, the authors mentioned that BEVFusion-C is one of the SOTA methods in popular public leaderboards, which is not currently the case. The effectiveness of the proposed method should show its universal capability on multiple methods with at least 2-3 different camera settings, such as some methods spercifically designed camera-only detection like BEVFormer [1] and Sparse4D [2].\n- It is not spercifically described that how to adopt the proposed method to exsiting network. It would be helpful as the proposed method serves as a plug-and-play component.\n\n\n[1] Li, Zhiqi, et al. \"Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers.\" European conference on computer vision. Cham: Springer Nature Switzerland, 2022.\n\n[2] Sun, Wenchao, et al. \"SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representation.\" arXiv preprint arXiv:2405.19620 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)