Cross-Dataset Sensor Alignment: Making Visual 3D Object Detector Generalizable
Authors: Liangtao Zheng
Yicheng Liu
Yue Wang
Hang Zhao
Venue: CoRL 2023 Poster (2023)
Keywords: 3D object detection
Model Generalization
Autonomous Driving

Abstract: While camera-based 3D object detection has evolved rapidly, these models are susceptible to overfitting to specific sensor setups. For example, in autonomous driving, most datasets are collected using a single sensor configuration. This paper evaluates the generalization capability of camera-based 3D object detectors, including adapting detectors from one dataset to another and training detectors with multiple datasets. We observe that merely aggregating datasets yields drastic performance drops, contrary to the expected improvements associated with increased training data. To close the gap, we introduce an efficient technique for aligning disparate sensor configurations ---a combination of camera intrinsic synchronization, camera extrinsic correction, and ego frame alignment, which collectively enhance cross-dataset performance remarkably. Compared with single dataset baselines, we achieve 42.3 mAP improvement on KITTI, 23.2 mAP improvement on Lyft, 18.5 mAP improvement on nuScenes, 17.3 mAP improvement on KITTI-360, 8.4 mAP improvement on Argoverse2 and 3.9 mAP improvement on Waymo. We hope this comprehensive study can facilitate research on generalizable 3D object detection and associated tasks.

Cross-Dataset Sensor Alignment: Making Visual 3D Object Detector Generalizable
Liangtao Zheng 1.5+
Yicheng Liu
Yue Wang
Hang 7
Shangaai Qi Zhi Institute University of Southern California
11S, Tsinghua University
Shanghai AI Lab SCC San Diego
While camera-based 3D object detection has evolved rapidly, these models are susceptible to overfitting to specific sensor setups. For example, in autonomous driving, most datasets are collected using a single sensor configu- ration. This paper evaluates the generalization capability of camera-based 3D object detectors, including adapting detectors from one dataset to another and training detectors with multiple datasets. We observe that merely aggregating datasets yields drastic performance drops, contrary the expected improvements associated with increased training data. To close the gap, we introduce an efficient technique for aligning disparate sensor configurations - combination of camera intrinsic synchronization, camera extrinsic correction, and ego frame alignment, which collectively enhance cross-dataset performance remarkably. Compared with single dataset baselines, we achieve 42.3 mAP improvement on KITTI, 23.2 mAP improvement on Lyft, 18.5 mAP improvement on nuScenes, 17.3 mAP im- provement on KITTI-360, 8.4 mAP improvement on Argoverse2 and 3.9 mAP improvement Waymo. We hope this comprehensive study facilitate research generalizable 3D object detection and associated tasks.
Abstract:
Keywords: 3D Object Detection, Model Generalization, Autonomous Driving
Introduction
3D object detection has emerged as an important task for robots. For example, autonomous vehi- cles require precise localization of traffic participants, such cars, pedestrians, and bicycles, to ensure safe driving. Consequently, 3D object detection has garnered significant attention, leading to improved accuracy across several benchmarks [1, 2, 3]. Nonetheless, a common limitation of existing methods [4, 5, 6, 7, 8, 9, 10] is their tendency to be trained and evaluated on the same benchmark. This practice overlooks the influence of data diversity, often under the assumption that training and testing datasets are uniformly distributed an assumption that might not always stand in real-world applications, especially when detector deployed across varied vehicle models. This raises concerns regarding the capability of these methods to learn from and adapt to diverse range of datasets.
To investigate this, we initiated a straightforward experiment, training model on one dataset and testing it on another. The results revealed a severe decline in performance: a detector trained on the
Argoverse2 [2] dataset experiences a 70.4% performance drop when evaluated on the Waymo [3] dataset, compared the counterpart trained directly on Waymo. Then, we add the nuScenes [11] dataset to augment the training data volume. However, the model with additional data fails to achieve meaningful performance (5.2 mAP) when tested on Waymo. This outcome amplifies critical question within the field of 3D perception: how to effectively utilize diverse data sources during training.
'19zhengOucsd.co Corresponding at: hangzhao@mail.. wttphhao@@miil..tnnnuuuaaa edu. cn
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

nuScenes
KITTI
KITTI-360
Lyft
Argoverse2
Waymo
tucinal chtut yrstes
Intrinsic
Extrinsic
Coordinate
Similar in different datasets. We provide image patches of the resolution and LiDAR point clouds of the same scale. Objects similar 3D shapes and distances differ significantly in shapes. Bottom: Different Sensor Suite Parameters. The focal length and resolution vary, forming different imaging planes. The camera longitudinal offset from the center also varies.
Figure 1: Top:
Why incorporating additional data hampers model performance. What differences between the datasets are responsible for such catastrophic failures? Suspecting the disparities in sensor configura- tions be the issue, we conduct another concise experiment to validate this hypothesis. Fig. we train a 3D detector on Waymo images with 2070 mm focal length and evaluated on images of different focal lengths, achieved through digital zooming. Optimal performance was observed when the focal lengths of both training and testing images were aligned. This finding remained consistent even when applied to disparate datasets, e.g. Argoverse2.
This observation reinforces our hypothesis concerning the significant influence camera parameter variations on 3D detection. The underlying reason is rooted in the nature of imaging [12]: an image serves 2D projection, capturing and rendering visual information from the 3D physical world. As depicted in Fig. 1, varying sensor configurations lead to unique projections.
We term this issue as sensor misalignment across different datasets. Our in-depth analysis underscores the pivotal roles of intrinsic, extrinsic, and ego coordinate system in this misalignment, as detailed in ($ 3.4). To mitigate this issue, we introduce straightforward strategies that leverage sensor parameters to compensate for biases input signals. First, we resize all the input images unify the focal lengths. Second, an Extrinsic Aware Feature Sampling incorporated into the detection pipeline to counteract the effects of camera translations. Third, ego frame alignment is employed to resolve ambiguities in the ego frame definition, addressing the intertwined issues of camera height and ego center. Our method yields massive improvement in the generalization capability of the detector, with an average increment of 29.5 mAP when adapting nuScenes to other datasets. In summary, the main contributions of this paper include:
A thorough evaluation pinpointing the critical issues resulting in performance decline during dross-dataset testing and multi dataset training. Our findings highlight three key elements: intrinsic, extrinsic, and the ego coordinate system.

---

Tocal Length [mm]
2070
Focal Length
1200
1500
1780
2070
2400
2700
Waymo
Waymo
map
0.1
0.6
12.5
57.7
18.8
3.3
Argo2
mAp
37.8
Training
Testing
Figure 2: Training and testing the same focal length setting gives optimal results.
simple yet effective sensor alignment method counteract this issue by correcting the input signals, leading notable performance boosts across all evaluated datasets.
Remarkable performance enhancements across various datasets. Compared direct transfer, our approach achieves average improvement of 29.5 mAP in cross-domain adaptability. Additionally, our jointly trained models outperform those trained on individual datasets, even surpassing models specifically trained on Lyft, KITTI, and KITTI-360 datasets without utilizing these datasets during the training phase.
2 Related Work
Camera-basee 3D Detection. Recently, Significant advancements have been made in camera-based
3D object detection in the Bird's ' Eye View (BEV) space [4, 6, 13, 5, 14, 15, 7, 16, 17]. The majority of these approaches transform 2D image features into 3D space by camera parameters. Inspired by
LSS [18], certain methods [6, 7, 16] estimate depths for image features and shoot them to predefined 3D grid to create a BEV feature map. Another branch of methods utilizes object queries [19]. They generate 3D queries [4, 13, 5, and project them onto the image plane to sample features. Notably, many works derivatives of DETR3D [4] and BEVDet [6], sharing substantial similarities.
Domain Adaptation. This line of methods aims to improve model performance from the source to the target domain [20, 21, 22, 23, 24, 25, 26, 27, 28, 29]. In paper [30], the authors explore the impact of data distribution on the cross-dataset performance LiDAR-based 3D detectors. In [31], Wang al. analyze the impact of camera intrinsic parameter on image features and depth estimation based on BEVDepth [7]. Diverging from focusing on single dataset, our study extends experiments to training multiple datasets, addressing misalignments in distribution during both training and inference phases. Besides intrinsic, we include the role of camera extrinsic and the ego coordinate system in causing such misalignments.
Multi-dataset training. number of studies have improved generalization capability by training on combined datasets. In monocular depth estimation, MiDAS [32] illustrates the efficacy of mixing five datasets from complementary sources. Uni3D [31] focuses on joint training strategies and their impact on LiDAR-based 3D object detection. Both studies mentioned that naively adding datasets does not guarantee improvement. We echo this sentiment in vision-centric 3D object detection, and propose a solution through cross-dataset sensor alignment.
3 Experiment and Analysis
3.1 Experiment Protocols
KITTI-360 Datasets, Our experiments involve six datasets: Argoverse2 KITTI (ITTKKT13666 [33], Lyft [34],
nuScenes [11] and Waymo [3], with a focus on camera-based 3D object detection data. Each of these datasets offers ground-truth 3D bounding box labels for various object types. An overview of these datasets is available in Table 1. We have standardized different dataset formats to the MMDetection3D [35] format for cohesive analysis.

---

Table 1: Datasets overview.
Dataset
Abbr.
Mframe
Image resolution
Object type
MRGB
Argoverse2
A
26,687
(2048, 1550)
26
Kitti
K
7,481
(1224, 370)
Kitti-360
K360
61,569
(1408, 376)
26
(2 fisheye)
Lytt
22,680
(1920, 1080)(1224, 1024)
nuScenes
N
28,130
(1600, 900)
Waymo
W
39,614
(1920, 1280)
5
3D Object Detectors. Aiming for method applicable both multi-view and single-view detection, employ BEV detectors, DETR3D [4], and BEVDet [6] our baselines, steering clear of image- based detectors like FCOS3D [36] due to their proven limitations in multi-view scenarios [4].
Metrics. We adopt the LET-3D-AP [37] metric in line with the 2022 Waymo Open Dataset Chal- lenge [3]. All dataset categories are merged into three primary classes: vehicle, pedestrian, and bicycle. We present the LET-3D-AP for each using IoU thresholds of 0.5, 0.3, and 0.3 within unified perception range of 51.2 meters.
For clarity, we only showcase monocular detection results of DETR3D, and the average mAP for the three classes in the main paper. More detailed insights, including multi-view detection results, BEVDet experiments, individual class mAP, and training specifics, are elaborated in the Appendix.
3.2 Training on one dataset and testing across datasets
We began by training detectors individual datasets and testing their performance across different datasets. The "Direct" block in Table 2 illustrates both in-domain (i.e., train and test on the same dataset) and cross-domain performance. According to the numbers in bold font, DETR3D exhibits satisfactory in-domain mAP on Waymo and Argoverse2 (AV2) but falters on Lyft, nuScenes, KITTI, and KITTI-360. One cause the declination is diverse data collection conditions: Lyft collected by 20 different autonomous vehicles, while nuScenes includes data in Singapore and the USA, causing difficulty in the model's convergence. Another cause is insufficient data volume and pixel limitation. KITTI only has 4,000 training samples, and KITTI-360 has the smallest focal length, rendering pedestrians and cyclists nearly undetectable. Regarding cross-domain mAP, it almost drops to 0 for most dataset pairs. This downfall cannot be attributed to domain shifts in the environment or object size, evidenced by the failed transfer between KITTI and KITTI-360, which are collected in the same city.
An auxiliary experiment depicted in Fig. 2 underscores the model's sensitivity to focal length. The model's performance dips on Waymo itself with focal length deviation but improves markedly when AV2 is resized to Waymo's 's focal length. This indicates intrinsic variation, as depicted in Fig. 1, to be a core issue. In the ensuing subsection, we demonstrate that merely expanding the volume of training data is insufficient to overcome this challenge.
3.3 Training on multiple datasets and testing across datasets
We augment data diversity by sequentially adding datasets into the training mix and develop six distinct models 3. The "Direct" section of Table 3 showcases fluctuating performance metrics as the dataset expands. There's an mAP increase for nuScenes from 36.3% to 46.2% but an observable decline upon the integration of KITTI-360. Similarly, KITTI's mAP recedes from 41.4% to 36.3%.
From a broader view, neither cross- mross-domain nor in-domain performance (avg.S and avg. T) achieve meaningful improvement despite the increased data volume. The detector continues overlook the intrinsic disparities within the input images, even with mixed dataset.
'eere, our adding order is nuScenes. AV2, Lyft, KITTI, KITTI-360, and Waymo. However, the trend of performance is invariant the order. See section 5.4 in the Appendix for another result starting with Waymo.

---

Table 2: Cross-domain performance DETR3D [4] trained single dataset. "Direct" means direct transfer. "avg.T" stands average target domains. bold font highlights in-domain performance. See for details of "K-syne", "E-aware", and "Ego-sync".
Setting
srcldst
N
A
L
K
K360
W
avg. T
N
36.3
0.8
1.8
0.0
0.0
1.1
0.7
A
0.2
48.0
0.1
0.0
0.0
17.4
3.5
L
0.5
0.1
37.3
0.4
0.0
0.1
0.2
Direct
K
2.8
1.2
0.0
24.5
1.1
0.7
1.2
K360
0.1
0.2
3.2
26.1
0.1
0.7
W
0.1
8.9
0.0
0.0
0.0
58.8
1.8
N
40.8
25.5
18.6
29.7
18.0
23.4
23.0
A
13.2
51.4
7.5
6.6
4.6
38.8
14.1
L
1.0
1.3
44.0
8.1
5.7
1.5
3.5
K-syne
K
2.4
1.2
1.2
31.0
6.1
0.5
2.3
K360
14.6
14.7
7.3
34.6
34.7
8.2
15.9
14.5
37.8
14.3
9.4
5.6
57.7
16.3
N
43.1
33.6
32.8
33.0
18.4
33.0
30.2
A
24.4
48.1
34.1
18.1
37.4
24.5
K-syne,
15.7
19.6
47.1
20.0
12.9
18.9
17.4
E-aware,
K
7.1
8.7
10.2
29.1
9.3
2.4
7.5
and Ego-sync
K360
13.9
17.7
16.6
39.1
36.7
8.4
19.1
25.4
38.2
33.6
21.2
11.7
57.6
26.0
3.4 Analysis
In this section, we model 3D-2D correspondence of objects and scrutinize 3D detection pipeline. Our analysis reveals that apart from intrinsic, extrinsic and ego coordinate system also influence detection performance.
3D-2D correspondence of objects. We begin by examining the projection of 3D object via pinhole camera model. With frontal camera of focal length fa at coordinates (fz, ty,tt) relative to the ego frame origin and an object at (æ, U, 2), having 3D size S and pixel size Spinets their relationship can be formulated as:
Spixel fz S
(I)
where x tu indicates the depth in the camera frame, and each variable in the equation a scalar. Changes in fa and result in different Spiret values, causing the same object appear differently -a factor often overlooked in cross-dataset training and testing.
3D detection pipeline. To understand the impact of Eq. (1) on 3D detection, we trace the detection process. Initially, the detector projects 3D query point (z yo, 20) to 2D coordinate (u, v). then samples image features and predicts the object's attributes using both positional and semantic information:
H: (1(u s. s, s, s), Po)
bo. Co.
where (uu 1 3, + + + denotes an image patch centered at (u, v) with dimensions 2s x 2s. We simplify our analysis by treating this patch as the image feature, skipping the feature extraction process. The vector bo denotes the predicted position and size, while the sealar å the classification score. Additionally, according to the pinhole camera model, the projection from Pa to (u, u) follows:
d(u, v, 1)" KTP.
with K and being the intrinsic and extrinsic matrices, and = xo - l1 being the depth of Po in the camera frame, embodied KTPp- The mapping function H is learned during training.
Considering the query point is the object center, and the image patch contains the object, the implications of Eq. (1) extend to Eq. (2): variations in intrinsic K and extrinsie T scale the object

---

Table 3: Performance of DETR3D trained multiple datasets. "Direct" means direct merge for training and direct transfer testing- "avg.T" stands for average target domains. "avg.S" stands the average in source domains, See $ 4 for details of "K-sync", "E-aware" and "Ego-sync".
Setting
srcldst
N
A
L
K
K360
W
avg.S
avg. T
N
36.3
0.8
1.8
0.0
0.0
1.1
36.3
0.7
+A
40.5
49.2
0.5
0.0
0.0
5.2
44.9
1,4
+L
41.6
50.5
43.7
0.0
0.0
3.8
45.3
1.3
Direct
+K
41.5
49.7
46.0
41.4
1.1
3.6
44.6
2.4
+K360
42.6
54.3
46.8
36.3
29.7
3.3
41.9
3.3
+W
46.2
53.7
49.4
39.5
29.7
61.9
46.7
N
40.8
25.5
18.6
29.7
18.0
23.4
40.8
23.0
+A
45.5
50.0
25.1
35.8
21.3
44.2
47.8
31.6
+L
46.8
53.2
55.1
37.8
23.1
45.3
51.7
35.4
K-syne
+K
47.4
53.5
53.6
57.8
21.8
44.4
53.1
33.1
+K360
50.2
54.4
54.0
60.2
39.6
44.7
51.7
44.7
+
51.8
55.3
56.6
61.9
40.7
63.7
55.0
N
43.1
33.6
32.8
33.0
18.4
33.0
43.1
30.2
+A
52.1
52.7
38.4
42.2
23.2
40.7
52.4
36.1
K-sync,
+L
52.6
53.2
59.5
46.1
26.1
43.6
55.1
38.6
E-aware,
+K
51.0
54.7
60.2
63.9
28.4
44.6
57.5
36.5
and Ego-syne
+K360
50.0
55.0
59.8
65.0
42.7
45.2
54.5
45.2
+W
54.8
56.4
60.5
66.8
43.4
62.7
57.4
within the image, altering the contents of the image patch U, 8++8). Meanwhile, shifts the ego frame influence the value of query Po and object location (z. "...y22 2)* Observing identical 3D objects with different sensor configurations alters the distributions of both 2D features and 3DD positions, yielding inconsistent mapping function H. Consequently, detectors make incorrect predictions during cross-dataset testing and learn from conflicting data samples in multi-dataset training. In summary, the sensor deviation between datasets is three-fold:
Intrinsic. Variations in camera intrinsic parameters, particularly the focal length, cause objects of identical size and location to be rendered differently in images across datasets. Extrinsic. As indicated in Eq. (1), extrinsic parameters or camera poses, especially tu. also impact the apparent size of the object in images.
Ego coordinate system. Fluctuations in ego centers affect data distribution. Notable differences in ego height impair the reliability of query prior knowledge in cross-dataset testing.
These discrepancies are illustrated in Fig. where similar 3D information corresponds to highly distinct 2D image information with changes in the sensor suite.
Sensor Alignment Approaches
We introduce three efficient strategies to tackle the challenges: Intrinsic Synchronization, Extrinsic Aware Feature Sampling and Ego Frame Alignment. We observe that implementing the last two without Intrinsic Synchronization leads to sub-optimal outcomes Our approaches collectively create sensor-invariant 3D-2D mapping relationship. enhancing model consistency across diverse datasets.
4.1 Intrinsic Synchronization (K-syne) Among the factors impacting model performance, camera intrinsic parameters prove the most straightforward yet crucial to synchronize. Inspired by the intrisic-deccuppee technique prevalent in depth estimation [31, 32), we resize the images to a fixed focal length, fo, using bi-linear interpolation.
lso known the ground-truth labels.
See section 5.4 in the Appendix for ablation studies alignment approaches.

---

Frerre
Mayaa
(a) Intrinsic Synchronization
Extrinsic Aware Module
(c) Ego Frame Alignment
Figure 3: (a): Resizing both input images and their focal length to achieve unified focal length. (b): Altering the fixed sampling region to vary in size, dependent the distance between the camera center and query points(yellow to red). (c): Aligning varied ego frames by adjusting the ego origin in accordance with the actual height and dataset distribution.
As depicted in Fig. 3(a), we align all focal lengths with that of Waymo, which has the largest focal length.
As shown in Table 2, simple resizing makes huge improvement. Compared to testing naively, over 24 mAP gains is achieved when transferring from nuScenes to Argoverse2 and KITTI. From KITTI- 360 to KITTI, DETR3D attains mAP of 34.6%, a result on par with in-domain evaluation. This enhancement aligns with the minimal domain gap between these two datasets, except for their focal length difference (707mm vs. 552mm). Table 3 displays the result of models trained on multiple datasets, where both in-domain and cross-domain performances exhibit substantial uplifts. The adverse effect previously associated with KITTI-360 is mitigated, signaling resolution to the issue of conflicting data samples. Consequently, models are now efficiently utilizing the increased data volume to enhance performance.
4.2 Extrinsic Aware Feature Sampling (E-aware)
We introduce the Extrinsic Aware Feature Sampling (E-aware) to counteract the challenges posed by variations in camera extrinsics, specifically the frontal translation tz. Our focus is on the impact of tx on the apparent object size Spinel and the related content within a fixed image receptive field 2s x 28, as explained in Eq. (1) and Eq. (2).
Given the assumption that 3D query Po as the object's 3D center, modify the receptive field of Po to be proportional to 2g - ensuring that the sampled image content remains consistent across varying l This modification is implemented by sampling more points followed by average pooling, analogous to the ROI Align process [38].
To validate the effectiveness of E-aware, we simulate changes camera position through random translations within range of [-2m, 2m). Our method exhibits enhanced robustness these po- sitional fluctuations, indicated in Table The influence of by is found to be minimal, while adjustments for tg are incorporated in the subsequent approach. Evaluations of E-aware under cross-dataset testing and multi-dataset training are also combined with the next approach.
4.3 Ego Frame Alignment (Ego-sync)
Variations in the definition of the ego frame across datasets, particularly the height of the ego centers, lead to inconsistencies in data distribution. Our initial strategy involves transforming the ego x-y plane to the ground, which ensures consistent physical interpretations of the z-coordinates across datasets. However, this straightforward approach yields no significant improvements. We turn to a nuanced strategy, employing DETR3D trained on the Waymo dataset as a reference. The ego center's x-z coordinates for each dataset are adjusted to align with this reference. A grid search, conducted resolution of 0.5m, identifies the optimal alignment settings that maximize performance, as visualized in Fig. 4. We find that not only the height, but also the frontal position of the ego center play pivotal role, since influence the distribution of object's x-coordinates in each dataset.

---

Waymo
nuScenes
Lyft
Ego Height wrt. The Ground
KITTI-360
KITTI
Argoverse2
Ego Frontal Shift (x)
Figure 4: Exploring optimal performance through grid-search modifications of the and z-coordinates of ego
centers, integration of K-syne and F-aware. The results indicate that both the height and frontal position of the ego center significantly impact performance metrics.
Table 4: Results of random jittering experiments nuScenes, with the x-axis aligned the car's direction of movement the y-axis perpendicular to it.
methods
None
on y
on
on x.y
Direct
36.3
34.7
27.9
26.9
w/ E-aware
36.9
35.5
35.4
34.9
The comprehensive alignment ego frames, depicted Fig. yields enhanced performance metries in Table 2 and Table For instance, when training six diverse datasets, DETR3D exhibits mAP enhancement of up 27.3% on KITTI compared to the baseline direct merging approach.
Conclusion
In this paper, we meticulously examined the obstacles hindering image-based detectors from deliver- ing optimal performance and adaptability across various autonomous driving datasets. We pinpointed the root the issue to the inconsistent 3D-2D mapping relationships, primarily caused by disparate sensor configurations encompassing camera intrinsic, extrinsic, and ego coordinate systems. We demonstrate that simple sensor alignment techniques can significantly alleviate this performance degradation. Our approach yielded an average enhancement of 29.5 mAP in cross-dataset testing from nuScenes to other datasets, capitalizing nuScenes' diverse data distribution. We also achieve 21, 17.2, 6.3, 18.4. and 24.2 mAP boosts when AV2, Lyft, KITTI, KITTI-360 and Waymo serve as the source domain. Unlike many existing studies that only focus on vehicles, our evaluation metric also takes into account pedestrians and bicycles, offering a more comprehensive assessment.
In multi-dataset training, we fully exploit the potential of data volume, with 18.5, 8.4, 23.2, 42.3, 17.3, and 3.9 mAP gaining by combining 6 datasets instead of training them separately. Compared to direct merging, achieve average performance boost of more than 10 mAp on all datasets. We believe that our insights will stimulate further research in multi-dataset training and domain adaptation for vision-centric 3D object detection and localization. We emphasize the importance of applying data corrections before incorporating additional datasets or developing new computer vision algorithms to bridge the remaining domain gaps.
Limitations
While our study provides valuable insights into addressing sensor misalignment image-based detectors for autonomous driving, there are limitations consider. First, due to the lack of datasets with highly diverse camera poses, the detection performance. Second, We were unable to scale method scale in-the-wild
unable to explore the impact of camera rotation

---

dataset due to annotation scarcity. A potential solution could be employing semi-supervised 3D detectors as baselines, and we leave it to future work.

---

Acknowledgments
This work is supported by the National Key R&D Program of China (2022D00111000,
References
[1] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(11):231-1133, 2013.
[2] B. Wilson, Qi, T. Agarwal, Lambert, J. Singh, S. Khandelwal, Pan, R. Kumar, A. Hart- nett, J. K. Pontes, al. Argoverse 2: Next generation datasets for self-driving perception and forecasting, arXiv preprint arXiv:2301.00495 2023.
[3] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2454, 2020.
[4] Y. Wang, V. C. Guizilini, T. Zhang, Y. Wang, H. Zhao, and J. Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In Conference Robot Learning, pages 180-191. PMLR, 2022.
[5] Y. Liu, T. Wang, X. Zhang, and J. Sun. Petr: Position embedding transformation for multi-view 3d object detection. Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVII, pages 531-548. Springer, 2022.
[6] Huang, G. Huang, Z. Zhu, Y. Ye, and D. Du. Bevdet: High-performance multi-camera 3d object detection in bird-eye-view. arXiv preprint arXiv:2112.11990 2021.
[7] Y. Li, Z. Ge, G. Yu, J. Yang, Z. Wang, Y. Shi, J. Sun, and Z. Li. Bevdepth: Acquisition of reliable depth for multi-view 3d object detection. arXiv preprint arXiv:2206.1009) 2022.
[8] Z. Li, W. Wang, H. Li, Xie, C. Sima, Lu, Y. Qiao, and Dai. Bevformer: Learning bird's-eye-vics representation from multi-camera images via spatiotemporal transformers. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX, pages 1-18. Springer, 2022.
[9] J. Park, C. Xu, S. Yang, K. Keutzer, K. Kitani, M. Tomizuka, and W. Zhan. Time will tell: New outlooks and a baseline for temporal multi-view 3d object detection. arXiv preprint arXiv:2210.0244 10.02443 2022.
[10] X. Lin, T. Lin, Z. Pei, L. Huang, and Z. Su. Sparse4d: Multi-view 3d object detection with sparse spatial-temporal fusion. arXiv preprint arXiv:2211.1058) 2022.
[11] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuscenes: multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 11621-11631, 2020.
[12] T. Wixted and S. L. Thompson-Schill. Stevens' Handbook of Experimental Psychology and Cognitive Neuroscience, Language and Thought, volume 3. John Wiley & Sons, 2018.
[13] X. Chen, T. Zhang, Y. Wang, Y. Wang, and H. Zhao. Futr3d: A unified sensor fusion framework for 3d detection. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 172-181, 2023.
[14] Y. Liu, Y. Zhang, Y. Wang, F. Hou, J. Yuan, Tian, Y. Zhang, Z. Shi, J. Fan, and Z. He. A survey of visual transformers. IEEE Transactions Neural Networks and Learning Systems, 2023.
10

---

[15] Y. Liu, Y. Wang, Y. Wang, and H. Zhao. Vectormapnet: End-to-end vectorized hd map learning. arXiv preprint arXiv:2206.089900, 2022.
[16) Y. Li, H. Bao, Z. Ge, J. Yang. J. Sun, and Z. Li. Bevstereo: Enhancing depth estimation in multi-view 3d object detection with dynamic temporal stereo. arXiv preprint arXiv:2209. 10248, 2022.
[17] C. Yang, Y. Chen, H. Tian, C. Tao, X. Zhu, Z. Zhang, G. Huang, H. Li, Y. Qiao, L. Lu, et al. Bevformer v2: Adapting modern image backbones to bird s-ywe-view recognition via perspective supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17830-17839, 2023.
[18] J. Philion and S. Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pages 194-210. Springer, 2020.
[19] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16, pages 229..29. Springer, 2020.
[20] Y. Ganin and V. Lempitsky. Unsupervised domain adaptation by backpropagation. In Interna- tional conference on machine learning, pages 1180-1189. PMLR, 2015.
[21] C.-D. Xu, X.-R. Zhao, X. Jin, and X.-S. Wei. Exploring categorical regularization for domain adaptive object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11 11724-11733, 2020.
[22] Z. He and L. Zhang. Domain adaptive object detection via asymmetric tri-way faster-renn. In European conference on computer vision, pages 309-324. Springer, 2020.
[23] G. Zhao, G. Li, R. Xu, and L.. Lin. Collaborative training between region proposal localization and classification for domain adaptive object detection. In European Conference on Computer Vision, pages 86-102. Springer, 2020.
[24] D. Acuna, J. Philion, and S. Fidler. Towards optimal strategies for training self-driving percep- tion models in simulation. Advances in Neural Information Processing Systems, 34: 34:666616 2021.
[25] K. Muandet, D. Balduzzi, and B. Schölkopf. Domain generalization via invariant feature representation. In International Conference on Machine Learning, pages 10-18. PMLR, 2013.
[26] H. Li, S. J. Pan, S. Wang, and A. C. Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5400-5409, 2018.
[27] Q. Dou, D. Coelho de Castro, K. Kamnitsas, and B. Glocker. Domain generalization via model-agnostic learning of semantic features. Advances in Neural Information Processing Systems, 32, 2019.
[28] M. Facil, B. Ummenhofer, H. Zhou, L. Montesano, T. Brox, and Civera. Cam-convs: Camera-aware multi-scale convolutions for single-view depth. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages 11826-11835, 2019.
[29] Z. Li. Z. Chen, A. Li, L. Fang. Q. Jiang. X. Liu, and J. Jiang. Towards model generalization for monocular 3d object detection. arXiv preprint 205.::2205.11664 2022.
[30] Y. Wang, Chen, Y. You, I.. F. Li, B. Hariharan, M. Camphell, K. Q. Weinberger, and W.-L. Chao. Train in germany, in the usa: Making 3d object detectors generalize. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11713-11723, 2020.

---

[3] S. Wang, X. Zhao, H.-M. Xu, Z. Chen, D. Yu, J. Chang, Z. Yang, and F. Zhao. Towards domain generalization for multi-view 3d object detection bird-eye-view. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13333-13342,
2023.
[32] R. Ranftl, K. Lasinger, D. Hafner, K. Schindler, and V. Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transactions on pattern analysis and machine intelligence, 44(3):1623-1633 2020.
[33] Y. Liao, J. Xie, and A. Geiger. Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[34] R. Kesten, M. Usman, Houston, Pandya, K. Nadhamuni, A. Ferreira, M. Yuan, B. Low, A. Jain, P. Ondruska, et al. Lyft level 5 av dataset 2019. urihttps://tevel.. lyft. com/dataset, 1:3, 2019.
[35] M. Contributors, MMDetection3D: OpenMMLab next-generation platform for general 3D object detection. https: attps://githun com/ com/open-tn haaaaa/mammmmmtttttiii/// 2020.
[36] T. Wang, X. Zhu, J. Pang, and D. Lin. Fcos3d: Fully convolutional one-stage monocular 3d object detection. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 913 113-922.44 2021.
[37] W.-C. Hung, H. Kretzschmar, V. Casser, J.-J. Hwang, and D. Anguelov. Let-3d-ap: Lon- gitudinal error tolerant 3d average precision for camera-only 3d detection. arXiv preprint arXiv:2206.07700 2022.
[38] K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
12

---

Supplementary materials of Cross-dataset Sensor Alignment: Making Visual 3D Object Detector Generalize
Anonymous Author(s)
Affiliation
Address email
Section I: brief introduction to the six datasets.
Section 2: data format conversion and dataset merging settings.
Section 3: details of training settings.
Section 4: analysis and results of BEVDet.
Section 5: additional results from DETR3D.
Datasets
Argoverse2. The Argoverse2 dataset [1] is collected across six cities in the U.S., including Pittsburgh, 8 Detroit, Austin, Palo Alto, Miami, and Washington D.C. It encompasses data captured various weather conditions and at different times of the day. The dataset includes images from two grayscale stereo cameras and
cameras that provide 360-degree coverage. It offers 3D annotations 11 frame rate of 10Hz. To align with the frame rate in the nuScenes dataset, we sub-sample the
12 Argoverse2 dataset, resulting in 21,982 frames for training and 4,705 for validation, with a frame rate
13 of 2Hz.
14 KITTL The KITTI [2] object detection benchmark consists of 7,481 frames for training. These scenes captured in clear weather and during daytime around Karlsruhe, Germany. The dataset provides images from two RGB cameras and two grayscale cameras, forming two stereo pairs. For our study, solely utilize the left RGB camera. Following [3], we separate the data into 3,712 training frames and 3,769 validation frames.
18
19 KITTI-360. The KITTI-360 dataset [4] is significantly larger than KITTI, comprising 61,569 valid
20 frames with 3D annotations frame rate of 10Hz. The labelled data obtained from nine video
21 clips. To create training and validation split, utilize the first 80% of each video clip for training and the remaining 20% for validation. This results in training set containing 49,253 frames and
23 validation set containing 12,316. Unlike KITTI, the KITTI-360 dataset provides RGB images from
24 two frontal perspective cameras and two side fish-eye cameras. Similar to the KITTI settings, we exclusively use the images from the left frontal camera in our study.
25
26 nuScenes. The nuScenes dataset [5] contains 28130 training and 6019 validation keyframes. The scenes collected around Boston, USA and Singapore in multiple weathers and during different time frames. For each frame, the dataset provides six images that collectively cover a 360-degree 29 view.
28
Lyft. The Lyft Level dataset [6] consists of 22,680 annotated frames captured around Palo Alto, 30
31 USA. during clear weather conditions and daytime. Each frame within the dataset includes images from six surrounding view cameras well as long-focal-lenge frontal camera. It is essential to mention that this dataset is collected using 20 independent vehicles, and the surrounding view images
Submitted to the 7th Conference Robot Learning (CoRL 2023). Do not distribute.

---

34
have two different resolutions. Following the approach outlined in MMDetection3D [7], we partition the dataset into 18,900 frames for training and 3,780 frames for validation.
35
36
Waymo. The Waymo [8] dataset is collected across Phoenix, Mountain View, and San Francisco, encompassing various weather conditions and different times of the day. It includes images from five cameras and offers 3D annotations at frame rate of 10Hz. The training set consists of 158,081 frames, while the validation set contains 99,990 frames. To align with 2Hz frame rate, we sub-sample the dataset, resulting in 31,616 frames for training and 7,998 frames for validation.
37
39
40
Converting Datasets into a Unified Format
41
42
This section provides a detailed explanation of how we convert Argoverse2, KITTI, KITTI-360, Lyft, nuScenes, and Waymo datasets into unified format. specifically focus on the issues related to coordinate systems and 3D annotations that arise when merging these datasets. We convert data under MMDetectionn3 v1.1.0.
43
44
46
2.1
Coordinate Systems
47
Regarding sensor configuration, the datasets differ in terms three types of coordinate systems: ego frame, LiDAR frame, and camera frame. The definition of camera is clear, so we primarily focus on the former two. Each dataset typically includes at least one LiDAR mounted on the vehicle's 's roof. The origin of the LiDAR frame is commonly located at the center of the top LiDAR if there is no specification.
48
49
51
The ego frame is more confusing as the origin defined differently across the datasets. Argoverse2, nuScenes and Waymo, the ego origin is located at the center of the car's rear axle. Argoverse2, is approximately 33em above the ground, while in the latter two datasets, it is projected onto the ground plane. Lyft does not explicitly specify the location; however, based the statistical analysis of 3D annotations, it is also considered on the ground. These four datasets have corrected their axes, ensuring the z-axis consistently points upwards from the road surface. On the other hand, for KITTI 58 and KITTI-360, the Inertial Measurement Unit (IMU) defines the ego frame. Across all the datasets, 59 the x-axis aligns with the car's longitudinal direction, while the y-axis points to the left.
52
53
54
57
80
Regarding LiDAR point clouds and 3D annotations, Argoverse2 and Waymo define them in the ego frame, while KITTI, KITTI-360, Lyft, and nuScenes define them in the LiDAR frame. Consequently, during training, consider the LiDAR centers of the latter datasets the "ego centers'.
63 In terms of ego frame alignment, for Argoverse2, KITTI, and KITTI-360, we simply lower their ego
64 centers by 0.33m, 1.73m, and 1.73m, respectively, to align them with the road surface. For Lyft and nuScenes, we transform the entire coordinate system their original ego frames, which are also pressed against the road.
65
86
67
2.2
Object Filtering
68
To ensure consistency and data quality, we discard object annotations that fall outside the camera view. This accomplished by projecting the eight corners of each object's 3D bounding box onto the image plane. The object annotation removed if all eight corners are outside the image boundary. Additionally, filter annotations based on specific range in the x, y. and coordinates, namely [-51.2, 51.2] x [-51.2, 51.2) x [-5.0 4.0]. As every dataset includes LiDAR data, also discard annotations with no LiDAR points within the 3D bounding box since they may be occluded.
89
70
71
73
74
2.3
Merging Categories
75 To unify the category labels across datasets, we merge the categories within each dataset into three classes: vehicle, pedestrian, and bicycle. This taxonomy closely resembles Waymo's 's classification but with little difference in the bicycle category. Waymo excludes bicycles without rider, whereas

---

Table 1: The original categories in each dataset vs. merged categories
Dataset
Vehicle
Podestrian
Bicycle
REGULAR VEHICLE, LARGE VEHICLE,
PEDESTRIAN. WHEELED RIDER.
BUS, BOX TRUCK, TRUCK,
BYCYCLE, BYCYCLIST
Arguverse2
MOTORCYCLE, VEHICULAR TRAILER, TRUCK CAB, SCHOOL BUS
OFFICIAL SIGNALER
KITTI
Car, Van, Trunk, Tram
Pedestrian, Person Sitting
Cyclist
bus, car, caravan, motorcycle,
KITTI-360
bicycle, rider
trailer, train, truck, unknown vehicle
cur. truck, bus, emergency vehicle. other vehicle, motorcycle
Lyft
pedestrian
bicycle
car, truck, construction vehicle, bus, trailer, motoreycle
nuScenes
pedestrian
bicycle
Waymo
Car
Podestrian
Cyclist
Table 2: Percentage of cach class in cach datasct.
Dataset
Vehicle
Pedestrian
Cyclist
N
70.3%
26.6%
3.1 %
A
71.9%
26.9%
1.2%
L
93.4%
3.7%
2.9%
K
84.8%
11.4%
3.8%
K360
89.9%
4.9%
5.1 %
W
64.3%
34.7%
1.0%
78 include such objects when relabeling the other datasets. Table 1 shows the mapping of all categories
79 the three classes. Any types not listed the table are discarded during the merging process. Table
80 also shows the percentage of each class in each dataset.
Training Details
81
For all detectors, the image backbone is Resnet-50 [9] pretrained ImageNet [10].
83
DETR3D. We use images with original resolution and the original training policy [11].
84
BEVDet. The input images are 1/2 width and height. We use adam [12] with weight decay
85
1 10-7 as optimizer, and train it for 24 epochs with batch size 64 and initial learning rate 2 10-4 which will be decreased 10 times 20th and 24th epoch. We also set the depth bins be [1, 140).
86
87
During training, we deploy data augmentations in BEV space and image input for BEVDet. We follow the original settings in [13], except that we do random scaling with a range [0.8, 1.2]. After scaling, we crop the bottom part of the images from every dataset and pad them to unified resolution:
88
89
90
960 x 448.
Analysis and Results of BEVDet
91
92
We have verified our sensor alignment strategy on BEVDet, which leverages depth estimation to shoot 2D image features to 3D space. The results shown in Table 3 and Table 4. Since BEVDet represents another type of BEV detector against DETR3D (LSS-based vs. Query- there are some differences in the alignment approaches.
95
96
4.1 Sensor Alignment Approaches
97
Intrinsic Synchronization (K-syne). Because BEVDet requires inputs of fixed resolution, image 88 resizing is not very compatible since introduces dynamic resolution. Instead, we sync focal 39 length by applying intrinsic-decoupled depth estimation [14, 15, 16] do. is confusing that the performance ("K-syne") in Table and Table 4) does not improve much. We visualize the prediction in Fig. 1. Surprisingly, this module predicts depth correctly. The reason for failure is the wrong heights.

---

Table 3: 3D-mAP of BEVDet [13] trained single dataset. "Direct" direct tramsfer, "K-syne" Intrinsic Synchronization ("K" stands for the intrinsic matrix K), "Ego-sync" means Ego Frame Alignment.
Setting
sreldst
N
A
L
K
K360
W
avg. T
N
29.5
0.0
0.3
0.0
0.0
0.0
0.1
A
0.0
34.3
0.0
0.0
0.0
9.0
1.8
L
0.0
0.0
31.8
0.1
0.2
0.0
0.1
Direct
0.0
0.0
0.1
9.2
0.0
0.0
0.0
K360
0.0
0.0
0.0
0.2
19.5
0.0
0.0
W
0.0
0.4
0.0
0.0
0.0
45.1
0.1
30.7
0.2
3.9
5.0
2.2
A
2.4
37.9
0.0
0.0
0.0
4.2
1.3
L
0.2
0.0
32.0
0.3
0.6
0.0
0.2
K-syne
K
0.2
0.0
0.1
10.7
1.5
0.0
0.4
K360
0.0
0.1
2.7
11.0
18.3
0.0
2.8
W
0.3
17.8
0.0
0.0
0.0
47.6
3.6
N
31.9
9.8
2.8
6.3
3.7
0.6
4.6
7.0
37.8
4.6
2.9
1.2
3.1
3.8
L
0.5
0.0
31.6
2.6
3.5
0.0
1.3
K-sync + Ego-sync
0.2
0.3
1.3
10.0
1.6
0.2
0.7
K360
0.0
0.0
3.4
12.4
21.3
0.0
3.2
11.6
26.3
8.8
10.8
3.9
47.6
12.3
Table 3D-mAP BEVDet trained on multiple datasets. "Direct" means direct merge and transfer.
Setting
sreldst
N
A
L
K
K360
W
avg.S
avg. T
29.5
0.0
0.3
0.0
0.0
0.0
29.5
0.1
+A
33.7
38.8
0.0
0.1
0.0
8.7
36.2
2.2
+L
36.2
40.0
38.2
0.1
0.1
4.5
38.1
1.6
Direct
+K
32.1
40.3
37.7
29.9
0.2
5.7
35.0
3.0
+K360
33.4
40.0
39.0
32.5
23.4
8.7
33.7
8.7
+W
30.8
37.7
38.4
34.6
22.3
45.2
34.8
30.7
0.2
3.9
5.0
0.5
30.7
2.2
+A
34.8
41.2
1.4
1.2
0.4
4.3
38.0
1.8
+L
37.1
40.5
38.7
4.6
3.6
16.8
38.8
8.3
K-syne
K
38.7
41.1
38.5
32.9
7.1
12.2
37.8
9.6
KK36
35.3
42.5
39.4
36.9
23.2
6.4
35.5
6.4
+W
36.7
46.8
39.2
38.3
25.2
52.3
39.8
N
31.9
9.8
2.8
6.3
3.7
0.6
31.9
4.6
36.8
41.2
7.2
11.8
2.5
3.2
39.0
6.2
+L
38.7
44.0
36.4
16.4
10.3
2.6
39.7
9.8
K-syne Ego-syne
KK
37.5
41.5
38.8
34.2
11.0
9.4
38.0
10.2
+K360
39.4
43.9
38.4
40.2
27.3
2.7
37.8
2.7
+W
40.8
46.1
39.7
42.7
28.1
51.4
41.5
102
When we use a BEV metric that ignores heights or executes Ego Frame Alignment, the performance improves immediately. See 4.2 and $ 4.3 for more details of BEV metric and intrinsic-decoupled depth estimation.
103
104
Extrinsic Aware Feature Sampling (E-aware). This module is no longer applicable since the depth estimation module does not predict depths from the ego center. On the contrary, it predicts from the camera optical center and naturally bypasses the impact of extrinsic. However, even if the depth of image features are correctly inferred, we hypothesis that model still struggles estimating the dimensions of objects, which may be addressed by introducing extrinsic embedding.
106
107
108
109
Ego Frame Alignment (Ego-syne). We the same Ego-syne settings in the main paper. In Table 3, transferring from Waymo to other datasets has an improvement of 12.2 in 3D-mAP, In

---

Figure I: Visualization objects in point clouds and images. The top is prediction made by BEVDet and the bottom shows groundtruths.
112 Table 4, BEVDet achieves a performance boost of 6.7 mAP when training on all datasets. The similar mAp gains proves the importance of ego frame definition once again.
113
114
4.2
3D Metric vs. BEV Metric
115
The poor cross-dataset improvement after applying K-syne seems inconsistent with the one gained in DETR3D. However, after careful examination, we find that although K-syne corrects the depth prediction, it cannot correct object heights.
118
The visualization detection results proves this point. As shown Fig. 1, in bird-eye view, the predicted objects have become very close to the ground truth after we apply K-syne. However, large consistent offset appears on the image plane: the objects are predicted lower than they should be. Although the depths are estimated correctly, the heights are wrong, causing the 3D mAP be nearly zero
119
120
121
123
We show the power of K-syne by changing our 3D metric into a BEV metric. In Table 5, we switch the metric by setting the center heights to zero for both prediction and ground truth objects. A significant improvement emerges after we apply both BEV metric and K-sync, up to points.
124
125
The reason is simple: K-syne only corrects depths. In DETR3D, objects inferred from 3D query points that contain height information, while BEVDet collapses 3D grid into BEV pillars, losing the height information. LSS- based [17] methods often assume all objects to be on flat ground. Under this setting, it ignores height information and can not deal with changes in altitude, such as irregular terrains and ego frame changes. This could be solved by setting 3D voxel grid instead of BEV pillars, or post-procee process the heights according the terrain.
127
128
129
130
131
132
4.3
Intrinsic Decoupled Module
133
The dense depth estimation module in BEVDet requires fixed input resolution throughout training and testing, which prevents BEVDet from changing the input resolution flexibly DETR3D can. As alternative, We scale the predicted depth according to the focal length. We call it Intrinsic Decoupled Module.
134
5

---

Table 5: BEV-mAP of BEVDet [13] trained single dataset with Intrinsic Synchronization. We show avg.T
improvement compared to 3D-mAP
Setting
src/dst
N
A
L
K
K360
W
A avg.T
N
33.6
22.5
8.7
12.4
4.4
16.8
+10.8
A
13.6
39.7
7.2
8.0
3.4
21.6
+9.5
L
3,0
0.9
34.3
3.2
5.0
1.7
+2.6
K-syne
K
0,9
2.5
3.2
13.4
2.7
1.1
+1.7
K360
0.2
0.5
9.5
13.4
22.4
0.3
+2.0
W
16.1
29.1
10.2
10.6
3.7
50.3
+10.3
137
Assuming that have depth estimation network, which is trained on dataset with fixed focal length (e.g. nuScenes with f=1266), it will predict objects' depth according to their pixel size. Given Spiret as the pixel size of a certain object and d as the metric depth (in meters), the network learns mapping:
139
140
M Spivel + d.
(I)
Intuitively, if an object looks small, it predicts large depth, and vice versa, and if we resize the
142
image to be smaller, all the objects look smaller, so the predicted depths increase. However, the 3D locations of objects do not change with image resizing, so predictions are eventually wrong.
143
144
We want the model learns and predicts mapping that invariant the focal length changes, so we scale-invariant depth d":
145
d'
x d. fo
where / is the input focal length, and fo is constant. It can be understood the "reference focal length", i.e, let the depth network "feel" as if it were working on single camera, although receives 148 images of various focal lengths from different datasets.
147
149 In practice, force the model learn:
M Spinel + d",
(3)
150
and we recover the metric depth by:
d == fo f x d:
(4)
for shooting image features to BEV grid later.
5
Additional Results from DETR3D
152
In this section, we first provide additional results monocular 3D detection using DETR3D.
154
5.1
Ablation Studies by Cropping the Input Images
155
To investigate whether the model relies on other visual cues for object detection, we conduct an experiment where we crop the input images at different positions during testing. Table 6, we find no performance drop in DETR3D, whereas BEVDet shows a substantial decrease. This indicates that DETR3D does rely on objects' position in the images. On the other hand, BEVDet, which incorporates depth network in its architecture, relies more on this kind of pictorial cues, suggested in prior work [18].
156
157
158

---

Table 6: Results of DETR3D and BEVDet which trained Waymo, using images cropped different positions during testing.
Cropped height
[192,992]
[288,1088]
[384,1184]
[480,1280)
Origin
DETR3D BEVDet
56.9
59.3
59.3
59.7
57.7
20.8
30.5
36.4
31.5
39.0
Table 7: Ablation study of synchronizing focal length different values. focal lengths.
We add "en to indicate the original
Train
focal length
N
A
L
K
K360
W
not synced
36.3
0.8
1.8
0.0
0.0
1.1
1260
35.7
21.9
13.8
27.1
16.9
19.2
N
2070
40.8
25.5
18.6
29.7
18.0
23.4
3100
41.7
26.2
18.7
28.5
17.7
25.8
4140
43.3
26.4
19.4
31.8
17.5
26.3
not synced
0.2
48.0
0.1
0.0
0.0
17.4
1780
11.8
46.8
7.2
6.4
5.4
38.4
A
2070
13.2
51.4
7.5
6.6
4.6
38.8
3100
12.1
51.1
8.9
7.5
5.1
40.7
4140
11.5
53.9
9.2
6.1
4.0
40.9
not synced
0.5
0.1
37.3
0.4
0.0
0.1
2070
1,0
1.3
44.0
8.1
5.7
1.5
L
3100
1.1
1.3
41.0
8.2
4.8
1.4
4140
1.0
1.6
42.9
10.5
3.2
1.6
not synced
0.1
0.2
0.0
3.2
26.
0.1
550*
8.2
4.5
3.3
18.4
25.9
5.5
K360
2070
14.6
14.7
7.3
34.6
34.7
8.2
3100
14.0
15.4
9.4
33.6
35.9
7.5
not synced
0.1
8.9
0.0
0.0
0.0
58.8
2070
14.5
37.8
14.3
9.4
5.6
57.7
W
3100
14.6
38.1
13.6
7.7
3.2
62.6
5170
10.1
38.7
10.7
8.8
2.0
62.1
161
5.2
Synchronize Focal Length to Different Values
162
This subsection shows that our Intrinsic Synchronization strategy works with different focal lengths. We perform an ablation study by increasing the synchronized focal length value both training and testing. In Table 7, we observe that enlarging the images does improve mAP; however, the extent of improvement diminishes as the input image size increases. We argue that this phenomenon can be attributed to the fact that smaller objects become easier detect in larger images.
163
164
185
186
187
5.3
Ablation Studies on Sensor Alignment Approaches
168
evaluate various combinations of modules across all datasets and present the results Table observe that each component contributes to the overall performance; however, only after aligning the intrinsic parameters does the extrinsic and ego coordinate system start impact the detection per- formance. While the Extrinsic Aware Feature Sampling (E-aware) may cause drop in performance, we argue that this module provides extrinsic robustness in real-world scenarios.
169
170
5.4 Multi-dataset Training Beginning with Waymo
174 We begin with Waymo the first dataset and gradually add datasets into the training set. Table shares the same mAP trend with Table 3 in the main paper, which means observation is invariant to the addition order. Here, KITTI-360 drags down the performance again. This decline can be
7

---

Table 8: Ablation study the elfectiveness of each module in alignment approaches. All models trained on Waymo. "Ego" stands Ego Frame Alignment. "Avg.T" stands for the average cross-datasel performance.
K-sync
E-aware
Ego
N
A
L
K
K360
W
avg. T
0.1
8.9
0.0
0.0
0.0
58.8
1.8
0.0
8.4
0.0
0.0
0.0
58.3
1.7
5.0
0.0
0.0
0.0
59,4
1.0
0.0
3.5
0.0
0.0
0.0
59.4
0.7
14.5
37.8
14.3
9.4
5.6
57.7
16.3
24.7
39.4
33.0
21.2
13.6
57.7
26.4
14.1
37.7
17.0
9.3
5.8
57.6
16.8
V
V
25.4
38.2
33.6
21.2
11.7
57.6
26.0
Table 9: 3D-mAP of DETR3D trained on multiple dataset, beginning with Waymo.
Setting
sre/dst
W
N
L
K360
avg.S
avg.T
W
58.8
0.1
8.9
0.0
0.0
0.0
58.8
1.8
+N
60.4
38.4
11.2
0.2
0.0
0.0
49.4
2.8
+A
63.0
42.7
54.8
0.1
0.0
0.0
53.5
0.0
Direct
+L
60.2
44.4
53.1
47.3
0.0
0.0
51.2
0.0
+K
63.1
45.5
53.4
49.2
44.3
1.9
51.1
1.9
+K360
61.9
46.2
53.7
49.4
39.5
29.7
46.7
0.0
attributed to significant amount of discordant data, as illustrated in Fig. 2, which provides statistics 178
on the data volume across different datasets.
179
5.5
Per-class and Per-location Evaluation Results
Table 10 and Table 11 are the extended version of Table 2 and Table 3 in the main paper, showing per-class mAP each dataset. Furthermore, Table 12 shows the evaluation result of the best model in Table on each city in each dataset.
182
16
14
12
Framelt x 10000
10
8
6
4
2
0
W
N
+A
+L
+K
+
K360
Waymo
nuScenes
AV2
Lyft
KITTI
K360
Figure 2: Data volume of each datasct under monocular detection setting

---

Table 10: 3D-mAP of DETR3D trained single dataset(full version). The performance is reported in the format vehivclclsss/ pedestrians/ bicycles).
Senting
woldel
XN
A
I.
K
K360
W
55.3 (56 7/36 7/1577)
0.8 (0-1
(166611112))7)
0.0 (0.00)000)))
0.0 0.0:000000000
1.1 (OU6/1 (/21114) 44
0.2 (0, (40,))
48,0 (33 (738803 7/317)
0.1 (0.0)1,00.1)
0.0 (0.00) L/0.0)
0.0 (00000000..000
17.4 (19.7/14 1/18.33)
Direct
K
0.5 10 (0.900770.0)
2.8 e4 0/30
0.1 (0. DO. (O.0)
1.2 12:20.911.. 20.511.30 3015)
0.0 LDD (0.00)
37.3 (70.54 (70.5416884499
24.5 (60 2/25 L/8.
0.4 (0.54) 7/0
0.0 0.0.0.000 (A0,))
[. <0.9/2, 1814)
0.7 (0.20.411.3
0.1 (0.090334000)
K350
0.363 2/0.2/0.00
3.210 7/2.2)
26, 26,3000,244,5133 5/13.7
W
0.3 1A0.0)
5.9 (14.5% 2/3.1)
0.0 (LLDN.ONLL0)
0.0 (C.000./00.0
0,0 (00.GG
58.8 (78, (28.1/50.3341
N
20X (SSE54 (21
25.5 (AS 1/25 125.7
IS (39 (2910)
20.7 7477:4111 1224 22337)
(37.001 (46))
23.4 [3T 3/7.9
13.2 3.77555))
51.4 (74 8643 5(366)))))
(16 7/3 99
666 91. 8/9.
(12 2/0 8A0
38.8 (60 3/24
K syne
K
(2.045944)
8/1 2/0
1.3 300600
46 D (76,4200,9544
(3.000550))
R (17 (17.344 7/2
4)
6.1 (13.322772.55
5.7 (12 22..0/2.44
0.5 (3.240220.1)
1.5 (/D
K360 W
14.6 (36.009
14.7 21.79
(20,0/16600))
34.6 (59.9/2 L/189)
34,3 (69, (10.022))))
8.2 (22.780. 7/1.1)
37,8067 3/44 U119)
322 5/8.9)
9.4 34.8/17
5.613 1/2.60
ST. 77.708885002244
33.0 2711.3)
47225
K-ynne
15.2(95.1210:31.))
19.6/42.6133
7/27
S22
5/1
1,2
1,5/99
(V2.)
37 4158
5.22
swane,
Ego-syn
7.8012.6/77.000
30.33
221.9.22.8/2.99
10
29. 1:46 5/28. (/966
5/23.2)
9,3 248.9777)
2-4(3./225511.)
25.4045.0226355.0
38.20033/44/11.7)
33.863..810.820.77
2122/1355/553224
11,3225 23.33
57. 57.607..400994222
Table 11: 3D-MAP of DETR3D trained on multiple dataset, beginning with nuScenes (full version). The performance is reported in the format of all(vchicles/ pedestrians/ bicycles)
Setting
solist
K
K360
W
36.3 706.7/36./15.777 2/15.7)
CO.91 (40.3)
(1.0).1/2.7)
(0.0/).01L.))
1.1 (0.60)
40.5
(60 (60.4/8..0 9/22.11
69.2 (76.0M2.322) .3
(0.7/0.54.33
(0.0/000000)
5.2 (4.554 2/6
Direct
7/9.6/22.11
50.5
(78.442.3/9003)
46.0
(74.5/26
1/34
(...4.214 2/2
34.44.344 2/2
+3360
2/90.//2222
54.3 8440.0
(76.2/26 5/37.1)
36.3
(60.62)6/20))
(4,2333333 (9)
(79.8447-0/38.5
(606694/99.))
4/45
122.1/24 2/2377
(37.473387779)
45. 5/65
(18.1)
(42.5/15.6000
(67
K sync
1
46.8/
(64.3/98.0//94
1/28.99
53.5 (79.345.6/335
53.2079.5446.00 6/3..)
53.6
15.1 4/46.2)
57.5
8C50 7/306/3319
7/66.9)
4717.2/111111
0147
60.2
9447 (47.2/51 .4)
99.6 (72.7716.8/99) 3)
(7) 6/38
2440 8/57.3)
55.3 144
9
(.511.1
(73
N
43, 1033 ORSS 0/1 4))
33.6662 4/30, (0004/7.9)
32. 8/60 3/18//
13.0045 0/25 1.7
4C37/00 8:5.5)
33,00 3/28
and Ego syne
awuze
K-sync
E-
+A
+
+K
4K360
$2.6/08 2/38))
$2. 1088 L/508/37744
50.00 5/504//9..3 3)
5..067 9/81 3/33.3)
55.0051 4/47 4/36. 2)
$2.7 8047.3322
$3.2 1/47,4/3..2 2)
56.7 8/47, 7136.5) 5)
59.5/ 5/55 6/45 3/47.44
50 2/35 6/41 5/50.5)
58.4: 43.4:/144225 5/21 2)
59 8/86 9943 4)
63.9833 2/58 (/5.0)
46.1 1611 0/57 8/38.9)
65.0085 4/54 S/55.) I)
42.2/54 9/33 655.99
42 42.7175 3/3.33
25, 1(47 (147.999977/11.99
28. AL 5/22 9M13.5)
25. 2/1.00006..//113
45 2168
44.6.7.
40.7/64. 1/35, 6/22.55
43.6/67. 1/3
4))
+W
54.8/72.7/52 012227/522550 5/89.1)
56. 30.4182.3499.053.0
(0.5:87.445.77484
66. 86..085.22 2755. 2/5.52255../57.2)
43.426.3122.333.3
62.7/83. 77583.4566644777
183
5.6
Results from Surrounding-view Detection
184
We extend our data alignment strategies to surrounding view detection and verify their effectiveness. All input images are of 1/2 height and width. The perception range is still 51.2m, but includes the area behind the ego car.
185
186
187
A Ablation studies on sensor alignment approaches. In Table 13, DETR3D is trained on Argoverse2. nuScenes and Waymo, and tested in six datasets.
189
Data diversity vs. data volume We test if the model benefits from data diversity more than data volume. Given that Waymo and nuScene are of similar data volume, we use different percentage of
190
191
training data from the two and test the model on three datasets. As shown in Table 14, mixing the data achieves better performance.
192

---

Table 12: Evaluation results location, using DETR3D with all sensor alignment approaches. The LET-3D. mAP is reported in the format of altiveliclespeddssnnnx/yyeeees
Dataset
Location
LET-3D-AP
ATX
44.6 (69. 00997764.3/0000
DTW
71.5 (79.3 /99.//60./755.
MIA
46.8 (84.0/41.1// 1/15.2)
Argoverse2
PAO
64.3 (91. 16/51.4/499.7 4/49.7)
PIT
59.9 (82.0/53.2/44.44)
WDC
39,9 (80. 1/30)
KITTI
Germany
66.8 1/57.2)
KITTI-360
Germany
(76.3/222333
Lyft
Palo Alto
60.5
boston-seaport
(7.4455..551.
singapore-hollandvillae
30.3 (67.4/23.440001
nuScenes
singapore-onenorth
50.5 (66.8/53.0/31..
singapore-queenstoonn
51.6 088855/589//7.5
other
44.9 ///./////////6//
Waymo
phx
63.3 (87 (87.0/55, 7/47.1)
sf
65.1 (83.7/58.8/55 8/52.8)
Table 13: Surrounding view 3D detection results: ablation study sensor alignment approaches. All models are trained on Argoverse2, nuSeenes and Waymo.
K-sync
E-aware
Ego
N
W
K360
avg.
48.0
40.4
54.8
0.6
6.2
0.7
25.1
48.6
41.0
53.8
0.0
3.6
0.0
24.5
49.5
39.7
54.7
1.8
7.4
1.8
25.8
47.4
40.8
53.3
0.2
4.4
0.6
24.4
52.2
46.5
55.2
22.2
22.0
11.0
34.9
50.1
47.2
54.2
30.1
36.3
22.1
40.0
Y
52.0
46.7
55.5
31.1
26.2
15.8
37.9
52.1
47.5
54.8
31.4
39.7
24.0
41.6
Table 14: Surrounding view 3D detection results: models are trained on different combinations of Waymo and
nuScenes.
lest/ (NN)
0.00+1.00
0.01+0.99
0.10+0.90
0.33+0.67
0.5+0.5
0.67+0.33
1.00+0.00
1.0
13.0
14.5
16.0
N
32.5
32.5
32.3
32.1
31.5
30.4
0.2
W
0.3
23.4
36.7
45.8
45.6
47.0
46,2
10

---

References
193
194 [1] B. Wilson, W. Qi, T. Agarwal, J. Lambert, J. Singh, S. Khandelwal, B. Pan, R. Kumar, A. Hart- nett, J. K. Pontes, et al. Argoverse 2: Next generation datasets for self-driving perception and forecasting, arXiv preprint wrXiv:2301.0049) 2023.
195
196
2] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(11):1211-1237, 2013.
199
[3] X. Chen, K. Kundu, Y. Zhu, A. G. Berneshawi, Ma, S. Fidler, and R. Urtasun. 3d object
200
proposals for accurate object class detection. Advances in neural information processing systems, 28, 2015.
201
[4] Y. Liao, J. Xie, and A. Geiger. Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
203
204
205
[5] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference computer vision and paltern recognition, pages 11621-116311 2020.
208
[6] R. Kesten, M. Usman, Houston, T. Pandya, K. Nadhamuni, A. Ferreira, M. Yuan, B. Low, A. Jain, P. Ondruska, et Lyft level 5 av dataset 2019. urlhtpss:/tevell.. lyft. com/dataset, 1:3, 2019.
209
210
211
[7] M. Contributors, MMDetection3D: OpenMMLab next-generation platform for general 3D object detection. bttapa://githbb com/ http://emmmmlmlmmmeeetttiioaaaaa htt////mdetetiinn///
212
2020.
213
Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2446 2454, 2020.
215
216
217
[9]
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
220
[10] J. Deng, W. Dong, R. Socher, Li, K. Li, and L. Fei-Fei. Imagenet: large-scale hierarchical image database, 2009 IEEE conference on computer vision and pattern recognition, pages 248 288.255. Ieee, 2009.
221
222
[1I] Y. Wang, V. C. Guizilini, T. Zhang, Y. Wang, H. Zhao, and J. Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In Conference on Robot Learning. pages 180-191. PMLR, 2022.
224
225
226
[12] Loshchilov and F. Hutter. arXiv:1711.05101, 2017.
Decoupled weight decay regularization. arXiv preprint
228
[13] J. Huang, G. Huang, 2. Zhu, Y. Ye, and D. Du. Bevdet: High-performance multi-camera 3d object detection in bird-eye-view. arXiv preprint 1rXi1:211..11790, 2021.
229
[14] S. Wang, X. Zhao, H.-M. Xu, Z. Chen, D. Yu, J. Chang, Z. Yang, and F. Zhao. Towards domain generalization for multi-view 3d object detection in bird-eye-view. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13333-13342, 2023.
232
233
[15] R. Ranftl, K. Lasinger, D. Hafner, K. Schindler, and V. Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transactions pattern analysis and machine intelligence, 44(3):1623-1637) 2020.

---

237
[16] D. Park, R. Ambrus, V. Guizilini, J. Li. and A. Gaidon. Is -seudo-lidar needed for monocular 3d object detection? In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 3142-3152, 2021.
238
239
[17] J. Philion and S. Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow; UK, August 23-28, 2020, Proceedingg, Purt XIV 16, pages 194-210. Springer, 2020.
242
[18] V. Dijk and G. d. Croon. How do neural networks depth in single images? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2183-2191, 2019.
12

---

Derivation of Eq.1 and Eq. 3 in the main paper
In this page, we will derive Eq. 1 and Eq. 3 from the main paper step by step. We will first analyze Eq. 3 since Eq. 1 can be easily derived from Eq.
Derivation of Eq. 3
Given an origin at a fixed location of our ego car, with the x-axis along the direction of travel and the z-axis perpendicular to the ground pointing upwards, we define a right-handed Euclidean coordinate system as the ego frame. Assume we have forward-facing camera along the x-axis, centered at (tz, tyy tz). According to the pinhole camera model, we have camera intrinsic matrix K and extrinsic matrix R t). where R is a 3 x 3 rotation matrix. Now R, K, t can be represented as follows:
f
0
Cx
K
0
fg
Cy
0
0
-
0
R =
0
0
-1
1
0
0
ty
tz
=
t
where fa, fy are the focal lengths along the x- and y-axis of the camera coordinate system, and Cz, Cy are the pixel offsets on the image. For simplicity we assume far = =fy and denote them as f so:
f
C₂
K =
0
f
Cy
0
0
1

---

Now for a point Po (0 yo, z0) in the ego frame, we want to find the relationship between Po and its projection on the image plane. Let's assume its image position as (u, v)
For the first step, we transform Po into the camera coordiate system:
0
-1
1
0
yo + t
y
Peam = = X Po +t =
0 0
-1
1
x
Yo
+
t
=
20 + tz
1 0
0
20
20 t
Here, we assign d 20 te as the 'depth' of Po in the camera, and use X - + ty Y -0
20 + t as shorthand.
For the second step, we transform Peam to the image coordinate system:
f
0
Cx
X
fx + cod
Pimg
K X P cam
0
f
Cy
+
Y
fY + cyd
0
0
d
d
For the last step, we collapse depth and project Pimg onto the imaging plane to obtain the values of (u, v):
fx
+
Cx
P2D = Pimg/ d
fY
+ Cy
v
1
1
In summary:
Pimg = d(u, v, 1)
Ping =
K x Peam = K x R X Po + +t)
So:
diu,v, v, 1)T = K(Rpo + t)
Assuming that the points are represented in homogeneous coordinatee, then Po
(xo, yo, 20, 1) T 1)T and we can simplify the relationship:
R t
v, = K(Rpo +t) = KK(
+t)=
Po) = KTPp
0

---

Now:
d(u, U, 0, = -KTpo,
where
R t
T=
d 20 t
0 1
Derivation of Eq. 1
With the derivation and notations of Eq. 3, we can directly derive Eq. 1. Assume that a 3D box (e.g., a car) is centered at Po. and the origin of ego frame is on the ground, then 2zo is the height dimension of the box. We project P1 (I0, yo, 0) and p2 = (xo, yo, 2z0) onto the image to get the upper and lower bounds of the box, and the difference between them should be the pixel height of this box. From the derivation of Eq. 3, we know:
fY
f(- 20 + tx)
v
d
d
Similarly, for P1 and P2 we have:
ftz d
U1
+ Cy.
f(-220 + tz)
U2
+ Cy
d
So the pixel height of the box would be:
2z0
2z0
heighteD = |v1 U2
f x
X
d
I0
t
Let the 3D size S = 2z0, and the 2D pixel size Spizel
heightaD, then we have the Eq. 1 in
main paper:
Spizel = ff x S
E0 - t


Reviews:
[{"review_id": "nu4vPyhAhx", "reviewer": "Reviewer_vndD", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a method to make 3D object detectors generalize by focusing on cross-dataset sensor-setup alignment. This paper argues that differences in camera intrinsics and extrinsic, and ego-frame alignment, also attributed as \"sensor misalignment\" plays a significant role in the generalization capability of a detector. To support this extensive empirical studies were conducted on 6 different datasets. The proposed simple sensor alignment approaches with intrinsic Synchronization, EAM module, and ego-frame alignment are able to address the aforementioned problems. Despite their simplicity, they are proven to work very well in practice and improve the mAP metric substantially. ", "summary_of_recommendation": "3D object detection is a very relevant and important problem in the field of robotics. This paper explains the fundamental problems of why just adding new data can not improve the performance of a detector and did a very good job of addressing those problems with simple yet effective solutions. These kinds of findings (both theoretical and empirical) are very crucial for the advancement of research in this direction. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Please see the above comments", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall, this paper is well-motivated, very well-written, and easy to follow. The visualizations made it easier to understand the problems and solutions. Lastly, the results are impressive. \n\nThere are a few bits in the main paper that were a bit difficult for me to understand. I would appreciate it if the authors can address them in the rebuttal. \n\n1. In the abstract, line 5, the authors mentioned about weather conditions being the same between datasets can be a cause of overfitting. However, there is no mention of this issue in the main paper. Even the augmentations are dropped (line 129). I would like to know why. \n\n2. Adding more datasets results in more data for each of the categories. However, it can also exacerbate the data-imbalance problem. For example, adding KITTI-360 that has only vehicles but no cyclists or pedestrians. I would like to know if the authors looked at the data statistics and addressed this problem during training.\n\n3. I could not understand the derivation of Eq. 1 and Eq. 3. I would appreciate it if the authors can explain how those equations are extracted from the pinhole camera model. Also, please give the dimensions of each variable (S, s_pixel, and p0) used in them. \n\n4. In lines 287-288, I could not understand how the optimal ego settings are selected based on Figure 4. Please explain it a bit more. \n\n5. Lastly in the limitations section, the authors mention that they could not study the impact of camera rotation. By this do they mean most of the data in the datasets is collected on a straight road? If yes, can the evaluation be done on the isolated sequences where the vehicle is turning? ", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "LJnV3JV3tU-", "reviewer": "Reviewer_iJm9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work focus on generalizable camera-only 3D detector for self-driving: train on one dataset and test on the others.\nSpecifically, the challenges exist in intrinsic, extrinsic and ego-coordinate system difference.\nThis work propose some simple yet effective solution to address these issues: resize image to match the intrinsic,  Extrinsic Aware Module  to handle the extrinsic and ego-frame alignment to handle different ego height..\nExtensive experiments are conducted on 6 datasets to validate the effectiveness.", "summary_of_recommendation": "Overall, this work studies an interesting task. But the baselines are a bit too weak, the solution is somewhat trivial but do shows improvement. \nI gave an weak accept because it's an interesting task/setting that no other work has addressed (but my reading is limited).\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please address my comments on the weakness section.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Strength\n1. Interesting and important task: make the camera-based 3D detector generalizable.\n2. Extensive experiments and convincing results.\n3. Clear writing (but a bit wordy, especially in the table/fig)\n\n## Weakness\n1. The baseline performance is too trival and a bit lack practicality. I think no one will directly use a detector trained on different sensor configurations. \nA reasonable setting for monocular 3D detection is to detect the object with respect to the camera origin, and then transform to vehicle frame by applying the extrinsics.\nFor example, a monocular detector could be designed in this way: first detect 2D bbox, then predict depth, rotation and box3d size, and then transform the vehicle frame. We can applied the intrinsics and extrinsics in the post-processing, so I think it can be generalizable to different dataset.\n2. No experiments on boosting the performance with extra data. e.g. train with 5+1 dataset and test on the last one,and see if the performance can be improved with more trained dataset.\n\n", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "KVYhbH-ehoO", "reviewer": "Reviewer_JvHZ", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In this work, the authors investigate the generalization of visual 3D object detection in autonomous driving across datasets by aligning the sensor configuration. The authors claim that simply aligning the sensor configuration including camera intrinsic, camera extrinsic, and ego coordinate system can significantly mitigate the performance gap between different datasets.", "summary_of_recommendation": "Overall, I think the paper is well-written and well-structured. However, the problem and method proposed by the authors are trivial to me and very specific to the chosen baseline. There is not much novelty w.r.t. the algorithm. Therefore, I would lean toward a weak rejection.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Questions:\n\n  In Sec 4.2, could the authors explain why the image patch and p_0 are invariant to t_x according to the equation in L.258?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pro:\n\n  1. The authors conducted comprehensive experiments to show that simply combining multiple datasets cannot improve the model’s generalization capability if the datasets are collected with different setups (with different data distributions).\n  2. This paper is well-written, well-structured, and easy to follow.\n\nCons:\n\n  1. The performance drop due to different camera intrinsic/focal lengths between training and testing datasets is trivial to me. Resizing images from their original focal length to a predefined focal length sounds reasonable but cannot address the spatial misalignment among the datasets during collection (ratio/scale of the object in the image) and sounds trivial. \n\n  2. Sec 4.2 regarding the effect of camera extrinsic is unclear to me, in order to get the recipient field in Fig.3(b), a real-world size S of the object is required, which is not possible in real-world given various types of cars. Furthermore, I believe combining multiple datasets with wide extrinsic range could alleviate this problem.\n\n  3. The Intrinsic Synchronization proposed by the authors requires the camera intrinsic of each image/dataset to be given as prior knowledge, which somehow still limits the generalization in the wild. \n\n  4. The authors proposed three modules, namely intrinsic, extrinsic, and ego frame alignment, which to me are trying to do the same thing, i.e., shifting multiple datasets to the same distribution. Especially the ego frame alignment is achieved by grid search and seems specifically useful to the query-based method like DETR3D, which also limits the generalization. For example, I think a pure image-based detector with large-scale multiple datasets for training can simply deal with this problem.\n\nMinor:\n\n  1. I believe there’s a typo in L. 136: ….thresholds at 0.5, 0.3 and 0.3…\n\n  2. L. 138,  average the -> the average\n\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "8psyn_OYKS", "reviewer": "Reviewer_NKfY", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work studies the generalization of camera-based 3D object detection models across different datasets for autonomous driving. It shows that existing models are highly sensitive to variations in sensor configuration, and proposes a method to align different sensor settings using camera intrinsic and extrinsic corrections, as well as ego frame alignment. These alignment steps are shown to be crucial for cross-dataset generalization (simply merging all datasets as they are and training yields much worse results) -- overall, the results include 6 common AV object detection datasets and two 3D object detection models, and improvements are shown for all.", "summary_of_recommendation": "As mentioned, really great empirical results across a wide range of AV datasets! The paper requires quite a bit of polishing / clarifications prior to acceptance, however (see \"WEAKNESSES\"). Weak reject for now, then let's see during the rebuttal.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please see my bolded questions / statements under \"WEAKNESSES\" and prioritize those. Then try to address any other comments that include question-marks, and or things that should be clarified.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**STRENGTHS**\n\n* The paper's contributions result in **very significant (game-changing) cross-dataset generalization results in the realm of 3d detection for AV datasets**.\n\n* The above is **achieved with quite simple data preprocessing tricks** -- nice!\n\n* **The contribution of each of the three different alignment techniques are well-motivated** (see Section 3.4) and empirically (see e.g. Table 3, Table 4, and Table 4 in the SM).\n\n* Good that so **many (6) datasets are used** and evaluated on.\n\n* Good that **more than one detection algorithm is tested** (both DETR3D and BEVDet).\n\n**WEAKNESSES**\n\n* The paper focuses on 3d object detection for AV datasets. **I feel however that the text is lacking some discussion around the same sensor alignment methodology on other vision tasks for AV datasets** -- do the authors hypothesize that similar sensor alignments are helpful there too? Why/why not? Have others done that already perhaps?\n\n* **Quite a few things were not clear to me and/or some things were structurally a bit off:**\n    - I didn't understand why the mAP was the same for Argo2 and Waymo in Fig. 2 (?).\n    - Line 62: \"... over 20 mAP on most datasets\". Line 70: \"... more than 25 mAP on all datasets\" <-- these statements are somewhat inconsistent.\n    - Line 97: \"... avoid employing data augmentation ... \" <-- this is unclear to me.. why would one skip that? In particular, the data augmentation might have helped the other approaches that do not apply the various alignment techniques (**making the overall comparisons less fair (?)**).\n    - Line 118: Unclear to me what MMDetection3D is. Please add some reference in the paper.\n    - Line 136: \"... thresholds at 05, 0.3 and 0.3 ... \" <-- I guess one of these is a typo (?).\n    - Some tables occur quite far from where they are mentioned and in \"reverse\" order (so Table X may be higher up in the paper than Table Y, even though X was introduced after Y). Table 3 is for example quite far away.\n    - Speaking of tables, the various terminologies in the table are not introduced prior to introducing the table (e.g. \"Direct\"). Please add all abbreviations and explain all properties / settings clearly before a table gets introduced.\n    - There are quite a few numbers / tables (good!) but it can get hard to grasp everything clearly, so I would recommend adding another column besides \"avg.\" which is something like \"avg-not-same\" i.e. which takes the average only for those DIFFERENT datasets (thus avg-not-same explicitly averages the scores for cross-dataset generalization, by not including the same dataset on which it was trained). It could also be good to highlight the best entries in each table with bold or similar.\n    - Why is \"Sync Intrinsic\" for N different in Table 4 compared to Table 3? (they are the same for \"Direct\" N and \"Sync Extrinsic and Ego center\").\n    - Line 312: '... lack of autonomous driving datasets with highly diverse camera poses ... ' <-- **is that true, why would there be so few camera poses?**\n\n* **Some things are a bit lacking regarding experimental evaluations, e.g:**\n    - I understood why one had to take the intersection among all classes to get a universal method. However, **it would also have been good to add some results which include more than just those 3 categories** (that would imply that some of the 6 datasets would have to be omitted). The models could then anyway be evaluated on the omitted datasets (albeit not for all categories, as some are missing from those datasets).\n    - **It was not clear to me whether \"Sync Extrinsic and Ego center\" also includes intrinsic synchronization (?)** If not, I recommend showing results for when using all three things at the same time.\n    - **For the other BEVDet detector (SM), why did you only use the \"Direct\" approach for its evaluations, i.e. why did you not showcase your sensor adjustment strategy here?**\n\n* The paper seems **quite hastily written with quite a few typos** across the paper (and SM), such as:\n    - Line 88: No space between \"domain\" and \"[20, 21, 22, 23, 24, 25, 26, 27, 28]\" (this happens in many other places too).\n    - Line 88: \"Unlike theses methods are trained on ...\" <-- \"Unlike these methods WHICH are trained on ..., our research ...\" (skip 'whereas' here too).\n    - Line 99: \"... are several works have ... < \"... are several works THAT have ...\"\n    - Line 104: There is too little vertical space between this line and Table 1.\n    - Line 169: \"... the data diversity, We ...\" <-- should be small 'w' on 'we'.\n    - Line 192: \"Fig. 1 and Fig. 1\" <-- duplicate\n    - Etc etc <-- please go over the paper in detail to fix all typos etc.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates the generalization of camera-based 3D object detectors across different datasets for autonomous driving – showing that existing models are highly sensitive to variations in sensor configuration, and proposes a method to align different sensor settings using camera intrinsic and extrinsic corrections, and ego frame alignment. Experiments show that the proposed method for cross-dataset transfer improves over simply merging all datasets as they are, with results on 6 AV object detection datasets and two 3D object detection models.

Reviewers agree that paper is interesting (iJm9, NKfY), well-motivated (vndD, iJm9), easy to follow (vndD, JvHZ), with convincing results across datasets (vndD, iJm9, NKfY). For the rebuttal, reviewers requested additional experiment clarifications and statistics (vndD, iJm9, JvHZ, NKfY) as well as additional experiments with sensor adjustment on BEVDet (NKfY). Post-rebuttal, the ratings have improved to strong accept, weak accept, weak reject, weak accept. There are still a few outstanding concerns on novelty, and the constraints on the design of the base detector (JvHZ) – in particular, the writing may benefit from including a brief discussion on the latter.

Overall, while the paper does not introduce a new algorithm, it shows that tracing the problem back down it roots (i.e. the data) may yield effective and simple solutions. Cross-dataset transfer is also a timely topic in the community, and this work should encourage data retrospection that extends beyond the AV space. I believe the paper to be an excellent reminder of the value behind a first-principles approach, and presents a refreshing take on how "fixing existing data" can be just as (if not more) fruitful at time when "scaling up / more data" remains all the rage.
Decision: Accept (Poster)