Unified Domain Generalization and Adaptation for Multi-View 3D Object Detection
Authors: Gyusam Chang
Jiwon Lee
Donghyun Kim
Jinkyu Kim
Dongwook Lee
Daehyun Ji
Sujin Jang
Sangpil Kim
Venue: NeurIPS 2024 poster (2024)
Keywords: Domain Generalization.+Domain Adaptation.+Multi-view 3D Object Detection.+Autonomous driving.+Domain Generalization.
Primary area: machine_vision
TL;DR: Label-Efficient Domain Adaptation for Multi-view 3D Object Detection

Abstract: Recent advances in 3D object detection leveraging multi-view cameras have demonstrated their practical and economical value in various challenging vision tasks.
However, typical supervised learning approaches face challenges in achieving satisfactory adaptation toward unseen and unlabeled target datasets (i.e., direct transfer) due to the inevitable geometric misalignment between the source and target domains.
In practice, we also encounter constraints on resources for training models and collecting annotations for the successful deployment of 3D object detectors.
In this paper, we propose Unified Domain Generalization and Adaptation (UDGA), a practical solution to mitigate those drawbacks.
We first propose Multi-view Overlap Depth Constraint that leverages the strong association between multi-view, significantly alleviating geometric gaps due to perspective view changes.
Then, we present a Label-Efficient Domain Adaptation approach to handle unfamiliar targets with significantly fewer amounts of labels (i.e., 1$\%$ and 5$\%)$, while preserving well-defined source knowledge for training efficiency.
Overall, UDGA framework enables stable detection performance in both source and target domains, effectively bridging inevitable domain gaps, while demanding fewer annotations.
We demonstrate the robustness of UDGA with large-scale benchmarks: nuScenes, Lyft, and Waymo, where our framework outperforms the current state-of-the-art methods.

Unified Domain Generalization and Adaptation for Multi-View 3D Object Detection
Gyusam Chang" Dongwook Leea
Jiwon Lee Dachyun Ji
Donghyun Kim Sujin Jang2t
Jinkyu Kim Sangpil Kim
Korea University
Samsung Advanced Institute of Technology (gajang98 d d_kim, jinkyukim, spk7 @korea... kr {ji1.lee, dw12.lee, derek. ji, s s.steve. ..steveejang jang) @samsuna com
Abstract
Recent advances in 3D object detection leveraging multi-view cameras have demon- strated their practical and economical value in various challenging vision tasks. However, typical supervised learning approaches face challenges in achieving satis- factory adaptation toward unseen and unlabeled target datasets (i.e., direct transfer) due to the inevitable geometric misalignment between the source and target do- mains. In practice, also encounter constraints on resources for training models and collecting annotations for the successful deployment of 3D object detectors. In this paper, we propose Unified Domain Generalization and Adaptation (UDGA), a practical solution to mitigate those drawbacks. We first propose Multi-view Overlap Depth Constraint that leverages the strong association between multi-view, significantly alleviating geometric gaps due to perspective view changes. Then, we present Label-Effficeet Domain Adaptation approach to handle unfamiliar targets with significantly fewer amounts of labels (i.e., 1% and 5%). while preserving well-defined source knowledge for training efficiency. Overall, UDGA framework enables stable detection performance in both source and target domains, effectively bridging inevitable domain gaps, while demanding fewer annotations. We demon- strate the robustness of UDGA with large-scale benchmarks: nuScenes, Lyft, and Waymo, where our framework outperforms the current state-of the-art methods.
Introduction
3D Object Detection (3DOD) a pivotal computer vision task in various real-world applications such autonomous driving and robotics. Recent progress in 3DOD [+4 have showcased remarkable advancements, primarily due to the large-scale benchmark datasets (5 and the introduction of multiple computer vision sensors (e.g.. LiDAR, multi-view cameras, and RADAR). Among these,
camera-bar baserd-baaed multi-view 3DOD g 12 has drawn significant attention for its cost-efficiency and rich semantic information. However, significant challenge remains largely unexplored: accurately detecting the location and category of objects in the presence of distributional shifts between the source and target domains (i.e., data distributional gaps between the training and the testing datasets).
To successfully develop and deploy Multi-view 3DOD models, we need solve two practical problems: (1) the geometric distributional shift across different sensor configurations, and (2) the limited amount of resources (e.g., insufficient computing resources, expensive data annotations). The first problem poses challenge in learning transferable knowledge for robust generalization in novel
'These authors contributed equally.
Corresponning authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

SoumelLyt)
T Targee(nuScenes)
Average
0.52
0.51
0.52
0.56
0.49
0.44
0.45
Domain ddapiation
Torget 5%
Oracle
CAM-
DG-BEV
PD-DEV
PD-BEV
Ours (DG)
DT +Full FT
Ours (UDGA)
Convs
(UDA)
Figure 1: Comparison of performance in both source and target domains (Tab, 6). Here, "Average" (orange dots) refers to mean NDS both the source and target domains. We draw comparisons with prior methods CAM-Conv [3]. DG-BEV (14) and PD-BEV [I5] offering an empirical lower and upper bounds, DT and Oracle. Note that we only use 5% of the target label for Domain Adaptation.
domains. The second issue inevitably requires efficient utilization computing resources for training and inference, as well as label-efficient development of 3DOD models in practice. To tackle these practical problems, we introduce Unified Domain Generalization and Adaptation (UDGA) strategy, which addresses series of domain shift problems (i.e., learning domain generalizable features significantly improves the quality of parameter- and label-efficient few-shot domain adaptation).
Prior studies aim to learn domain-agnosti knowledge alleviating domain shifts from drastic view changes in cross-domain enviromments. DG-BEV [14] disentangles the camera intrinsic parameters and trains the network with a domain discriminator for view-invariant feature learning. Similarly, PD-BEV (15) renders implicit foreground volumes and suppresses the perspective bias leveraging semantic supervision. However, these approaches struggle to capture optimal representations. high- lighting that there still room for improvements novel target domains (i.e., up to -50.8% Closed Gap compared to Oracle). To tackle these drawbacks, first advocate a Multi-view Overlap Depth Constraint that leverages occluded regions between adjacent views, which serve as notable triangular clues to guarantee geometric consistency. This approach effectively addresses perspective differences between cross-domain environments by directly penalising the corresponding depth between adjacent views, and shows considerable generalization capacity (up +75.8% Closed Gap compared to DT).
Nevertheless, the development of algorithms running on edge devices (i.e., autonomous vehicles) faces the challenge of limited resources, which requires efficient utilization of computing systems. To resolve these challenges, we carefully design a go-to strategy, Label-Efficient Domain Adaptation, that bridges two different domains with cost-effective transfer learning. Precisely, motivated by Fine-Tuning (PEFT) [6)18), we focus on smooth adaptation to target domains by fully exploiting well-defined source knowledge. Specifically, leveraging plug-and-ple extra parameters, we substantially adapt to target domains while retaining information from the source domain (+14% Average gain compared to DT+ FT shown in Fig. 1.. As result, note that UDGA practically expand base models, efficiently boosting overall capacity under limited resources.
Parameter-Effiiee
Given landmark datasets in 3DOD, nuScenes (6,, Lyft 7 and Waymo [5], we validate the effective- ness our UDGA framework for the camera-based multi-view 3DOD task. Notably, we achieve state-of-the-art performance in cross-domain environments and demonstrate the component-wisg effectiveness through ablation studies. To summarize, our main contributions are as follows:
We introduce the Unified Domain Generalization and Adaptation (UDGA) framework, which aims to learn generalizable geometric features and improve resource efficiency for enhanced practicality in addressing distributional shift alignments.
We advocate depth-scale consistency across multi-view images to effectively address 3D geometric misalignment problems. To this end, we leverage the corresponding triangular cues between adjacent views to seamlessly bridge the domain gap.
We present label- and parameter-effiiient domain adaptation method, which requires fewer annotations and fine-tuning parameters while preserving source-domain knowledge.
We demonstrate the effectiveness of UDGA multiple challenging cross-domain bench- marks (i.e., Lyft - nuScenes, nuScenes >> Lyft, and Waymo nuScenes). The results show that UDGA achieves new state-of-the-art performance in Multi-view 3DOD.

---

2 Related Work
2.1 Multi-view 3D Object Detection
3D object detection 4. 19 20 26 is fundamental aspect of computer vision tasks in the real world.
Especially, Multi-view 3D Object Detection leveraging Bird's Eye View (BEV) representations [
12 [] have rapidly expanded. We observe that this paradigm is divided into two categories: (i) LSS- based (27 [2] and (ii) Query-based [8 IO]. The former adopts explicit methods leveraging depth estimation network, and the latter concentrates on implicit methods utilizing the attention
mechanism of Transformer [29]. Recently, these methods 19 30 31 significantly benefit from improved geometric understanding leveraging temporal inputs. Also, methods 32: 35
that directly guide the model using the LiDAR teacher model significantly encourage BEV spatial details. In particular, this approach is being adopted to gradually replace LiDAR in real-world scenarios; however, still suffers from poor generalizability due to drastic domain shifts (e.g., weather, country, and sensor). To mitigate these issues, we present a novel paradigm. Unsupervised Domain Generalization and Adaptation (UDGA), that effectively addresses geometric issues leveraging multi-view triangular clues and smoothly bridge differenet domains without forgetting previously learned knowledge.
2.2 Bridging the Domain Gap for 3D Object Detection
Due to the expensive cost of sophisticated sensor configurations and accurate 3D annotations for autonomous driving scenes, existing works strive to generalize 3D perception models in various data distributions. Specifically, they often fail to address the covariate shift between the training and test splits. To bridge the domain gap, existing approaches have introduced noteworthy solutions below.
baased Wang et al. [6 introduced Statistical Normalization (SN) mitigate the differences in object size distribution across various datasets. ST3D (37 leveraged domain knowledge through random object scale augmentation, and their self-training pipeline refined the pseudo-labels. SPG (38) aims to capture the spatial shape, generating the missing points. 3D-CoCo [39] contrastively adjust the domain boundary between source and target to extract robust features. LiDAR Distillation 440 generates pseudo sparse point sets in spherical coordinates and aligns the knowledge between source and pseudo target. STAL3D 41 effectively extended ST3D by incorporating adversarial learning. DTS [42 randomly re-sample the beam and aim to capture the cross-density between student and teacher models. CMDA 2 aim to learn rich-semantic knowledge from camera BEV features and adversarially guide seen sources and unseen targets, achieving state-of-the-art UDA capacity.
Camera-based. While various groundbreaking methods based on LiDAR have been researched, camera-based approaches are still limited. Due to the elaborate 2D-3D alignment, not only are LiDAR-based approaches not directly applicable, but conventional 21) visual approaches 43 46
cannot be adopted either. To mitigate these issues, STMono3D [47 self-supervise the monocular 3D detection network in teacher-student manner. DG-BEV 114] adversarially guide the network
from perspective augmented multi-view images. PD-BEV [5 explicitly supervise models by the RenderNet with pseudo labels. However, camera domain generalization methods cannot meet the performance required for the safety, struggling to address the practical domain shift the perspective change. To narrow the gap, we introduce Unified Domain Generalization and Adaptation (UDGA) framework that effectively promotes depth-scal consistency by leveraging occluded clues between adjacent views and then seamlessly transfers the model's 's potential along with few novel labels.
2.3 Parameter Efficient Fine-Tuning
Recent NLP works fully benefit from general-purposs Large-language Models (LLM). Additionally, they have proposed Ctrameter-ffficicee Fine-Tuning (PEFT) [6 50 to effectively transfer LLM power to various downstream tasks. Specifically, PEFT preserves and exploits previously learned universal information, fine-tuning only additional parameters with a few downstream labels. This paradigm enables to notably reduce extensive computational resources, and large amounts of task-specific data and also effectively address challenging domain shifts in various downstream tasks as reported by [51]. Inspired by this motivation, to address drastic perspective shifts between source and target domains, we design Lffeie-ifiiient Domain Adaptation that fully transfers generalized source potentials to target domains by fine-tuning only our extra modules with seo-shot target data.
3

---

Source
-33
-54
-39
-53
Target
-60
-0- NDS
@ @-MAP
-67
-64
-80
Front Left
Front
Front Right
Source
AHeight
APitch
AAlI
(a) Translated perspective views
(b) Performance Drop compured to Source
Figure 2: (a) An illustration of multi-view installation translation difference. The first (i.e., source)
and second (i.e., target) rows are two perspective views of the same scene captured from different installation points. The translation gap between these views is substantial, approximately 30%. (b) Source trained network shows poor perception capability in target domain, primarily due extrinsic shifts. In Heiigh, mAP and NDS have dropped up to -67% compared to source. Note that we simulate the camera extrinsic shift leveraging CARLA 52 (refer to Appendix A for further details).
3 Methodology
3.1 Preliminary
Multi-view 3D Object Detection fundamental computer vision task that involves safely localizing and categorizing objects in a 3D space exploiting 2D visual information from multiple camera views.
Especially, recent landmark Multi-view 3D Object Detection models I0 33] are formulated as follow; arg min L(Y, D(V(I, K, T)). where Y represents the size (1, w. h), centerness (cx, cy, ez). and rotation of each 3D object. Also, I = (2 .2.,, }n Rx x3 K, and denotes multi-view images, intrinsic and extrinsic parameters. Specifically, these models, which fully benefit from view transformation modules V, encode 2D visual features alongside the 3D spatial environment into bird's eye view (BEV) representation. First, these works adopts explicit methods (BEV view transformation P as shown in Eq. [ exploiting depth estimation network. Subsequently, Detector Head modules D supervises BEV features with 3D labels Y in three- dimennionni manner.
[R]]
V(I, K, T) P(2 & D,K,T),
3.2 Domain Shifts in Multi-view 3D Object Detection
In this section, we analyze and report de facto domain shift problems arising in the Autonomous Driving system. shown 3.T. recent works adopt camera parameters K and T as extra inputs in addition to multi-view image 7. As reported by [14), assuming that the conditional distribution of outputs for given inputs, is the same across domains, it is explained that shifts in the domain distribution are caused by inconsistent marginal distributions of inputs. To mitigate these issues, recent generalization approaches [14 53
13 (44 often focus on covariate shift in geometric feature representation mainly due to optical changes (i.e., Focal length, Field-of- View, and pixel size).
This the only part of story. We experience drastic performance drops (up -54% / -67% performance drop in NDS and mAP, respectively, as shown in 2 (b)) from non-intrinsic factors (i.e., only extrinsic shifts). Especially, we capture phenomenon wherein the actual depth scale from an ego-vehicle's 's visual sensor to an object (Fig 2 (a) red boxes) varies depending on the sensor's s installation location. Followed by Pythagorean theorem, as the height difference Åh increases, the depth scale difference Ad also increases accordingly. Note that this is not limited to height solely; any shifts in deployment translation (e.g., along the x, y, or axis) lead to changes in actual depth scale. As a result, perspective view differences significantly hinder the model 's dimensianaa geometric understanding by causing depth inconsistency. To address above drawbacks, we introduce novel penalising strategy that effectively boost depth consistency in various camera geometry shifts.

---

Mult-viiw
View Transformer
Translation
BEV Nock
BEV Pool Pool
BEV
Scale
DeptiNel
Finetune
Orientation
(i Multi-view Overlap Depth Constraint
Label-Efficient DA
Depth projection adjacent views
Dyp Dipi(du, dv.
T((KT=))
Oper. Block
Adapter
Convolution
Project
Ka, (Te,
Kay (Toz
Norm layer
Activation
Fadjs
Fadh
Activation
Project Up
Eum
Lavz
Photometric Reprojection
(2)
Oper. Block
Adapter
adjjecett
Ind/s
Jad
+
An overview of our proposed methodologies. Our proposed methods comprise two major parts: (i) Multi-view Overlap Depth Constraint and (ii) -ffel-Efficient Domain Adaptation (LEDA). In addition, our framework employs two phases (i.e., pre-training, and then fine-tuning). Note that we adopt our proposed depth constraint in both phases, and LEDA only the fine-tuning phase.
Figure 3:
3.3 Multi-view Overlap Depth Constraint
augment multi-view images to generalize challenging perspective view gaps. However, these strategies suffer from poor generalizability in cross-domain scenarios, primarily due to the underestimated extent of view change between different sensor deployments as reported in section 3.2 To alleviate perspective gaps, introduce Multi-view Overlap Depth Constraint, effectively encouraging perspective view-invariant learning. Here, start from three key assumptions: First, perspective shifts between adjacent cameras in multi-view modalities are non-trivial and varied, closely akin to those observed in cross-domains (e.g., nuScenes >> Lyft). Second, visual odometry techniques such Structure from Motion (SfM) and Simultaneous Localization and Mapping (SLAM) often benefit from improved depth consistency through relationships between adjacent views (e.g., relative pose estimation). Third, in multi-view modalities, overlap regions serve as strong geometric triangular clues, seamlessly bridging between adjacent views. However, under conditions where camera parameters input, off-the-shelf pose estimation [17 leads to ambiguity in learning precise geometry. To mitigate these issues, we introduce a novel depth constraint (Fig. (i)) with overlap regions between adjacent cameras.
Motivation. Recently, previous efforts 55
Approach. To achieve generalized BEV extraction, directly constrain depth estimation network from adjacent overlap regions between multi-view cameras. Also, we advocate that multi-frame image inputs substantially complement geometric understanding in dynamic scenes with speedy translation and rotation shifts. To this end, we formulate corresponding depth D leveraging spatial and temporal adjacent views. First, we calculate overlap transformation matrices Ti- from Eq.
Di K2(T, (KK
(2)
where K and 7 are the intrinsic and extrinsic camera parameters. pī-j and denote correspond- ing pixels between adjacent views and / represent depth prediction. Then, we directly penalise unmatched corresponding depth D* to smoothly induce perspective-agnosti learning as follow Eq.
Then.
Lou Σ ((D,DD'.)) (1.f)
where d represents Fuclidean Distance. Also, we observe that the photometric reprojection error significantly alleviate relative geometric ambiguity. Especially, slow convergence may occur mainly
5

---

due to incorrect relationships in small overlap region (about 30% of full resolution). To mitigates these concern, we effectively boost elaborate 2D matching. formulating C, as follow Eq.
hp Σ
1,, ; Ti-sj, P(+)))
(4)
(1.5)
where represents point clouds generated by D, and pe is photometric error by SSIM 62] Also, (t) denotes billnear sampling on RGB images. Concretely, we take two ad advantages leveraging Lp narrow occluded regions: First, Ep effectively mitigates the triangular misalignment. Second, Cp potentially supports insufficiently scaled Cov. Ultimately, we alleviate perspective view gaps by directly constraining the corresponding depth and the photometric matching between adjacent views.
3.4 Label-Efficient Domain Adaptation
Motivation. There exist practical challenges in developing and deploying multi-view 3D object detectors for safety-critical self-driving vehicles. Each vehicle and each sensor requires its own model that can successfully operate various conditions (e.g., dynamic weather, location, and time). Furthermore, while collecting large-scale labels in diverse environments is highly recommended, it is extremely expensive, inefficient and time-consuming. Among those, we are particularly motivated to tackle the following: (i) Stable performance, (ii) Efficiency of training, (iii) Preventing catastrophic forgetting, and (iv) Minimizing labeling cost. To satisfy these practical requirements, we carefully design an efficient and effective learning strategy, Label-Efficient Domain Adaptation (LEDA) that seamlessly transferring and preserving their own potentials leveraging a few annotated labels.
Approach. In this paper, we propose Label-Efficient Domain Adaptation, a novel strategy to seamlessly bridge domain gaps leveraging a small amount of target data. To this end, we add extra parameters [48] consisting of bottleneck structures (i.e., projection down down and up @up layers).
A(x) (o (zN())))
(5)
where a and BN indicates activation function and batch normalization. We parallelly build A alongside pre-trained operation blocks B (e.g., convolution, and linear block) in Fig. and Eq. 6
y = (xx) + A(x:,
Firstly, we feed x into Odown to compress its shape to [H/r, W/rl, where r is the rescale ratio, and then utilize dup to restore to [H. WT. Secondly, we fuse each outputs from B, and Adapter by exploiting skip-connections that directly link between the downsampling and upsampling paths. By doing so, these extensible modules allow to capture high-resolution spatial details while reducing network and computational complexity. Plus, it notes worthy that they are initialized by a near-identity function to preserve previously updated weights. Finally, our frameworks lead to stable recognition in both source and target domains, incrementally adapting without forgetting pre-trained knowledge.
3.5 Optimization Objective
In this section, we optimize our proposed framework UDGA using the total loss function Crotal (as shown in Eq. 7 during both phases (i.e., pre-train and fine-tune). Ldet denotes the detection task loss.
Liotal Aaet Coa + >pp
(7)
where we grid-search Adet, Aos and Ap to harmonize Cdets Lov and Lp. Specifically, Liorat supervises during generalization and A during adaptation, respectively. As result, these strategies enable efficient learning of optimal representations in target domains while preserving pre-trained ones.
4 Experimental Results
In this section, we showcase the overall performance of our methodologies on landmark datasets for 3D Object Detection: Waymo (5), Lyft [7]. and nuScenes [6]. The three datasets have different specifications; thus, we convert them to a unified detection range and coordinates for accurate comparison. We also adopt only seven parameters to achieve consistent training results under the same conditions: the location of centerness (x. y, 2), the size of box (1, w. lt), and heading angle Additionally, summarize 3D Object Detection datasets and implementation details in Appendix A

---

Table 1: Comparison of Domain Generalization performance with existing SOTA techniques. The
bold values indicate the best performance. Note that all methods are evaluated on 'car' category.
Task
Method
NDS
mAP
mATE.
mASEJ
mAOE4
Closed Gapt
Oracle
0.587
0.475
0.577
0.177
0.147
Direct Tronsfer
0.213
0.102
1.143
0.239
0.789
CAM-Comys Convs
13
0.181
0.098
1.198
0.209
1.064
-8.6%
Lyft nuSeenes
Single-DGOD
44
0.198
0.105
1.166
0.222
0,905
-4.0%
DG-BEV 14
0.374
0.268
0,764
0.205
0.591
+43.0%
PD-BEV
0.344
0.263
0.746
0.186
0.790
+35.0%
0.421
0.759
+55.6%
Oracle
0.684
Direct Transfer
0.296
0.112
0,997
0.176
0.389
CAM-Convs
0.316
0.145
0.999
0.173
0.368
+5.256
noSeenes - Lyft
Single-DGOD
0.332
0.159
0.949
0.174
0.358
+9.3%
DG-BEV
0.437
0.287
0.771
0.170
0.302
+36.3%
PD-BEV
0,458
0.304
0,709
0,169
0,289
441.8%
Ours
0.487
0.324
0,709
0.162
0.180
+49.22%
Oracle
0.587
0.475
0.577
0.177
0.147
Direct Transfer
0.133
0.032
1.305
0.768
0.532
CAM-Convs
0.215
0.038
1.308
0.316
0.506
+18.1%
Waymo + nuSeenes
Single-DGOD
0.007
0.014
1,000
1.000
1,000
-27.8%
DG-BEV
0.472
0.303
0.689
0.218
0.171
+74.7%
Oracle
0.649
0.552
0.528
0.148
0.085
Direct Transfer
0.178
0.040
1.303
0.265
0,790
nuScenes Waymo
CAM-Convs
Single-DGOD
0.185
0.164
0.045
0.034
1,301
1,305
0.262
0.255
0.855
0,773
+1 +1.5%
-3.0%
DG-BEV
0.415
0.297
0.822
0.216
0.372
+50.3%
Ours
0.459
0.349
0.754
0.289
0.250
+59.7%
4.1 Evaluation Metric
In this paper, following DG-BEV 14 evaluation details, we adopt the alternative metric NDS (as shown in Eq. (8 that aggregates mean Average Precision (mAP), mean Average Translation Error (mATE), mean Average Scale Error (mASE), and mean Average Orientation Error (mAOE).
NDS
33AAP +
Σ (1-min1,, (TTn))
(8)
IIPETE
We reconstruct the unified category for Unified Domain Generalization and Adaptation as follows: the 'car' for nuScenes and Lyft, and the 'vehicle" for Waymo. Furthermore, we only validate performance in the range of x, axis from -50m to 50m. Note that we offer an empirical lower bound Direct Transfer (i.e., directly evaluating the model pre-trained in the source domain only), and an empirical upper bound Oracle (iee evaluating the model fully supervised in the target domain). We report Full F.T. (i.e., fine-tuning all parameters from the pre-trained source model) and Adapter (i.e., parameter efficient fine- mun-tunin without our proposed depth constraint methods from the pre- -reitra source model) Furthermore, we formulate Closed Gap-represnnting the hypothetical closed gap by
NDSmodel NDSpieet Transfer
Closed Gap
X 100%.
(9)
NDSomale NDSpirect Transfer
4.2 Experiment Results
Performance Comparison in Domain Generalization. As shown in Tab. we showcase four challenging generalization scenarios, and quantitatively compare our proposed methodology with existing state-of-fe of-t-off-hh-artt methods, which include CAM-Cony [13), Single-DGOD [44), DG-BEV [14], and PD-BEV [5). Here, we observe that these methods still struggle to fully pilot geometric shifts from perspective changes in cross-domain scenarios. Importantly, in Lyft - nuScenes, existing methods suffer from the orientation error mainly due to significantly different ground truth directions (i.e., only recovering 0.198 mAOE). In nuScenes Waymo (i.e., one of the most challenging

---

Table Comparison of UDGA performance on BEVDepth with various PEFT modules, SSF [50]. and Adapter [48] We construct six different target data splits from 1% to 100% Additionally, # Params denote the number of parameters for training. Note that represents 'Do not support'.
NDS mAPt
Task
Method
# Params
1%
5%
10%
25%
50%
100%
Onazle
51.7M
0.587
Full FTT
51.7M
0.476 0076/00 00366
0,515 00.434
0.547/0.443
0.577 /0.464
0.590
0.610 / 0.500
SSP 550
IM
1.245/0.079
0.294 00.1/0.112
0.266
0.374/0.200
0.421 /0.327
0,439/ /0.275
Lyft - nuScnnes
Adapter- B
21.3M
0,465 0..283
0.481 /0.365
0..31/
0.558
/0.444
0.569/0.400
/0.460
0.581 /0.473
Adapter-S
8.8M
0.326/0.134
0.
0.444
0.255
0.465/ 00,283
0,509 00039
0.538 /.443
Ours
8.8
0.526 0.404
0.563 70.444
0.573/ 0.457
1L592/0.448
0.609 /05510
0.614/ 0.507
Oracle
51.7M
0,684 0.602
FT
51,7M
0,531
0.594 70.473
0.650 0.650/0.549
/0.587
miScenes Lyf
SSF
Adapter-R
21.3M
1M
0,499/0 /9.38
0.316/0.115
0,355 0.154
0.556 0.466
0.386 / 0.185
0.584 00.45
0.633 /0,532
0,420/ /0,230
0.670/0 0,5564
0.447 / 002269
0.684 0.5566
0.470 0.300
Adapter-S
8.8M
0.420 0.230
0.463 0.325
0.500 / 0.356
0.537 / /0.400
0.561 / /0:426
0.573 /0.44
Ours
888
0.578 0.462
0.613 0.506
0.638 0.537
0.665 / 0.572
0.675 / 0.586
0.706 0.626
NDS
(Urs(UDGA)
Pull
NDS
Ours(UDGA)
Full FT Ovacie (0.68)
0.55
Adapter-S
Adapter-B
Oraele (0.59)
0.55
Adapicr-B
Ours(UDGA)
Ly0 -uuSemos
Full FT
nuSeenes +1yf
UDGA)
0.45
SSF
Adapter-B
Ours(DG)
0.15
Adapics-S
Full FT OursDD)
SSF
Adrpter-B
0.35
DG-BEV
0.35
Adapter-S
PD-BEV
o Adapter-S
DG-BEV PD-BEV
0.25
SSF
0.25
SSF
0.15
DD
100%
Direct Transfer (0.21)
0.15
DD
DA
DA 100%
Direct Transfer (0.30)
15
25
35
55 (M)
15
25
35
45
55 FP
Figure 4: Performance relative to training parameters. The Domain Generalization task is represented in blue, while the Domain Adaptation task is divided into two stages: 1% in gray and 100% in red.
scenarios due to the rear camera drop), previous approaches still show significant gap compared to Oracle (ie.,-49.7%) -49.7% Closed Gap). In this paper, our novel depth constraint notably addresses these issues, outperforming existing SOTAs (cspecially, up to +4.7% NDS and +12.6% Closed Gap better than DG-BEV in Lyft nuScenes). Especially, leveraging triangular clues to explicitly supervise occluded depth contributes significantly to improving geometric consistency compared to prior approaches 14 [5 44 [3). Overall, we demonstrate that our novel approaches significantly enhance perspective-invaree invariance, featuring strong association in occluded regions between multi-views.
Performance Comparison in UDGA. In Tab. 2 we show that our proposed Unified Domain Gener- alization and Adaptation performance compared with various PEFT approaches (i.e.. SSF [50], and Adapter (48). SSF directly scale and shift the deep features extracted by pre-trained operation blocks, leveraging additional normalization parameters. Adapter represents sole module performance without our proposed constraint; Adapter-B, B, and Adapter-S S denotes base, and small version, respectively.
Existing PEFT paradigms benefit from fine-tuning only extra parameters, retaining previously updated weights. However, we observe that these paradigms do not successfully adapt to the covariate shifts originated by challenging geometric differences as reported in section 3.2 More specifically, SSF and Adapter-S, which exploit a small number of parameters, begin to capture transfcrable representations and then marginally adapt the 10% data split. Also, Adapter-B B leveraging 21.3M parameters provide poor adaptation capability (i.e., inferior to Scratch and Full FT in Lyft + nuScenes 100%).
However, our proposed strategy seamlessly adapt to target domains in 1%, and 5%, effectively bridge perspective gaps. Furthermore, our proposed strategy show superior performance gain (outperforming Scratch in Lyft - nuScenes 50%, and Full FT in both Lyft- - nuScenes, and nuScenes - Lyft 100%). effectively adapting to novel targets. It is noteworthy that the most effective adaptation is achieved by updating extra parameters (less than 20% of the total), which demonstrates the practicality and efficiency of our novel UDGA strategy as shown in Fig. 4 addition, unlike Full FT, proves that our UDGA framework stably adapts to the target without forgetting previously learned knowledge as
8

---

Table Ablation studies on UDGA (10% Adaptation). and A represents pre-trained blocks and LEDA blocks, respectively. Note that we train B and A. alternatively (i.e., pre-train and fine-tune).
Pre-truin B (100% source)
Fine-tume A (10% target)
Lyft
nuSeenes
nuScenes
Lyft
ddet
Lae
Cp
Lea
Lou
Lp
NDS*
mAP1
NDS*
mAPt
0.213
0.102
0.296
0.112
d
0.403
0.262
0.485
0.323
/
0.421
0.281
0.488
0.309
V
0..44
0.255
0.500
0.356
0.516
0,407
0.590
0,482
0.552
0.441
0.632
0.530
0.638
0.537
0.573
0.457
Table 4: Ablation studies Domain Generalization with our novel depth constraint modules, Lou
and Cp. Lidar and SS each represents LiDAR depth supervision and Self-Supervised overlap depth.
Task
Lidar
SS
Method
NDS'
mAP1
mATE)
mASE!
mAOE!
La La Con Cy.
0.213
0.102
1.143
0.239
0.789
0.396
0.266
0.758
0.172
0.495
Lyft + nuSeenes
Dov
0.403
0.262
0.757
0.183
0.426
Lov Lop
0.407
0.265
0.747
0.179
0.428
Low Lip aug
0.421
0.296
0.112
0.997
0.176
0,389
La Liv Lyp.
0.483
0.327
0.718
0.163
0.204
muScenes lyyh
Low
0.485
0.323
0.731
0.161
0.171
Dov Lp.
0.487
0.324
0.709
0.162
0,180
Lov Lep
ext
0.488
0,309
0,705
0.169
0.123
shown in Fig. and Tab, 6 Overall, our proposed method demonstrate the effectiveness of training strategy in various experimental setups, efficiently expanding to targets with about 20% of overall parameters. Note that we report additional experiments and details of UDGA in Appendix C
4.3 Ablation studies
Exploring the Synergy Between Modules. To better understand the role of each module, we present ablation studies of UDGA in this experiment (Tab. Precisely, we aim to analyze the pros and cons in both training steps (i.e., pre-train B and fine-tune A), with the objective of effectively elucidating the plausibility UDGA. First, the strategy trained from scratch leveraging our depth constraint significantly recovers performance drop from the sensor deployment shift (up +20.8% NDS). However, this strategy finds it difficult to provide a practical solution for Multi-view 3DOD, mainly due to unsatisfying generalizability. Additionally, although LEDA without Lov and Cp yields improved performance, it fails to transfer its previously learned potential, resulting in only +2.3% NDS compared
individual depth constraint. To tackle these issues, concentrate bridging two distinct domains by capturing generalized perspective features. Especially, our depth constraint (only trained during pre-training B) significantly encourages understanding of the target in LEDA during fine-tuning A with a 10% split, addressing the geometric covariate shift (+30.3% NDS). Furthermore, UDGA strategy using Cov and C, in both phases learns the transferable knowledge and shows impressive improvement (+42.5% NDS). Finally, UDGA successfully presents an effective and efficient paradigm for Multi-view 3DOD, highlighting notable recovery novel target scenarios.
Effect of Overlap Depth Constraint. In Tab, we carefully evaluate our depth constraint compo- nents in various cross-domain environments. Here, La denotes depth supervision by LiDAR. Also, we design ext aug that globally rotate ground truths with randomly initialized angle to release the direction shift. More importantly, we observe that perspective view shifts from different sensor deployments lead severe translation and orientation errors. To tackle these issues, we advocate that Loe. which leverages strong relationships between adjacent views, effectively alleviating perspective gaps compared La (recovering up to +19% NDS Lyft - nuScenes). L, relieves slight mis- alignment, encouraging depth-scale consistency. Additionally, our exi aug substantially boost stable generalization, suppressing orientation emrors (up to +1 +1.4% additional NDS gain). Consequently. our novel objectives (Cov and Lp) demonstrate their efffcctiveness, significantly tackling geometric errors.

---

Front Left
Front
Front Right
60
GT
50
40
DEL CONTOR
30
10
0(m)
Figure Qualitative depth visualizations of front view lineups Lyft. The top row illustrates sparse depth ground truths projected from LiDAR point clouds. The middle and bottom rows are the qualitative results of BEVDepth and Ours, respectively. Yellow boxes highlight the improved depth.
4.4 Qualitative Analysis
To qualitatively analyze the effectiveness Multi-view Overlap Depth Constraint, we present additional visualized results in Fig. For accurate comparison, we conduct binary masking leveraging given sparse depth ground truths. In middle row, BEVDepth fail perceive hard samples (e.g., far distant and occluded objects) in yellow boxes, mainly due to different extent of deformation relative to perspective as reported in section 3.2 We aim to tackle this problem, explicitly bridging adjacent views in various dynamic scenes. Precisely, in bottom row, we showcase distinguishable results in yellow boxes, capturing semantic details from various view deformation. As as results, we qualitatively demonstrate that our proposed method effectively encourage depth consistency and detection robustness, significantly improving geometric understanding in cross-domain scenarios.
5 Conclusion
Limitations. While our work significantly improves the adaptability of 3D object detection, cannot guarantee seamless adaptation due several limitations, including: (1) The performance does not match that of 3D object detection models using LiDAR point clouds. (2) Our Multi-view Overlap Depth Constraint relies the presence of overlapping regions between images. (3) Achieving fully domain-agnostic approaches without any target labels remains challenging. As a result, it is essential incorporate a fallback plan when deploying the framework in safety-critical real-world scenarios.
Summary. Multi-View 3DOD models often face challenges in expanding appropriately unfamiliar datasets due to inevitable domain shifts (i.e., changes in the distribution of data between the training and testing phases). Especially, the limited resource (e.g., excessive computational overhead and taxing expensive and taxing data cost) leads to hinder the successful deployment of Multi-Vien View 3DOD. To mitigate above drawbacks, we carefully design Unified Domain Generalization and Adaptation (UDGA), practical solution for developing Multi-Vice 3DOD. We first introduce Multi-view Overlap Depth Constraint that advocates strong triangular clues between adjacent views, significantly bridging perspective gaps. Additionally, present Label-Efficient Domain Adaptation approach that enables practical adaptation novel targets with largely limited labels (i.e., 1% and 5%) without forgetting well-aligne source potential. Our UDGA paradigm efficiently tunettue additional parameters leveraging significantly fewer annotations by effectively transferring from source to target domain. In summary, our extensive experiments in various landmark datasets(e.g. nuScenes, Lyft and Waymo) show that our novel paradigm, UDGA, provide practical solution, outperforming current state-of-the the-art models Multi-view 3D object detection.
10

---

Acknowledgments and Disclosure of Funding
This work was primarily supported by Samsung Advanced Institute of Technology (SAIT) (85%), partially supported by Institute of Information & communications Technology Planning & Evaluation (ITP)) grant funded by the Korea government (MSIT) (No. 2019-0-00079, Artificial Intelligence Graduate School Program (Korea University). 5%), and Culture, Sports and Tourism R&D Pro- gram through the Korea Creative Content Agency grant funded by the Ministry of Culture, Sports and Tourism in 3024(Interaational Collaborative Research and Global Talent Development for the Development of Copyright Management and Protection Technologies for Generative AI, RS-2024- 00345025, 5%; Research on neural watermark technology for copyright protection of generative Al 3D content, RS-2024-00348466 5%).
References
[1] Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai. Transfusion: Robust lidar-camera fusion for 3d object detection with transformers. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 1090-1099, 2022.
[2] Gyusam Chang, Wonseok Roh, Sujin Jang, Dongwook Lee, Daehyun Ji, Gyeongrok Oh, Jinsun Park, Jinkyu Kim, and Sangpil Kim. Cmda: Cross-modal and domain adversarial adaptation for lidar-based 3d object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 972-980, 2024.
[3] Tingting Liang, Hongwei Xie, Kaicheng Yu, Zhongyu Xia, Zhiwei Lin, Yongtao Wang, Tao Tang, Bing Wang, and Zhi Tang. Bevfusion: A simple and robust lidar-camera fusion framework. 2022.
Advances in Neural Information Processing Systems, 35:
[4] Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang. Huizi Mao, Daniela L. Rus, and Song Han. Bevfusion: Multi-task multi-sensor fusion with unified bird's- s-eye view representation. In 2023 IEEE international conference on robotics and automation (ICRA), pages 2774-2781. IEEE, 2023.
[5] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard. Vijaysai Patnaík, Paul Tsui, James Guo. Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2446-2451 2020.
[6] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 11621-11631, 2020.
[7] John Houston, Guido Zuidhof, Luca Bergamini, Yawei Ye, Long Chen, Ashesh Jain, Sammy Omari, Vladimir Iglovikov, and Peter Ondruska. One thousand and one hours: Self-driving motion prediction dataset. In Conference on Robot Learning, pages 409-418. PMLR, 2021.
[8] Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon. Detr3d: 3d object detection from nulti-view images via 3d-to-2d queries. In Conference on Robot Learning, pages 180-191. PMLR, 2022.
[9] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird's-eye-viee representation from multi-camera images via spatiotemporal transformers. In European conference on computer vision, pages 1-18. Springer, 2022.
[10] Wonseok Roh, Gyusam Chang, Seokha Moon, Giljoo Nam, Chanyoung Kim, Younghyun Kim, Jinkyu Kim, and Sangpil Kim. Ora3d: Overlap region aware multi-view 3d object detection. arXiv preprint arXiv:2207 200.000665 2022.
[11] Junjie Huang, Guan Huang, Zheng Zhu, Yun Ye, and Dalong Du. Bevdet: High-performance multi-camera 3d object detection in bird-eye-view. arXiv preprint arXiv:2112. 2021.

---

[12] Yinhao Li, Zheng Ge, Guanyi Yu, Jinrong Yang, Zengran Wang, Yukang Shi, Jianjian Sun, and Zeming Li. Bevdepth: Acquisition of reliable depth for multi-view 3d object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 1477-1485, 2023.
[13] Jose M Facil, Benjamin Ummenhofer, Huizhong Zhou, Luis Montesano, Thomas Brox, and Javier Civera. Cam-convs: Camera-aware multi-scale convolutions for single-view depth. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11826-11835, 2019.
[14) Shuo Wang. Xinhai Zhao, Hai-Ming Xu, Zehui Chen, Dameng Yu, Jiahao Chang. Zhen Yang, and Feng Zhao. Towards domain generalization for multi-view 3d object detection in bird-eye- view. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13333-13343 2023.
[15] Hao Lu, Yunpeng Zhang, Qing Lian, Dalong Du, and Yingcong Chen. Towards generalizable multi-camera 3d object detection via perspective debiasing, arXiv preprint arXiv:2310.11346 2023.
[16] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09665 2021.
[17] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.0860 2021.
[18] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du Zhilin Yang. and Jie Tang. P-tuming v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint 10X002210..07602, 2021.
[19] Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18(10):3337, 2018
[20] Dennis Park, Rares Ambrus, Vitor Guizilini, Jie Li, and Adrien Gaidon. Is pseudo-lidar needed for monocular 3d object detection? In IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[21] Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin. Probabilistic and Geometric Depth: Detecting objects in perspective. In Conference on Robot Learning (CoRL) 2021, 2021.
[22] Tai Wang. Xinge Zhu, Jiangmiao Pang, and Dahua Lin. Fcos3d: Fully convolutional one-stage monocular 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 913-922, 2021.
[23] Garrick Brazil and Xiaoming Liu. M3d-rpn: Monocular 3d region proposal network for object detection. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9287-9296, 2019.
[24] Zechen Liu, Zizhang Wu, and Roland Tóth. Smoke: Single-stage monocular 3d object detection via keypoint estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 996-997, 2020.
[25] Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1490044999 2018.
[26] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12697-12705, 2019.
[27] Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28 2020, Proceedings. Part XIV 16, pages 194-210. Springer, 2020.
12

---

[28] Yingfei Liu, Tiancai Wang. Xiangyu Zhang, and Jian Sun. Petr: Position embedding transfor- mation for multi-view 3d object detection. European Conference on Computer Vision, pages 531-548. Springer, 2022.
[29] A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
[30] Jinhyung Park, Chenfeng Xu, Shijia Yang, Kurt Keutzer, Kris M Kitani, Masayoshi Tomizuka, and Wei Zhan. Time will tell: New outlooks and a baseline for temporal multi-view 3d object detection. In The Eleventh International Conference on Learning Representations, 2022.
[31] Chenyu Yang, Yuntao Chen, Hao Tian, Chenxin Tao, Xizhou Zhu, Zhaoxiang Zhang, Gao Huang, Hongyang Li, Yu Qiao, Lewei Lu, et al. Bevformer v2: Adapting modern image backbones bird's seeeev-eee recognition via perspective supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17830-17839, 2023.
[32] Yu Hong, Hang Dai, and Yong Ding. Cross-modality knowledge distillation network for monocular 3d object detection. In ECCV, Lecture Notes in Computer Science Springer, 2022.
[33] Peixiang Huang, Li Liu, Renrui Zhang, Song Zhang, Xinli Xu, Baichao Wang, and Guoyi Liu. Tig-bev: Multi-view bev 3d object detection via target inner-geometry learning, arXiv preprint arXiv:2212.139793 2022.
[34] Zehui Chen, Zhenyu Li, Shiquan Zhang, Liangji Fang, Qinhong Jiang, and Feng Zhao. Bevdistill: Cross-modal bev distillation for multi-view 3d object detection. arXiv preprint arXiv:2211.09388 2022.
[35] Sujin Jang, Dae Ung Jo. Sung Ju Hwang, Dongwook Lee, and Dachyun Ji. Stxd: Structural and temporal cross-modal distillation for multi-view 3d object detection. Advances in Neural Information Processing Systems, 36, 2024.
[36] Yan Wang. Xiangyu Chen, Yurong You, Li Erran Li, Bharath Hariharan, Mark Campbell, Kilian Q Weinberger, and Wei-Lun Chao. Train in germany, test in the usa: Making 3d object detectors generalize. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11713 11723, 2020.
[37] Jihan Yang, Shaoshuai Shi, Zhe Wang. Hongsheng Li, and Xiaojuan Qi. St3d: Self-training for unsupervised domain adaptation on 3d object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10368-10378. 2021.
[38] Qiangeng Xu, Yin Zhou, Weiyue Wang. Charles R Qi, and Dragomir Anguelov. Spg: Unsuper- vised domain adaptation for 3d object detection via semantic point generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15456, 2021.
[39] Zeng Yihan, Chunwei Wang, Yunbo Wang, Hang Xu, Chaoqiang Ye, Zhen Yang, and Chao Ma. Learning transferable features for point cloud detection via 3d contrastive co-training. Advances in Neural Information Processing Systems, 34:21493-211 21504,, 2021.
[40] Yi Wei, Zibu Wei, Yongming Rao, Jiaxin Li, Jie Zhou, and Jiwen Lu. Lidar distillation: Bridging the beam-induced domain gap for 3d object detection. In European Conference on Computer Vision, pages 179-195. Springer, 2022.
[41] Yanan Zhang, Chao Zhou, and Di Huang. Stal3d: Unsupervised domain adaptation for 3d object detection via collaborating self-training and adversarial learning. IEEE Transactions on Intelligent Vehicles, 2024.
[42] Qianjiang Hu, Daizong Liu, and Wei Hu. Density-insensitive unsupervised domain adaption on 3d object detection. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 17556-17566, 2023.
[43] Vidit Vidit, Martin Engilberge, and Mathieu Salzmann. Clip the gap: A single domain gen- eralization approach for object detection. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 3219-3229, 2023.
13

---

[44] Aming Wu and Cheng Deng. Single-domain generalized object detection in urban scene via syclic-dieentangled self-distillation. Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 847-856, 2022.
[45] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 16000-16009, 2022.
[46] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[47] Zhenyu Li, Zehui Chen, Ang Li, Liangji Fang, Qinhong Jiang, Xianming Liu, and Junjun Jiang. Unsupervised domain adaptation for monocular 3d object detection via self-training. In European conference on computer vision, pages 245-262. Springer, 2022.
[48] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages
PMLR, 2019.
[49] Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Scaling shifting your features: A new baseline for efficient model tuning. Advances in Neural Information Processing Systems, 35:109-123, 2022.
[50] Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950-19965 1965, 2022.
[51] Yue-Jiang Dong, Yuan-Chen Guo, Ying-Tian Liu, Fang-Lue Zhang, and Song-Hai Zhang. Ppea-depth: Progressive parameter-efficiet adaptation for self-supervised monocular depth estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 1609-1617, 2024.
[52] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In Proceedings of the 1st Annual Conference on Robot Learning, pages 1-16, 2017.
[53] Qiqi Gu, Qianyu Zhou, Minghao Xu, Zhengyang Feng, Guangliang Cheng, Xuequan Lu, Jianping Shi, and Lizhuang Ma. Pit: tion. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 8761-8770, 2021.
invariant transform for cross-fov domain adapta-
[54] Tzofi Klinghoffer. Jonah Philion, Wenzheng Chen, Or Litany, Zan Gojeic, Jungscock Joo, Ramesh Raskar, Sanja Fidler, and Jose M Alvarez. Towards viewpoint robustness in bird's eye view segmentation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 8515-8524, 2023.
[55] Yunhan Zhao, Shu Kong, and Charless Fowlkes. Camera pose matters: Improving depth prediction by mitigating pose distribution bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15759-15700 2021.
[56] Ke Wang, Bin Fang, Jiye Qian, Su Yang, Xin Zhou, and Jie Zhou. Perspective transformation data augmentation for object detection. IEEE Access, 044935-4447 2019.
[57] Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel Brostow. Digging into self-supervised monocular depth estimation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3828-3838, 2019.
[58] Wenjing Bian, Zirui Wang, Kejie Li, Jia- Wang Bian, and Victor Adrian Prisacariu. Nope- nerf: Optimising neural radiance field with no pose prior. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 4160-4169, 2023.
14

---

[59] Xiaoyang Lyu, Liang Liu, Mengmeng Wang. Xin Kong, Lina Liu, Yong Liu, Xinxin Chen, and Yi Yuan. Hr-depth: High resolution self-supervised monocular depth estimation. Proceedings of the AAAI conference artificial intelligence, volume 35, pages 2294-2301, 2021.
[60] Hang Zhou, David Greenwood, and Sarah Taylor. Self-supervised monocular depth estimation with internal feature fusion. arXiv preprint arX:v:2110.09482, 2021.
[61] Yi Wei, Linqing Zhao, Wenzhao Zheng, Zheng Zhu, Yongming Rao, Guan Huang, Jiwen Lu, and Jie Zhou. Surrounddepth: Entangling surrounding views for self-supervised multi-camera depth estimation. arXiv preprint
2022.
[62] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions image processing, 13(4): 600-612 2004.
[63] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[64] Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming Li, and Gang Yu. Class-balanced grouping and sampling for point cloud 3d object detection. 2019.
15

---

Appendices
Table of Contents
Datasets
16
B Implementation Details
C Additional Experiments
17
Broader Impacts.
20
A Datasets
Table 5: Dataset details. Note that each statistical information is calculated from the whole dataset.
Dataset
Cameras
LIDAR
# scenes
# 3D boxes
Points per Beam
Range
Location
Night
Rain
Highway
nuSeenes
32-bcam
1000
1.4M
1,084
< 100m
USA and Singapore
Lyft
64-beam
366
1.3M
1,803
< 100m
Wayme
64mm
1150
12M
2,258
< 100m
USA
CARLA
beam
20M
2,500
100m
We evaluate overall performance on landmark datasets for 3D Object Detection: Waymo [5]. Lyft (7). and nuScenes [6]. The three datasets have different point cloud ranges and specifications. Hence, convert them unified range and coordinates for accurate comparison. We also adopt only seven parameters to achieve consistent training results under the same conditions: center locations estimate practical degradation due
(2r, z). box size (1, w, h), and heading angle 0. Additionally, to changes in camera positioning, we conducted proof of concept by generating data similar to the nuScenes using the CARLA simulation. The details are as follows:
Waymo The Waymo dataset (5] consists of high-quality and large-scale data with 230K frames from all 1,150 scenes using multiple LiDAR scanners and cameras. Furthermore, for the generalization purpose, Waymo is recorded at diverse cities, weather conditions, and times. For object detection in 2D or 3D, Waymo provides point cloud-annotated 3D bounding boxes 3D data pairs and RGB image-annotated 2D bounding boxes as 2D data pairs.
nuScenes The nuScenes dataset [6] uses 6 cameras that cover full 360-degree range of view and a single LiDAR sensor to obtain 40K frames from 20-second-long 1,000 video sequences, which fully annotated with 3D bounding boxes for 10 object classes. The nuScenes dataset covers 28k annotated samples for training. Also, validation and test contain 6k scenes each. The nuScenes frames are captured in the same manner Waymo dataset for the data diversity. But unlike Waymo, nuScenes provides labels only for the point cloud data with 23 classes of 3D bounding boxes.
Lyft The Lyft dataset 7 is motivated by the impact of large-scale datasets on Machine Learning and consists of over 1,000 hours of data. This was collected by a fleet of 20 autonomous vehicles along fixed route in Palo Alto, California, over four-month period. It consists of 170,000 scenes (each scene is 25 seconds long) and contains 3D bounding boxes with the precise positions of nearby vehicles, cyclists, and pedestrians over time. In addition, the Lyft dataset includes high-definition semantic map with 15,242 labelled elements and a high-definition acrial view over the area.
CARLA To quantify the performance drop resulting from camera shifts, we employed an autonomous driving simulation powered by CARLA (52) 0.9.14 and Unreal Engine 4.26. We collected 24K frames for training and 1K frames for each evaluation, driving through Town10 under cloudless weather conditions between sunrise and sunset times. This dataset includes over 100 vehicles and 30 pedestrians in random locations. In Fig. [6] the Source utilizes 6 nuScenes-lin like cameras and 6 LiDARs, while the Target has perturbed sensors. From the Source sensors, the Height increases by 0.65m and the Pitch increases by 5 degrees. The All synthetically moves the x, y, z-coordinates by -0. 12m, 0.65m, and -0.2m/+0.2mm respectively, and rotates the yaw by -5/45 degrees, depending on their directions. Each target sets is collected simultaneously with the Source.
16

---

Table 6: Comparison of Unified Domain Generalization and Adaptation performance with state- of-the-art techniques. We validate our proposed methods with the same baseline model, named BEVDepth, on Cross-domain, The bold values indicate the best performance. Also, denotes 'Do not support'.
Source
Target
Task
Method
Branch
NDS / mAPt
NDS' mAPt
Direct Transfer
0.684 / 0.602 0.296 /
0.213 0.100 0.587 0.475
Oracle
DG-BEV 14 PD-BEV 15 PD-BEV
DG
0.675 / 00611
0.374 0.268
Lyft - nuSeenes
DG
0.677 / 0,593
0.344 0.263
UDA
0.672 /
0.280
Ours
DG
0.702 / 0.630
0.421 0.281
Ours (1%)
UDGA
0.702 0.630
0.526
Ours (5%)
UDGA
0.702 0.630
0.563 0.444
Direct Transfer
0.587 / /0.475
0.296 0.111
Oracle
0,213
0.684 0.602
DG-BEV PD-BEV PD-BEV
DG
0.578 00.47
0.437 0.287
nuScenes -> Lyft
0.458 0.304
UDA
0.476 0.316
Ours
0.623 0.511
0.487
Ours (1%)
UDGA
0.623 0.513
0.578 0.462.
Ours (5%)
UDGA
0.623 0.511
0.613 0.506
Direct Trunsfer
0.649 0.552
0.133 0.032
Oracle
0.178 0.040
0.587 0.475
DG-BEV
DG
0.660 0.568
0.472 0.303
Waymo + nuSeenes
Ours
0.656 /0.547
0.477 0.326
Ours (1%)
UDGA
0.656 0.547
0.534 0,409
Ours (5%)
UDGA
0.656 0.547
0.571 0.448
Direct Transfer
0,587 00.475
0.178
Oracle
0.133 0.032
0.649 0.552
DG-BEV
DG
0.563 / 00.461
0.415 0.297
nuSeenes Waymo
Ours
DG
0.603 / 0.497
0.459 0.349
Ours (1%)
UDGA
0.603 / 0.497
0.509
Ours (5%)
UDGA
0.603 0.497
0.549 0.424
B Implementation Details
To validate the effectiveness of our proposed methods, we adopt BEVDepth and BEVFormer as our base detectors. Both detectors utilize ResNet50 (3] backbone that initialized from ImageNet- IK. Also, we construct BEV representations within a perception range [-50.0m, 50.0m] for both the X and Y axes. In BEVDepth, we reshape multi-view input image resolutions as follow: [256, 704) for nuScenes, [384, 704) for Lyft, [320, 704] for Waymo. As following DG-BEV [4], we train 24 epochs with AdamW optimizer by learning rate 2e-4 in pre-training phase. The training takes approximately 18 hours using one A 100 GPU. fine-tuning phase, we conduct an extensive grid search to determine the optimal learning rate proportional to the number of learnable parameters. Note that we extensively augment various image conditions as detailed in [14]
C Additional Experiments
In this appendix, we present additional experiments to validate the effectiveness of our proposed paradigm. First, Tab. 6 summarizes the overall results of our work from the perspective of domain shift. We also analyze how changes in camera positioning worsen the performance and evaluate whether existing augmentation methods can mitigate the deterioration. Additionally, we conduct ablation studies to enhance the LEDA structure, including comparisons with formal adapters. Finally, we present the comparison results with the transformer-basee detector. The qualitative analysis the multi-view results from proposed paradigm is included towards the end of this chapter.
17

---

Performance across domains. In this section, we compare our proposed UDGA with existing solutions (i.e., DG, UDA) in various cross-domain conditions (see Tab. 6). We aim practically mitigate perspective shifts without hindering well-defined source knowledge. Our DG branch achieves top performance, surpassing Direct Transfer the Source domain. The UDGA, which follows DG, improves Target accuracy without compromising Source performance. Especially, we advocate that UDGA enables efficient adaptation with significantly down-scaled data split (i.e., 1% and 5%).. Also, it is noteworthy that UDGA do not forget previously learned potentials, fully transferring to target domains (up to +1.22% NDS gain in Lyft=rnuScenes). Overall, UDGA provide a practical solution to address perspective view changes, efficiently adapting with only tiny split.
Table 7: Performance under CALRA-simulated domain changes. The model is trained exclusively on Source. The diff shows the Source- Target difference. The bold values indicate the worst difference.
Test domain
NDS't
mAPt
mATE,
mASE)
mAOE]
Source
0.666
0.811
0.229
0,122
0.043
Parget:Pitch
0.449
0.491
0.739
0.159
0.065
diff
-0.217
-0.319
0.510
0,036
0.022
Source
0.688
0.848
0.210
0.111
0.042
Height
0.313
0.280
1.362
0.179
0.090
diff
-0.374
-0,568
1.152
0,067
0.048
Source
0.687
0.847
0.211
0,216
0.372
Target:Al/
0.321
0.301
1.357
0,181
0.110
diff
-0.366
-0,546
1.146
0.069
0.071
Table 8: Performance of multi-view augmentations in domain shift. Gray highlight denotes 'Ours'.
Lyft + nuScenes
nuScenes
Lyft
Method
NDS t
MAP1
NDS*
mAPt
Direct Transfer
0.213
0.102
0.296
0.112
GT sampling
0.269
0.211
0.405
0.263
2D augmentation
0.269
0.221
0.423
0.263
3D augmentation
0.289
0.235
0.403
0,243
Extrinsic augmentation
0.298
0.223
0.436
0,255
CBGS (64)
0.265
0.196
0.349
0.215
DG-BEV
0.374
0.268
0,437
0.287
Ours
0.421
0.281
0.487
0.324
Practical domain shift analysis. We analyze the impact of changes in camera geometry on 3D object estimation. The experimental model is trained using only the source dataset on ResNet50-brr based BEVDet and then evaluated on three sets of (source, target) analyze performance differences. In Tab. 77 the performance of the source is similar in all test sets. On the other hand, the performance of the target decreases significantly in all cases. Since this experiment conducted the same environment with the same camera sensors, it demonstrates how much performance degradation caused by the position of the camera. The set with the largest performance drop in the target is Height, where the mATE value increased significantly. The target All exhibits the worst mASE and mAOE, while the other measures also deteriorated by a similar amount as Height.
Conventional augmentation methods enhance the robustness of the model. We evaluate some of them in Tab, 8 GT sampling and CBGS 664 are techniques designed balance ground truths. 2D augmentation directly augment multi-view inputs (i.e., image resize, crop and paste, contrast and brightness distortion). 3D and extrinsic methods are global augmentations that address both input and ground truths, and ground truths only, respectively. These methods enhance geometric understanding from input noises. However, dynamic view changes (i.e., cross-domain), they still suffer from geometric inconsistency and show poor generalization capability. Moreover, various 2D) approaches do not guarantee geometric alignments between 2D images and 3D ground truths and relevant studies have not been explored well, reported in [14] and [55].
Searching adapter structures. We explore various modules and structures to find a suitable adapter architecture. Tab. show which structures and locations affects the model's 's performance. For adapter locations, performance is optimal when adapters are attached to all modules, gradually
18

---

Table Performance comparison for each module (UDGA 5%). Gray highlight denotes 'Ours'.
Lyft + nuScenes
nuScenes + Lyft
Backbone
View transform
BEV encoder
Detection head
NDS
mAP
NDS
mAP
0.421
0.281
0.487
0.324
0.333
0.237
0,489
0.352
0.433
0.322
0,551
0.418
0.525
0.409
0.608
0.498
0.563
0.444
0.613
0.506
Table 10: Comparison with various adapter structures (UDGA 10%). Gray highlight denotes 'Ours'.
Lyft - nuScenes
nuScenes - Lyft
Method
Project Down
Project Up
# Params
NDS
mAPt
NDS
mAPt
Adapter-H
Conv.
Conv.
25.9M
0.547
0.439
0.592
0.484
Adapter-B
Conv.
Linear
21.3M
0.511
0.384
0.584
0.475
Sdapter-S
Linear
Conv.
8.8M
0.444
0.255
0,500
0.356
T
Linear
Linear
2.9M
0,282
0.262
0.398
0.376
Ours
Conv.
Linear
8.8M
0.573
0.457
0.638
0.537
improving with the addition of more. Exceptionally, attaching adapters only at the Detection Head leads to decline in LyfteenuSee In addition, Tab. [0] represenns the performance various adapter structures. The combination of Convolution and Linear layer respectively for Project Down and Up shows the best performance in both tasks. Note that training with fewer parameters(8.8M)) is more effective. However, suggest that large-scale parameters may require larger dataset more training, as we only trained on 10% of the target dataset for less than 20 epochs in this experiment.
Table 11: Comparison of UDGA performance on BEVFormer. We train with two different data splits 50%, and 100%. Additionally, Params denote the number of parameters for training. The bold values indicate the best performance. denotes 'Do not support'.
Task
Method
Params
NDS*
mAPt
Oracle
33.5M
0.635
0.534
Direct Transfer
33.5M
0.338
0.245
nuScenes Lyft
Full FT
33.5M
0.638
0.533
Ours (50%)
12.2M
0.596
0.477
Ours (100%)
12.2M
0.638
0.534
Comparison of UDGA on BEVFormer. To demonstrate the validation of UDGA, we further com- pare performance BEVFormer-snee (33.5M parameters) with Full FT. For accurate comparison, we provide Oracle, and Direct Transfer in nuScenes > Lyft task.
BEVFormer adopt Query-based view transformation modules V as follow Eq. 10
VLI, K,,KKT) Cross Prys, k F24):
(10)
where q, l and v represents query. value and key in Transformer, and then Pays denotes pre-defined anchor BEV positions by and T. Here, Query-based module benefits from Cross Alln with sparse query sets, implicitly learning geometric information. Thus, we reconstruct our UDGA paradigm without explicit depth constraints. First, we adopt linear-based bottleneck structures with Layer Normalization in Eq. II Oup and Φdown denote the projection up and down layer.
y ff(()) + &up (a (a(ooon (I)N()))))
(11)
where ffr denotes feed-forward networks, and I.N represents Layer Normalization. We conduct experiments by plugging these extra modules, which accounts for 36% of the total parameters, into BEVFormer. As a result, we achieve significant adaptation performance with the 50% data split. Notably, we demonstrate effectiveness, achieving parity with Full the 100% data split.
Additional qualitative analysis. In this section, we further visualize our depth quality various scenarios (i.c., Lyft, and nuScenes). Not only our overlap depth constraint significantly improve depth
19

---

Plcoett Left
Front
Froot Right
Back Right
Beck
Back Left
Soume
Height
Soumcs
Pich
Source
All
Figure 6: The paired sample of each evaluation set in Carla dataset.
consistency in occluded regions, but also show better spatial understanding for hard samples (e.g., far and low distinguishable objects). Additionally, we note that our method effectively complement insufficient contextual recognition caused by sparse depth gt in Fig. (b). Overall, we stably deploy Multi-View 3DOD by leveraging effective association between adjacent views.
D Broader Impacts.
Our framework is practical Al algorithm that enhances its generalization ability to handle domain changes robustly, enabling us to effectively reduce data costs and computing resources required for adaptation. Practically, our method makes it suitable for deployment in mass-produced vehicles, where the algorithm can inherit the knowledge of well-trained pretrained weights while self-learning to adapt each fleet environment. The adaptation learning process is also simplified, making casier to transfer improved pretrained networks. Furthermore. by demonstrating superior performance compared to previous methods that relied on LiDAR for auxiliary depth networks, approach reduces the dependency on lidar modality. This suggests the feasibility of excluding expensive LiDAR sensors from future autonomous vehicles.
20

---

Frand Leil
Foodl
Frani Right
Bark Right
Back
Back Lell
BEVDeptt
Outs
(a) Lyft
BEVDaH
Uhoss
(b) muSoones
Figure 7: Multi-view visualization of the depth estimation of BEVDepth and Ours for (a)Lyft and (b)nuScenes samples. In general, depth consistency was better in the Lyft dataset, while was difficult to make quantitative comparison in the case of nuScenes due to the sparseness of the LiDAR point clouds. The depth range is from Im 60m. Best viewed in color.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have summarized the contributions of our work at the end of Sec. (Introduction) as well as in the abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed and presented limitations in Sec. (Conclusion).
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided detailed information about the implementations in Sec. 4 (Experiment) and in Appendix A B
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

reproducible, they Answer: [No]
Justification: We plan to release source codes upon acceptance. At current phase, we have provided details of implementations and necessary references to prior works in Sec. (Experiment) and in Appendix A B
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.co cc/
public/guides htte///odduumsssioono...
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 7/nips. //nips.cc/pub. http:ss.ccppbblc/guuueeee Sttp:////soooSSeeesseeieeiia
ssioiiolioiyiyiic for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We have provided detailed information about the implementations in Sec. (Experiment) and in Appendix A B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have provided details of computing resources in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///aaappp.cooo//////cccccaa
Answer: [Yes]
Justification:
We conform and follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification: have discussed both impacts in Sec. 5 (Cnnlluiion..
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We discussed the potential risks of the proposed method, which we judge to pose no notable risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have provided credits for the benchmark dataset and cited the baseline models.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
26

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "K2qGDxsv06", "reviewer": "Reviewer_pgwk", "summary": "This paper focuses on multi-view 3D object detection. The authors proposed a unified domain generalization and adaptation-based detection method. To enhance the detection model for unseen datasets and address the geometric misalignment problem, the authors proposed a multi-view overlap depth constraint module and a label-efficient domain adaptation technique. The comprehensive experiments on lar-scale datasets, including nuScenes, Lyfy and Waymo have demonstrate the effectiveness of the proposed method.", "questions": "This paper seems well-shaped to me. However, I'm not an expert in this field, so I'm open to discussion with other reviewers if they hold opponent opinions to me.", "limitations": "Please refer to the above.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The storyline is clear, the authors provided detailed motivation analysis in the introduction and present their innovation clearly.\n- The performance is strong. The method shows SOTA performance in variance benchmarks with fast speed.\n- Most of the figures are clear. This paper is also well-written.", "weaknesses": "- There is no source code for review.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Code release**\nWe plan to release the source code upon acceptance. \n\nAdditionally, in this rebuttal, we provide further explanations, analyses, and results (please refer to **global response** and **Rebuttal PDF**). We hope that these materials will be helpful to you.\n\n*If you have any questions or need further discussion, please feel free to ask.*\n\n*Thank you.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IFRmHP7LdE", "reviewer": "Reviewer_LaL5", "summary": "The paper presents an adaptation of 3D object detectors to varying target environments using two major strategies. The proposed multi-view overlap depth constraint leverages associations across views to learn view-invariant features. Additionally, a LORA-like structure is designed for parameter-efficient adaptation, accommodating scenarios with limited target data. Experiments on three benchmark datasets demonstrate the effectiveness of the proposed approach, with minimal modification of parameters.", "questions": "Please refer to the weakness section.", "limitations": "The authors have not provided a discussion on the limitations of their work.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ The paper addresses a significant and practical issue in 3D object detection, aiding in the development of robust models for dynamically changing testing environments.\n+ The flowchart clearly illustrates the core components, making it easy for readers to grasp the main idea.\n+ Extensive experiments have been conducted, and the proposed strategy significantly outperforms the pre-trained source model and even surpasses full fine-tuning strategies.", "weaknesses": "- The two proposed strategies appear somewhat disconnected. While the multi-view idea is interesting, the technical details are confusing. Equation (5) includes three different objectives, but their importance or sensitivity is not discussed. The latter adaptation strategy resembles existing works and lacks specific design for the 3D detection task. This is also not correlated with the view-transform. The isolated adaptation modules seem more like a stacking of existing techniques rather than a cohesive addition.\n- More discussion and comparisons with previous multi-view augmentation strategies would help clarify the merits and innovations of the proposed approach. The current claim that existing strategies are poorly generalizable is somewhat unconvincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **(1) More details of LEDA.**\n\n**Disconnected strategies**  \nWe hope that further explanations of LEDA in global response will enhance your understanding. We empirically observe that direct fine-tuning approaches (with a small fraction of data) often fail to align between source and target, mainly due to dynamic perspective shifts. To mitigate this issue, our proposed depth constraint method softly addresses perspective gaps and enhances adaptation capacity to novel targets. **In Rebuttal PDF Table 1, LEDA (w/o $\\mathcal{L_{ov}}$ and $\\mathcal{L_{p}}$) struggles to bridge with novel target knowledge and show only 20.4% / 24.4% NDS / mAP gain. However, LEDA (w/ $ \\mathcal{L_{ov}} $ and $ \\mathcal{L_{p}} $) significantly transfer pre-trained knowledge to novel targets (34.2% / 42.5% NDS / mAP gain). As a result, we note that $ \\mathcal{L_{ov}} $ and $ \\mathcal{L_{p}} $ enable efficient learning by encouraging stable convergence even with tiny datasets.**\n\n**Ablation of objectives**  \nAdditionally, we study the importance of our Optimization Objectives ($\\mathcal{L_{ov}} $ and $ \\mathcal{L_{p}} $). Although, our depth constraint method effectively addresses domain gaps, it suffer from narrow overlap regions between adjacent views. To tackle this drawback, $\\mathcal{L_{p}}$ effectively boosts stable recognition during both pre-training and fine-tuning phase. Specifically,  $\\mathcal{L_{p}}$ yields max 1.9% / 1.9% NDS / mAP gain during pre-training and max 2.1% / 1.6% NDS / mAP gain during fine-tuning. As a result, we demonstrate that our proposed two methods synergistically enhance UDGA performance. \n\n**In-depth explanation of LEDA.**  \nIn this paper, we advocate that depth consistency play a pivotal role to bridge domain gaps. Especially, we show that UDGA stably releases geometric differences from perspective view changes and encourages optimal BEV representation learning as shown in Rebuttal PDF Table 3. **Precisely, UDGA achieves remarkable performance improvements of up to 10.0% in NDS and 9.5% in mAP in view transformation and BEV encoder layers, demonstrating its effectiveness.**\n\n### **(2) Comparison with Augmentation strategies.**\n\nWe also measured generalizability of previous multi-view augmentation strategies as shown in **Rebuttal PDF Tab.4**. We adopt conventional augmentation strategies to multi-view 3D object detection as follows:\n- GT sampling effectively address unbalanced labels by sampling ground truth objects in the 3D object detection.\n- 2D aug. directly augment multi-view inputs (i.e., image resize, crop and paste, contrast and brightness distortion).\n- 3D aug. globally rotate, re-scale and translate multi-view inputs and ground truths.\n- Extrinsic aug. also adopts global yaw rotation towards the random direction\n- CBGS re-balances to solve unbalanced ground truths.\n\nThese methods significantly enhance geometric understanding from input noises. However, in dynamic view changes (i.e., cross-domain), they still suffer from geometric inconsistency and show poor generalization capability. Moreover, various 2D approaches do not guarantee geometric alignments between 2D images and 3D ground truths and relevant studies have not been explored well, as reported in DG-BEV and [1]. To tackle these issues, we present a novel Multi-view Overlap Depth Constraint that effectively mitigate dynamic view changes in the cross domain. We also applied top 3 methods 2D, 3D, and extrinsic augmentation to our method and achieved state-of-the-art performance on both source and target. Unfortunately, we were unable to find any additional comparable papers for generalized multi-view 3D object detection. If you could provide us with relevant papers, we would be happy to analyze them together.\n\n*We sincerely appreciate your thoughtful review and will address above discussion in the revised version.*   \n\n*Thank you.* \n\n\n----\n[1] Zhao, Yunhan, Shu Kong, and Charless Fowlkes. \"Camera pose matters: Improving depth prediction by mitigating pose distribution bias.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.  \n[2] Zheng, Liangtao, et al. \"Cross-Dataset Sensor Alignment: Making Visual 3D Object Detector Generalizable.\" Conference on Robot Learning. PMLR, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8WR49oCa0f", "reviewer": "Reviewer_gahV", "summary": "This paper is about Unified Domain Generalization and Adaptation for outdoor 3D object detection. To address the geometric misalignment between the source and target domains, Multi-view Overlap Depth Constraint that leveraging the strong association between multi-views and Label-Efficient Domain Adaptation are proposed. Experiments are done with several cross-dataset settings.", "questions": "The experimental comparison and the novelty, importance of the work.", "limitations": "The limitation part has been addressed.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The domain generalization and adaption is important for 3D object detection.\n2. The writing is easy to follow.", "weaknesses": "1.\tIn the past works, the comparative methods, such as DG-BEV, and PD-BEV, reported the performance on both source and target domains. Yet it seems that the reported metrics in Table 1 do not align with the reported results in the two papers. What is the reason for such changes?\n2.\tIt is noteworthy that the NDS is specific for nuScenes and different for other datasets. It is important to discriminate such differences.\n3.\tThe authors claim that this work is toward 3D universal detection, yet there are other works toward 3D universal detection, such as [1][2]. It is better to illustrate the differences over these methods, both in discussion and experimental comparison.  Besides, there are other works devoted to multi-dataset training for 3D detection for generalization, also missed in the paper. Can the proposed method be applied in multi-source training and multi-target testing?\n4.\tThe Domain Adapter design seems simple, yet the motivation and potential of such design is missed. It is important since it  is  one of the two components of this work.\n[1] Towards universal LiDAR-based 3D object detection by multi-domain knowledge transfer. ICCV 2023.\n[2] Cross-Dataset Sensor Alignment: Making Visual 3D Object Detector Generalizable. PMLR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **(1) Explanation of Table 1**\nThank you for your insightful review. We understand that there was some confusion regarding the misaligned results in Table 1 due to insufficient explanation. Since previous methods did not adopt consistent experimental methods, a fair comparison is only possible on the target domain (i.e., **they did not evaluate the same model on both the source and target domains**). To address this issue, we adopt their highest scores for comparison, as detailed in Appendix Table 5. \nWe will make the necessary revisions to clarify this information and enhance the overall clarity of our manuscript.\n\n\n### **(2) Discussion with other metrics**\n\n*Note that, to ensure fair comparison, we adopt unified evaluation metric NDS as reported in DG-BEV and PD-BEV.*  \nWaymo and Lyft calculate 3D AP by directly measuring the intersection over union (IoU) between predicted and ground truth bounding boxes. In contrast, NDS measures differences between predicted and ground truth objects leveraging various indicators such as translation error, size error, orientation error, and BEV AP. **This approach enables capturing plausible failure cases (e.g., translation fails or orientation fails) and allows for practical analysis of domain shift issues, as shown in main paper Sec. 3.2 L#126-130, Sec. 4.2 L#207-209, Sec. 4.3 L#244-245 and L#250-251, and Appendix C L#505-513**.  \n\n### **(3) Discussion with the 3D universal detection**\n\nThank you for your meaningful discussion. [1] and [2] explore multi-domain learning and generalization to improve 3D object detection. Here, we present a brief overview as follows.\n\n|   Method   | Modality |  View  | Target-aware | Approach | Multi-source | Multi-target|\n|------|----------|:------:|:------------:|------------|:------------:|:--------:|\n| [1]  | LiDAR    |  multi |       X      | out-domain | O | O |\n| [2]  | Camera   | single |       O      | out-domain | O | O |\n| Ours | Camera   |  multi |       X      | in-domain  | O (a few extra targets) | O |\n\nThe Universal 3D Object Detection task is quite similar to our framework, and our proposed methods showcase considerable potential in this field (multi-source training and multi-target testing), as shown in main paper table 2. \nHowever, while universal training methods generalize external domain knowledge, we leverage internal knowledge, which do not rely on large-scale and diverse datasets.\nFurthermore, our framework can generalize without prior access to the target, making it more feasible to develop and deploy 3D detection in real-world scenarios.\n\n### **(4) More details of Label-Efficient Domain Adaptation**\nWe hope that this rebuttal will be helpful to you. We provide more details of LEDA in the global response, improving the motivation and potential of our proposed method. Although existing methods aim to generalize the domain shift between the source and the novel target, they often fail to provide a practical solution mainly due to unsatisfied generalization performance and show room for improvement. Additionally, costly resources exacerbate these issues and hinder the expansion of multi-view 3D object detection. To tackle these issues, we design a de-facto framework leveraging a efficient and effective learning strategy LEDA for multi-view 3D object detection. \n\nFirst, we show that our depth constraint method smoothly deals with sensor misalignment between source and target domains and effectively boost the adaptation capability of LEDA (as shown in Rebuttal PDF Table 1).  Also, our LEDA successfully transfers their own potentials to novel targets without forgetting previously learned knowledge as shown in main text Table 5. **As a result, we note that our proposed framework impressively releases costly resource issues and unstable performance as shown in Rebuttal PDF Figure 1 and presents practical solutions for real-world autonomous driving**.  \n\n*We sincerely appreciate your constructive review and will address the points discussed above in the revised manuscript.*  \n\n*Thank you.* \n\n--------\n[1] Wu, Guile, et al. \"Towards universal LiDAR-based 3D object detection by multi-domain knowledge transfer.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.  \n[2] Zheng, Liangtao, et al. \"Cross-Dataset Sensor Alignment: Making Visual 3D Object Detector Generalizable.\" Conference on Robot Learning. PMLR, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XeIeACHyuj", "reviewer": "Reviewer_2JqY", "summary": "The paper proposes an Unified Domain Generalization and Adaptation(UDGA) scheme for multi-view 3D object detection. The main components of the proposed method are (1) depth inconsistency-based constraints from multi-view and (2) an efficient domain adaptation scheme (LEDA). The multi-view depth inconsistency constraints improve the domain gap caused by geometric gaps when it comes to perspective view changes. LEDA offers adaptation with fewer amounts of labels while preserving the knowledge from the source domain.", "questions": "I have a few questions that correspond to the weakness that I mentioned.\n\n(1) Is there a justifiable reason why using Depth inconsistency constraint is superior to using deep RGB feature-based inconsistency ( as in DETR3D)?\n\n(2) As one of the main proposed contributions claimed, a detailed analysis of LEDA seems missing. How much does LEDA improve the proposed system compared to existing approaches in terms of efficiency and accuracy?\n\nAddressing those two questions properly will improve the quality of the paper and, subsequently, the reviewer's rating.", "limitations": "The limitations are addressed", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper clearly addresses which problem it focuses on, and each proposed module has solid goals to address existing limitations. The experimental results from multiple datasets also seem convincing.", "weaknesses": "I have a few concerns about the demonstration of the proposed method as well as addressing existing works.\n\n(1) The method part needs to be improved. For instance, specifying every input and output tensor dimension (i.e., depth estimation network, etc.) would significantly help readers understand what each module is doing more quickly. \n\n(2) Using the depth inconsistency seems similar to DETR3D's[1] inconsistency constraint on RGB features. In DETR3D, it is even mentioned that using the RGB feature is better than having explicit depth estimation.\nBased on this, the depth estimation network can introduce additional parameters for optimization while not being very helpful. I cannot see experiments or comparisons with DETR3D. \n\n(3) More related works need to be addressed. Currently, the paper focuses on multi-view-based domain generalization for 3D detection. However, there is also another active line of research with LiDAR-based domain adaptation for 3D detection, pioneered by ST3D[2]. Clearly addressing the difference between those researches will make the reader recognize the line of research that the paper focuses on. \n\n(4) The effectiveness of LEDA is not well demonstrated, as claimed in the paper, compared to existing approaches. \n\n[1] Detr3d: 3d object detection from multi-view images via 3d-to-2d queries, Wang, Yue, et al. CORL, 2022\n\n[2] St3d: Self-training for unsupervised domain adaptation on 3d object detection, Yang, Jihan. et al., CVPR 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **(1) Discussion with RGB feature-based methods**\n\nThank you for your constructive suggestion. As reported by DETR3D [4], CVT [5] and BEVFormer [6], RGB feature-based methods are considerably robust against calibration noise. **However, these methodologies experience significant performance degradation in dynamic view shift environments, such as cross-domain scenarios where calibration changes dramatically, as shown in table below (Lyft $ \\rightarrow $ nuScenes)**.\n\n| Method    | Backbone    | Neck           | Source(NDS*/mAP) | Target(NDS*/mAP) | # Params(M) |\n|-----------|-----------|----------------|:-----------------:|:-----------------:|:------------:|\n| BEVDepth  | ResNet50  |LSS            |   0.684 / 0.602   |   0.213 / 0.102   |     51.7     |\n| CVT       | ResNet50  |Transformer    |   0.658 / 0.563   |   0.231 / 0.066   |     51.2     |\n| DETR3D    | ResNet50  |Transformer    |   0.650 / 0.552   |   0.179 / 0.087   |     32.3     |\n| BEVFormer | ResNet50  |ST Transformer |   0.624 / 0.506   |   0.138 / 0.008   |     33.5     |\n| **Ours**  | **ResNet50**  |**LSS**        | **0.702 / 0.630** | **0.421 / 0.281** |   **51.7**   |\n\nAdditionally, we explore **how domain shift hinders stable 3D recognition in main text Sec. 3.2, Appendix C and Table 7**. Although, fair comparisons are challenging due to architectural differences, we validate that our proposed modules quantitatively mitigate dramatic view changes in the cross-domain (up to 20.9% NDS gain). \n\n### **(2) In-depth explanation of LEDA**\n \nWe hope that additional explanation of LEDA in global response will help you understand better. To improve the development of 3D object detection, we design Label-Efficient Domain Adaptation (LEDA) framework that successfully adapts to novel targets by only a few data and extra parameters, introducing a practical solution for real-world Autonomous Driving. To this end, we conduct structural experiments (Rebuttal PDF Table 2 and 3) to explore the optimal architecture that is both effective and efficient. Also, we validate that our depth constraint method encourages transfer learning from pre-trained source potentials to novel targets as shown in Rebuttal PDF Table 1.  **Especially, by comparing our methodology with existing state-of-the-art models and evaluating NDS by efficiency across various setups (Rebuttal PDF Figure 1), we quantitatively showcase that our proposed methods successfully bridge the gap between the source and novel target using a limited amount of training budget (i.e., training parameters and data). It is noteworthy that our proposed methodology outperforms existing works by 1% on novel targets with only less than 20% extra parameters.** Finally, we will obviously present and analyze in the revised version.\n\n### **(3) Status of LiDAR-based cross-domain object detection**\n\n\nLiDAR object detection research includes addressing performance degradation caused by domain shifts. Wang et al.[1] proposed statistical normalization to mitigate the differences in object size distribution across various datasets. ST3D[2] leveraged domain knowledge through random object scale augmentation, and their self-training pipeline refined the pseudo-labels. STAL3D[3] extended ST3D by incorporating adversarial learning. These methods were all proposed under the assumption of target-aware conditions, which presents the limitation that they cannot be applied without prior access to the target domain.\nThe table (Waymo $\\rightarrow$ KITTI) below extracts a subset of experimental results from STAL3D. LiDAR-based methods were less affected by domain shifts compared to those of cameras, though their performance did not reach the oracle level.\n| Method          | BEV AP | 3D AP |\n|-----------------|--------|-------|\n| _Oracle_          | 83.29  | 73.45 |\n| _Direct Transfer_ | 67.64  | 27.48 |\n| SN[1]           | 78.96  | 59.20 |\n| ST3D[2]         | 82.19  | 61.83 |\n| STAL3D[3]       | 82.26  | 69.78 |\n\nUnlike LiDAR modalities, these approaches fail to address the domain shift problem due to the poor quality of pseudo-labels resulting from the inaccurate geometric information in image-based 3D detection. To address the domain generalization for camera-based systems, we propose a depth constraint module and introduce the LEDA module to achieve oracle-level performance with label efficiency in the target domain.\n\n\n*Plus, we will clearly address the weaknesses (1) More details of our methods and (3) More related works in the revised version.*     \n\n*Thank you.*  \n\n\n------\n[1] Wang, Chen, et al. Train in Germany, Test in The USA: Making 3D Object Detectors Generalize, Computer Vision and Pattern Recognition, 2020.  \n[2] Yang, Shi, et al. ST3D: Self-training for Unsupervised Domain Adaptation on 3D Object Detection, Computer Vision and Pattern Recognition, 2021.  \n[3] Zhang, Zhou, STAL3D: Unsupervised Domain Adaptation for 3D Object Detection via Collaborating Self-Training and Adversarial Learning, IEEE TRANSACTIONS ON INTELLIGENT VEHICLES, 2024.\n[4] Wang, Yue, et al. \"Detr3d: 3d object detection from multi-view images via 3d-to-2d queries.\" Conference on Robot Learning. PMLR, 2022.  \n[5] Zhou, Brady, and Philipp Krähenbühl. \"Cross-view transformers for real-time map-view semantic segmentation.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.  \n[6] Li, Zhiqi, et al. \"Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers.\" European conference on computer vision. Cham: Springer Nature Switzerland, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate your effort to review our work. We have carefully read and considered all of the comments and suggestions provided by the reviewers. To assist with your understanding, we provide detailed analyses and additional experiments of Label-Efficient Domain Adaptation(LEDA) in the global response. Furthermore, we have addressed each reviewer's comments accordingly, and have provided detailed responses and references for each comment. We hope that this rebuttal will be helpful in clarifying any misunderstandings or concerns and will contribute to the overall evaluation of our work.

Thank you again for your time and consideration. We look forward to hearing your feedback.

----

### **The motivation of LEDA**
There exist practical challenges in developing and deploying multi-view 3D object detectors for safety-critical self-driving vehicles. Each vehicle and each sensor requires its own model that can operate in dynamic weather, location, and time conditions. Furthermore, collecting large-scale labels in diverse environments is extremely expensive and inefficient. Among those, we are particularly motivated to address the following issues:
1. **Stable performance**
2. **Efficiency of training**
3. **Preventing catastrophic forgetting**
4. **Minimizing labeling cost**

To satisfy these practical requirements, we carefully design an efficient and effective learning strategy, Label-Efficient Domain Adaptation (LEDA).

In **Rebuttal PDF Figure 1**, we evaluate the efficiency of LEDA compared to existing methods, particularly in terms of domain adaptation (DA) performance. LEDA achieves the highest accuracy with low parameters and data cost, demonstrating its practicality and effectiveness in real-world applications.


### **Technical details of LEDA**

Label-Efficient Domain Adaptation is a novel strategy to seamlessly bridge domains gaps leveraging a small amount of target data. 
To this end, we add extra parameters $\mathcal{A}$ consisting of bottleneck structures (i.e., projection down $\phi_{down}$ and up $\phi_{up}$ layers).
\begin{equation}
    \mathcal{A}(x) = \phi_{up}(\sigma(\phi_{down}(BN(x)))),
\end{equation}
where $\sigma$ and $BN$ indicates activation function and batch normalization. 
We parallelly build $\mathcal{A}$ alongside pre-trained operation blocks $\mathcal{B}$ (e.g., convolution, and linear block) in main paper Figure 3 (ii) and below Equation;
\begin{equation}
    y = \mathcal{B}(x) + \mathcal{A}(x),
\end{equation}
Firstly, we feed $x$ into $\phi_{down}$ to compress its shape to $[H/r, W/r]$, where $r$ is the rescale ratio, and then utilize $\phi_{up}$ to restore it to $[H, W]$. 
Secondly, we fuse each outputs from $\mathcal{B}$, and Adapter by exploiting skip-connections that directly link between the downsampling and upsampling paths. By doing so, these extensible modules allow to capture high-resolution spatial details while reducing network and computational complexity. Plus, it notes worthy that they are initialized by a near-identity function to preserve previously updated weights. Finally, our LEDA lead to stable recognition in both source and target domains, incrementally adapting without forgetting pre-trained knowledge.

To analyze the impact of different architectural choices on the performance of our model, we conduct two ablation studies on the structure of adapters.
- In **Rebuttal PDF Table 2**, we compare variations of the projection up and down layers. 
- In **Rebuttal PDF Table 3**, we compare different locations where the adapter can be attached. 

This allows us to understand how the structure of the adapter affects the model's performance.


### **Optimization Objective** 
We optimize our proposed framework UDGA using the total loss function $\mathcal{L}_{total}$ during both phases (i.e., generalization and adaptation). 

$$\mathcal{L_{total}} = \mathcal{\lambda_{det}L_{det}} +\mathcal{\lambda_{ov}L_{ov}} +\mathcal{\lambda_{p}L_{p}}$$
where we grid-search $\lambda_{det}$, $\lambda_{ov}$ and $\lambda_{p}$ to harmonize $\mathcal{L_{det}}$, $\mathcal{L_{ov}}$ and $\mathcal{L_{p}}$. Specifically, $\mathcal{L}_{total}$ supervises $\mathcal{B}$ during pre-training and $\mathcal{A}$ during fine-tuning, respectively. In **Rebuttal PDF Table 1**, we highlight the importance and the sensitivity of each objective and demonstrate the validity and the connection of both methods. 

Additionally, we present more comparison with existing multi-view augmentation in **Rebuttal PDF Table 4**. Among 5 dominant methods, we applied the top 3 methods, 2D, 3D, and extrinsic augmentation, to our method.

### **Limitations**
There are two limitations of our proposed UDGA for multi-view 3D object detection task. These limitations include: 
1. Our proposed method for calculating depth transformation between multi-view images relies on the presence of overlapping regions between images. The width of the overlap region impacts the accuracy of depth estimation, as demonstrated in Appendix C, Fig.6. 
2. It is structurally challenging to apply depth constraint techniques to query-based networks. This is because cross-attention networks, including BEVFormer and DETR3D, often project from 2D to 3D without passing through a depth net, making it difficult to obtain depth features and calculate correlations.

Meta-review: This paper initially received mixed scores, including two borderline rejects, one borderline accept, and one accept. Following a rebuttal by the authors, Reviewer LaL5 revised their score from borderline reject to borderline accept. While most concerns raised have been addressed, two issues remain unresolved: 1) the proposed multi-view approach's specific applicability to the Domain Generalization (DG) problem, and 2) inaccuracies in certain statements.

The first concern was raised by Reviewer LaL5. Despite this, the impressive performance improvements demonstrated by the proposed method lend credibility to its contribution to the field, and the AC finds this contribution acceptable.

The second concern involves incorrect statements highlighted by Reviewer 2JqY, including the assertion that only LiDAR-based methods require prior access to the target and that only the proposed method is susceptible to challenges posed by weather changes or sensor deployment. The AC suggests that these statements be revised in the final manuscript, noting that these revisions will not detract from the paper's primary contributions, such as its novelty and high performance.

Considering the above factors, the AC believes the paper possesses merits for the field of multi-view 3D object detection and thus suggests borderline acceptance. The AC strongly recommends that the authors address the specific issues raised by Reviewer 2JqY in the final version of the paper.
Decision: Accept (poster)