Title: Unified Domain Generalization and Adaptation for Multi-View 3D Object Detection
Venue: NeurIPS 2024
Accepted: yes
Keywords: Domain Generalization.+Domain Adaptation.+Multi-view 3D Object Detection.+Autonomous driving.+Domain Generalization.
Abstract: Recent advances in 3D object detection leveraging multi-view cameras have demonstrated their practical and economical value in various challenging vision tasks.
However, typical supervised learning approaches face challenges in achieving satisfactory adaptation toward unseen and unlabeled target datasets (i.e., direct transfer) due to the inevitable geometric misalignment between the source and target domains.
In practice, we also encounter constraints on resources for training models and collecting annotations for the successful deployment of 3D object detectors.
In this paper, we propose Unified Domain Generalization and Adaptation (UDGA), a practical solution to mitigate those drawbacks.
We first propose Multi-view Overlap Depth Constraint that leverages the strong association between multi-view, significantly alleviating geometric gaps due to perspective view changes.
Then, we present a Label-Efficient Domain Adaptation approach to handle unfamiliar targets with significantly fewer amounts of labels (i.e., 1$\%$ and 5$\%)$, while preserving well-defined source knowledge for training efficiency.
Overall, UDGA framework enables stable detection performance in both source and target domains, effectively bridging inevitable domain gaps, while demanding fewer annotations.
We demonstrate the robustness of UDGA with large-scale benchmarks: nuScenes, Lyft, and Waymo, where our framework outperforms the current state-of-the-art methods.

Unified Domain Generalization and Adaptation for Multi-View 3D Object Detection

Abstract

Recent advances in 3D object detection leveraging multi-view cameras have demonstrated their practical and economical value in various challenging vision tasks. However, typical supervised learning approaches face challenges in achieving satisfactory adaptation toward unseen and unlabeled target datasets ( i.e. , direct transfer) due to the inevitable geometric misalignment between the source and target domains. In practice, we also encounter constraints on resources for training models and collecting annotations for the successful deployment of 3D object detectors. In this paper, we propose Unified Domain Generalization and Adaptation (UDGA), a practical solution to mitigate those drawbacks. We first propose Multi-view Overlap Depth Constraint that leverages the strong association between multi-view, significantly alleviating geometric gaps due to perspective view changes. Then, we present a Label-Efficient Domain Adaptation approach to handle unfamiliar targets with significantly fewer amounts of labels ( i.e. , 1 % and 5 %) , while preserving well-defined source knowledge for training efficiency. Overall, UDGA framework enables stable detection performance in both source and target domains, effectively bridging inevitable domain gaps, while demanding fewer annotations. We demonstrate the robustness of UDGA with large-scale benchmarks: nuScenes, Lyft, and Waymo, where our framework outperforms the current state-of-the-art methods.

1 Introduction

3D Object Detection (3DOD) is a pivotal computer vision task in various real-world applications such as autonomous driving and robotics. Recent progress in 3DOD [1-4] have showcased remarkable advancements, primarily due to the large-scale benchmark datasets [5-7] and the introduction of multiple computer vision sensors ( e.g. , LiDAR, multi-view cameras, and RADAR). Among these, camera-based multi-view 3DOD [8-12] has drawn significant attention for its cost-efficiency and rich semantic information. However, a significant challenge remains largely unexplored: accurately detecting the location and category of objects in the presence of distributional shifts between the source and target domains ( i.e. , data distributional gaps between the training and the testing datasets).

To successfully develop and deploy Multi-view 3DOD models, we need to solve two practical problems: (1) the geometric distributional shift across different sensor configurations, and (2) the limited amount of resources ( e.g. , insufficient computing resources, expensive data annotations). The first problem poses a challenge in learning transferable knowledge for robust generalization in novel

1

0.44

0.45

0.44

0.52

0.51

0.52

0.56

0.49

0.63

Oracle

DT

CAM-

Convs

DG-BEV

PD-BEV

PD-BEV

(UDA)

Ours

(DG)

DT

+Full FT

Ours

(UDGA)

Source(Lyft)

Target(nuScenes)

Average

Target 5%

Domain

Adaptation

domains. The second issue inevitably requires efficient utilization of computing resources for training and inference, as well as label-efficient development of 3DOD models in practice. To tackle these practical problems, we introduce a U nified D omain G eneralization and A daptation (UDGA) strategy, which addresses a series of domain shift problems ( i.e. , learning domain generalizable features significantly improves the quality of parameter- and label-efficient few-shot domain adaptation).

Prior studies aim to learn domain-agnostic knowledge alleviating domain shifts from drastic view changes in cross-domain environments. DG-BEV [14] disentangles the camera intrinsic parameters and trains the network with a domain discriminator for view-invariant feature learning. Similarly, PD-BEV [15] renders implicit foreground volumes and suppresses the perspective bias leveraging semantic supervision. However, these approaches struggle to capture optimal representations, highlighting that there is still room for improvements in novel target domains ( i.e. , up to -50.8 % Closed Gap compared to Oracle). To tackle these drawbacks, we first advocate a Multi-view Overlap Depth Constraint that leverages occluded regions between adjacent views, which serve as notable triangular clues to guarantee geometric consistency. This approach effectively addresses perspective differences between cross-domain environments by directly penalising the corresponding depth between adjacent views, and shows considerable generalization capacity (up to +75.8 % Closed Gap compared to DT).

Nevertheless, the development of algorithms running on edge devices ( i.e. , autonomous vehicles) faces the challenge of limited resources, which requires efficient utilization of computing systems. To resolve these challenges, we carefully design a go-to strategy, Label-Efficient Domain Adaptation, that bridges two different domains with cost-effective transfer learning. Precisely, motivated by Parameter-Efficient Fine-Tuning (PEFT) [16-18], we focus on smooth adaptation to target domains by fully exploiting well-defined source knowledge. Specifically, leveraging plug-and-play extra parameters, we substantially adapt to target domains while retaining information from the source domain (+14 % Average gain compared to DT+Full FT as shown in Fig. 1). As a result, we note that UDGA practically expand base models, efficiently boosting overall capacity under limited resources.

Given landmark datasets in 3DOD, nuScenes [6], Lyft [7] and Waymo [5], we validate the effectiveness of our UDGA framework for the camera-based multi-view 3DOD task. Notably, we achieve state-of-the-art performance in cross-domain environments and demonstrate the component-wise effectiveness through ablation studies. To summarize, our main contributions are as follows:

We introduce the Unified Domain Generalization and Adaptation (UDGA) framework, which aims to learn generalizable geometric features and improve resource efficiency for enhanced practicality in addressing distributional shift alignments.

We advocate depth-scale consistency across multi-view images to effectively address 3D geometric misalignment problems. To this end, we leverage the corresponding triangular cues between adjacent views to seamlessly bridge the domain gap.

We present a label- and parameter-efficient domain adaptation method, which requires fewer annotations and fine-tuning parameters while preserving source-domain knowledge.

We demonstrate the effectiveness of UDGA on multiple challenging cross-domain benchmarks ( i.e. , Lyft → nuScenes, nuScenes → Lyft, and Waymo → nuScenes). The results show that UDGA achieves a new state-of-the-art performance in Multi-view 3DOD.

2 Related Work

2.1 Multi-view 3D Object Detection

3D object detection [4, 19, 1, 20-26] is a fundamental aspect of computer vision tasks in the real world. Especially, Multi-view 3D Object Detection leveraging Bird's Eye View (BEV) representations [11, 12, 8] have rapidly expanded. We observe that this paradigm is divided into two categories: (i) LSSbased [27, 11, 12], and (ii) Query-based [8, 28, 10]. The former adopts explicit methods leveraging depth estimation network, and the latter concentrates on implicit methods utilizing the attention mechanism of Transformer [29]. Recently, these methods [9, 30, 31] significantly benefit from improved geometric understanding leveraging temporal inputs. Also, methods [32-35] that directly guide the model using the LiDAR teacher model significantly encourage BEV spatial details. In particular, this approach is being adopted to gradually replace LiDAR in real-world scenarios; however, it still suffers from poor generalizability due to drastic domain shifts ( e.g. , weather, country, and sensor). To mitigate these issues, we present a novel paradigm, Unsupervised Domain Generalization and Adaptation (UDGA), that effectively addresses geometric issues leveraging multi-view triangular clues and smoothly bridge differenet domains without forgetting previously learned knowledge.

2.2 Bridging the Domain Gap for 3D Object Detection

Due to the expensive cost of sophisticated sensor configurations and accurate 3D annotations for autonomous driving scenes, existing works strive to generalize 3D perception models in various data distributions. Specifically, they often fail to address the covariate shift between the training and test splits. To bridge the domain gap, existing approaches have introduced noteworthy solutions as below.

LiDAR-based. Wang et al. [36] introduced Statistical Normalization (SN) to mitigate the differences in object size distribution across various datasets. ST3D [37] leveraged domain knowledge through random object scale augmentation, and their self-training pipeline refined the pseudo-labels. SPG [38] aims to capture the spatial shape, generating the missing points. 3D-CoCo [39] contrastively adjust the domain boundary between source and target to extract robust features. LiDAR Distillation [40] generates pseudo sparse point sets in spherical coordinates and aligns the knowledge between source and pseudo target. STAL3D [41] effectively extended ST3D by incorporating adversarial learning. DTS [42] randomly re-sample the beam and aim to capture the cross-density between student and teacher models. CMDA [2] aim to learn rich-semantic knowledge from camera BEV features and adversarially guide seen sources and unseen targets, achieving state-of-the-art UDA capacity.

Camera-based. While various groundbreaking methods based on LiDAR have been researched, camera-based approaches are still limited. Due to the elaborate 2D-3D alignment, not only are LiDAR-based approaches not directly applicable, but conventional 2D visual approaches [43-46] cannot be adopted either. To mitigate these issues, STMono3D [47] self-supervise the monocular 3D detection network in a teacher-student manner. DG-BEV [14] adversarially guide the network from perspective augmented multi-view images. PD-BEV [15] explicitly supervise models by the RenderNet with pseudo labels. However, camera domain generalization methods cannot meet the performance required for the safety, struggling to address the practical domain shift in the perspective change. To narrow the gap, we introduce a Unified Domain Generalization and Adaptation (UDGA) framework that effectively promotes depth-scale consistency by leveraging occluded clues between adjacent views and then seamlessly transfers the model's potential along with a few novel labels.

2.3 Parameter Efficient Fine-Tuning

Recent NLP works fully benefit from general-purpose Large-language Models (LLM). Additionally, they have proposed Parameter-Efficient Fine-Tuning (PEFT) [17, 16, 48-50] to effectively transfer LLM power to various downstream tasks. Specifically, PEFT preserves and exploits previously learned universal information, fine-tuning only additional parameters with a few downstream labels. This paradigm enables to notably reduce extensive computational resources, and large amounts of task-specific data and also effectively address challenging domain shifts in various downstream tasks as reported by [51]. Inspired by this motivation, to address drastic perspective shifts between source and target domains, we design Label-Efficient Domain Adaptation that fully transfers generalized source potentials to target domains by fine-tuning only our extra modules with few-shot target data.

-54

-33

-53

-67

-39

-64

-80

-60

-40

-20

0

Source

Height

Pitch

Target

NDS

mAP

-60

-40

-20

0

-80

Source

∆

Height

∆

Pitch

∆

All

Front Left

Source

Target

Front

Front Right

(a) Translated perspective views

8

Q

3 Methodology

3.1 Preliminary

Multi-view 3D Object Detection is a fundamental computer vision task that involves safely localizing and categorizing objects in a 3D space exploiting 2D visual information from multiple camera views. Especially, recent landmark Multi-view 3D Object Detection models [8, 10, 9, 11, 33] are formulated as follow; arg min L ( Y, D ( V ( I, K, T )) , where Y represents the size ( l, w, h ) , centerness ( cx, cy, cz ) , and rotation ϕ of each 3D object. Also, I = { i 1 , i 2 , ..., i n } ∈ R N × H × W × 3 , K , and T = [ R | t ] denotes multi-view images, intrinsic and extrinsic parameters. Specifically, these models, which fully benefit from view transformation modules V , encode 2D visual features alongside the 3D spatial environment into a bird's eye view (BEV) representation. First, these works adopts explicit methods (BEV view transformation P as shown in Eq. 1) exploiting depth estimation network. Subsequently, Detector Head modules D supervises BEV features with 3D labels Y in a three-dimensional manner.

3.2 Domain Shifts in Multi-view 3D Object Detection

In this section, we analyze and report de facto domain shift problems arising in the Autonomous Driving system. As shown in 3.1, recent works adopt camera parameters K and T as extra inputs in addition to multi-view image I . As reported by [14], assuming that the conditional distribution of outputs for given inputs, is the same across domains, it is explained that shifts in the domain distribution are caused by inconsistent marginal distributions of inputs. To mitigate these issues, recent generalization approaches [14, 53, 47, 13, 54] often focus on covariate shift in geometric feature representation mainly due to optical changes ( i.e. , Focal length, Field-of-View, and pixel size).

This is the only part of a story. We experience drastic performance drops (up to -54 % / -67 % performance drop in NDS and mAP, respectively, as shown in Fig 2 (b)) from non-intrinsic factors ( i.e. , only extrinsic shifts). Especially, we capture a phenomenon wherein the actual depth scale from an ego-vehicle's visual sensor to an object (Fig 2 (a) red boxes) varies depending on the sensor's installation location. Followed by Pythagorean theorem, as the height difference ∆ h increases, the depth scale difference ∆ d also increases accordingly. Note that this is not limited to height solely; any shifts in deployment translation ( e.g. , along the x, y, or z axis) lead to changes in actual depth scale. As a result, perspective view differences significantly hinder the model's three-dimensional geometric understanding by causing depth inconsistency. To address above drawbacks, we introduce a novel penalising strategy that effectively boost depth consistency in various camera geometry shifts.

(ii) Label-Efficient DA

View Transformer

(i) Multi-view Overlap Depth Constraint

BEV Pool

(1) Depth projection to adjacent views

(2) Photometric Reprojection

𝐹

!

𝐹

!"#

!

𝐹

!"#

"

𝐼

!

𝐼

!"#

!

𝐼

!"#

"

Translation

Orientation

x6

DepthNet

*

Freeze

Multi-view Input

Adapter

Activation

Project Up

Oper. Block

*

Adapter

Oper. Block

Norm layer

Activation

Convolution

*

𝑯

𝑫

𝑾

𝑖

to adjacent 2

𝑖

to adjacent 1

𝐹

!

~𝐷

!

𝑝

!

𝒅𝒖, 𝒅𝒗, 𝒅

𝑇

!

(𝐾

!

%&

)

𝐾

'

!

(𝑇

'

!

%&

)

𝐾

'

"

(𝑇

'

"

%&

)

(1)

(2)

𝓛

𝒐𝒗

𝟏

𝓛

𝒑

𝟏

𝓛

𝒐𝒗

𝟐

𝓛

𝒑

𝟐

x1

Image

Backbone

②

Finetune

①

Pretrain

Scale

Heads

BEV Neck

BEV

Backbone

3.3 Multi-view Overlap Depth Constraint

Motivation. Recently, previous efforts [55, 14, 54, 56] augment multi-view images to generalize challenging perspective view gaps. However, these strategies suffer from poor generalizability in cross-domain scenarios, primarily due to the underestimated extent of view change between different sensor deployments as reported in section 3.2. To alleviate perspective gaps, we introduce Multi-view Overlap Depth Constraint, effectively encouraging perspective view-invariant learning. Here, we start from three key assumptions: First, perspective shifts between adjacent cameras in multi-view modalities are non-trivial and varied, closely akin to those observed in cross-domains ( e.g. , nuScenes → Lyft). Second, visual odometry techniques such as Structure from Motion (SfM) and Simultaneous Localization and Mapping (SLAM) often benefit from improved depth consistency through relationships between adjacent views ( e.g. , relative pose estimation). Third, in multi-view modalities, overlap regions serve as strong geometric triangular clues, seamlessly bridging between adjacent views. However, under conditions where camera parameters are input, off-the-shelf pose estimation [57-61] leads to ambiguity in learning precise geometry. To mitigate these issues, we introduce a novel depth constraint (Fig. 3 (i)) with overlap regions between adjacent cameras.

Approach. To achieve generalized BEV extraction, we directly constrain depth estimation network from adjacent overlap regions between multi-view cameras. Also, we advocate that multi-frame image inputs substantially complement geometric understanding in dynamic scenes with speedy translation and rotation shifts. To this end, we formulate corresponding depth D ∗ leveraging spatial and temporal adjacent views. First, we calculate overlap transformation matrices T i → j from Eq. 2.

where K and T are the intrinsic and extrinsic camera parameters. p ∗ i → j and p i denote corresponding pixels between adjacent views and D represent depth prediction. Then, we directly penalise unmatched corresponding depth D ∗ to smoothly induce perspective-agnostic learning as follow Eq. 3

where d represents Euclidean Distance. Also, we observe that the photometric reprojection error significantly alleviate relative geometric ambiguity. Especially, slow convergence may occur mainly

due to incorrect relationships in small overlap region (about 30 % of full resolution). To mitigates these concern, we effectively boost elaborate 2D matching, formulating L p as follow Eq. 4:

where P represents point clouds generated by D , and pe is photometric error by SSIM [62]. Also, ⟨·⟩ denotes bilinear sampling on RGB images. Concretely, we take two advantages leveraging L p in narrow occluded regions ; First, L p effectively mitigates the triangular misalignment. Second, L p potentially supports insufficiently scaled L ov . Ultimately, we alleviate perspective view gaps by directly constraining the corresponding depth and the photometric matching between adjacent views.

3.4 Label-Efficient Domain Adaptation

Motivation. There exist practical challenges in developing and deploying multi-view 3D object detectors for safety-critical self-driving vehicles. Each vehicle and each sensor requires its own model that can successfully operate in various conditions ( e.g. , dynamic weather, location, and time). Furthermore, while collecting large-scale labels in diverse environments is highly recommended, it is extremely expensive, inefficient and time-consuming. Among those, we are particularly motivated to tackle the following: (i) Stable performance, (ii) Efficiency of training, (iii) Preventing catastrophic forgetting, and (iv) Minimizing labeling cost. To satisfy these practical requirements, we carefully design an efficient and effective learning strategy, Label-Efficient Domain Adaptation (LEDA) that seamlessly transferring and preserving their own potentials leveraging a few annotated labels.

Approach. In this paper, we propose Label-Efficient Domain Adaptation, a novel strategy to seamlessly bridge domain gaps leveraging a small amount of target data. To this end, we add extra parameters A [48] consisting of bottleneck structures ( i.e. , projection down ϕ down and up ϕ up layers).

where σ and BN indicates activation function and batch normalization. We parallelly build A alongside pre-trained operation blocks B ( e.g. , convolution, and linear block) in Fig. 3 (ii) and Eq. 6;

Firstly, we feed x into ϕ down to compress its shape to [ H/r,W/r ] , where r is the rescale ratio, and then utilize ϕ up to restore it to [ H,W ] . Secondly, we fuse each outputs from B , and Adapter by exploiting skip-connections that directly link between the downsampling and upsampling paths. By doing so, these extensible modules allow to capture high-resolution spatial details while reducing network and computational complexity. Plus, it notes worthy that they are initialized by a near-identity function to preserve previously updated weights. Finally, our frameworks lead to stable recognition in both source and target domains, incrementally adapting without forgetting pre-trained knowledge.

3.5 Optimization Objective

In this section, we optimize our proposed framework UDGA using the total loss function L total (as shown in Eq. 7) during both phases ( i.e. , pre-train and fine-tune). L det denotes the detection task loss.

where we grid-search λ det , λ ov and λ p to harmonize L det , L ov and L p . Specifically, L total supervises B during generalization and A during adaptation, respectively. As a result, these strategies enable efficient learning of optimal representations in target domains while preserving pre-trained ones.

4 Experimental Results

In this section, we showcase the overall performance of our methodologies on landmark datasets for 3D Object Detection: Waymo [5], Lyft [7], and nuScenes [6]. The three datasets have different specifications; thus, we convert them to a unified detection range and coordinates for accurate comparison. We also adopt only seven parameters to achieve consistent training results under the same conditions: the location of centerness ( x, y, z ) , the size of box ( l, w, h ) , and heading angle θ . Additionally, we summarize 3D Object Detection datasets and implementation details in Appendix A.

4.1 Evaluation Metric

In this paper, following DG-BEV [14] evaluation details, we adopt the alternative metric NDS ˆ ∗ (as shown in Eq. 8) that aggregates mean Average Precision (mAP), mean Average Translation Error (mATE), mean Average Scale Error (mASE), and mean Average Orientation Error (mAOE).

We reconstruct the unified category for Unified Domain Generalization and Adaptation as follows: the 'car' for nuScenes and Lyft, and the 'vehicle' for Waymo. Furthermore, we only validate performance in the range of x , y axis from -50m to 50m. Note that we offer an empirical lower bound Direct Transfer ( i.e. , directly evaluating the model pre-trained in the source domain only), and an empirical upper bound Oracle ( i.e. , evaluating the model fully supervised in the target domain). We report Full F.T. ( i.e. , fine-tuning all parameters from the pre-trained source model) and Adapter ( i.e. , parameter efficient fine-tuning without our proposed depth constraint methods from the pre-trained source model) Furthermore, we formulate Closed Gap -representing the hypothetical closed gap by

4.2 Experiment Results

Performance Comparison in Domain Generalization. As shown in Tab. 1, we showcase four challenging generalization scenarios, and quantitatively compare our proposed methodology with existing state-of-the-art methods, which include CAM-Conv [13], Single-DGOD [44], DG-BEV [14], and PD-BEV [15]. Here, we observe that these methods still struggle to fully pilot geometric shifts from perspective changes in cross-domain scenarios. Importantly, in Lyft → nuScenes, existing methods suffer from the orientation error mainly due to significantly different ground truth directions ( i.e. , only recovering 0.198 mAOE). In nuScenes → Waymo ( i.e. , one of the most challenging

ˆ

∗

Ours(DG)

Ours(UDGA)

Ours(UDGA)

0.15

0.25

0.35

0.45

0.55

0.65

-5

5

15

25

35

45

55

65

DG

DA 1%

DA 100%

Ours(DG)

Ours(UDGA)

Ours(UDGA)

0.25

0.35

0.45

0.55

0.65

0.75

-5

5

15

25

35

45

55

DG

DA 1%

DA 100%

0

5

15

25

35

45

55

65

0.15

0.25

0.35

0.45

0.55

0.65

0.15

0.25

0.35

0.45

0.55

0.65

0

5

15

25

35

45

55

65

NDS

! ∗

# P (M)

# P (M)

nuScenes

→

Lyft

Lyft

→

nuScenes

NDS

! ∗

Full FT

Oracle(0.59)

oFullFT

DG-BEV

PD-BEV

Direct Transfer(0.21)

FullFT

Oracle(0.68)

oFull FT

PD-BEV

DG-BEV

DirectTransfer(0.30)

Adapter-B

Adapter-S

SSF

Adapter-B

oAdapter-S

SSF

Adapter-B

Adapter-S

SSF

Adapter-B

oAdapter-S

°SSF

65

scenarios due to the rear camera drop), previous approaches still show a significant gap compared to Oracle ( i.e. , -49.7 % Closed Gap). In this paper, our novel depth constraint notably addresses these issues, outperforming existing SOTAs (especially, up to +4.7 % NDS and +12.6 % Closed Gap better than DG-BEV in Lyft → nuScenes). Especially, leveraging triangular clues to explicitly supervise occluded depth contributes significantly to improving geometric consistency compared to prior approaches [14, 15, 44, 13]. Overall, we demonstrate that our novel approaches significantly enhance perspective-invariance, featuring strong association in occluded regions between multi-views.

Performance Comparison in UDGA. In Tab. 2, we show that our proposed Unified Domain Generalization and Adaptation performance compared with various PEFT approaches ( i.e. , SSF [50], and Adapter [48]). SSF directly scale and shift the deep features extracted by pre-trained operation blocks, leveraging additional normalization parameters. Adapter represents sole module performance without our proposed constraint; Adapter-B, and Adapter-S denotes base, and small version, respectively.

Existing PEFT paradigms benefit from fine-tuning only extra parameters, retaining previously updated weights. However, we observe that these paradigms do not successfully adapt to the covariate shifts originated by challenging geometric differences as reported in section 3.2. More specifically, SSF and Adapter-S, which exploit a small number of parameters, begin to capture transferable representations and then marginally adapt at the 10 % data split. Also, Adapter-B leveraging 21.3M parameters provide poor adaptation capability ( i.e. , inferior to Scratch and Full FT in Lyft → nuScenes 100 % ).

However, our proposed strategy seamlessly adapt to target domains in 1 % , and 5 % , effectively bridge perspective gaps. Furthermore, our proposed strategy show superior performance gain (outperforming Scratch in Lyft → nuScenes 50 % , and Full FT in both Lyft → nuScenes, and nuScenes → Lyft 100 % ), effectively adapting to novel targets. It is noteworthy that the most effective adaptation is achieved by updating extra parameters (less than 20 % of the total), which demonstrates the practicality and efficiency of our novel UDGA strategy as shown in Fig. 4. In addition, unlike Full FT, it proves that our UDGA framework stably adapts to the target without forgetting previously learned knowledge as

shown in Fig. 1 and Tab. 6. Overall, our proposed method demonstrate the effectiveness of training strategy in various experimental setups, efficiently expanding to targets with about 20 % of overall parameters. Note that we report additional experiments and details of UDGA in Appendix C.

4.3 Ablation studies

Exploring the Synergy Between Modules. To better understand the role of each module, we present ablation studies of UDGA in this experiment (Tab. 3). Precisely, we aim to analyze the pros and cons in both training steps ( i.e. , pre-train B and fine-tune A ), with the objective of effectively elucidating the plausibility of UDGA. First, the strategy trained from scratch leveraging our depth constraint significantly recovers performance drop from the sensor deployment shift (up to +20.8 % NDS). However, this strategy finds it difficult to provide a practical solution for Multi-view 3DOD, mainly due to unsatisfying generalizability. Additionally, although LEDA without L ov and L p yields improved performance, it fails to transfer its previously learned potential, resulting in only +2.3 % NDS compared to our individual depth constraint. To tackle these issues, we concentrate on bridging two distinct domains by capturing generalized perspective features. Especially, our depth constraint (only trained during pre-training B ) significantly encourages understanding of the target in LEDA during fine-tuning A with a 10 % split, addressing the geometric covariate shift (+30.3 % NDS). Furthermore, UDGA strategy using L ov and L p in both phases learns the transferable knowledge and shows impressive improvement (+42.5 % NDS). Finally, UDGA successfully presents an effective and efficient paradigm for Multi-view 3DOD, highlighting notable recovery in novel target scenarios.

Effect of Overlap Depth Constraint. In Tab. 4, we carefully evaluate our depth constraint components in various cross-domain environments. Here, L d denotes depth supervision by LiDAR. Also, we design ext aug that globally rotate ground truths with randomly initialized angle α to release the direction shift. More importantly, we observe that perspective view shifts from different sensor deployments lead to severe translation and orientation errors. To tackle these issues, we advocate that L ov , which leverages strong relationships between adjacent views, effectively alleviating perspective gaps compared to L d (recovering up to +19 % NDS in Lyft → nuScenes). L p relieves slight misalignment, encouraging depth-scale consistency. Additionally, our ext aug substantially boost stable generalization, suppressing orientation errors (up to +1.4 % additional NDS gain). Consequently, our novel objectives ( L ov and L p ) demonstrate their effectiveness, significantly tackling geometric errors.

Front Left

Front

Front Right

GT

BEVDepth

Ours

60

50

30

20

10

0(m)

40

4.4 Qualitative Analysis

To qualitatively analyze the effectiveness of Multi-view Overlap Depth Constraint, we present additional visualized results in Fig. 5. For accurate comparison, we conduct binary masking leveraging given sparse depth ground truths. In middle row, BEVDepth fail to perceive hard samples ( e.g. , far distant and occluded objects) in yellow boxes, mainly due to different extent of deformation relative to perspective as reported in section 3.2. We aim to tackle this problem, explicitly bridging adjacent views in various dynamic scenes. Precisely, in bottom row, we showcase distinguishable results in yellow boxes, capturing semantic details from various view deformation. As as results, we qualitatively demonstrate that our proposed method effectively encourage depth consistency and detection robustness, significantly improving geometric understanding in cross-domain scenarios.

5 Conclusion

Limitations. While our work significantly improves the adaptability of 3D object detection, it cannot guarantee seamless adaptation due to several limitations, including: (1) The performance does not match that of 3D object detection models using LiDAR point clouds. (2) Our Multi-view Overlap Depth Constraint relies on the presence of overlapping regions between images. (3) Achieving fully domain-agnostic approaches without any target labels remains challenging. As a result, it is essential to incorporate a fallback plan when deploying the framework in safety-critical real-world scenarios.

Summary. Multi-View 3DOD models often face challenges in expanding appropriately to unfamiliar datasets due to inevitable domain shifts ( i.e. , changes in the distribution of data between the training and testing phases). Especially, the limited resource ( e.g. , excessive computational overhead and taxing expensive and taxing data cost) leads to hinder the successful deployment of Multi-View 3DOD. To mitigate above drawbacks, we carefully design Unified Domain Generalization and Adaptation (UDGA), a practical solution for developing Multi-View 3DOD. We first introduce Multi-view Overlap Depth Constraint that advocates strong triangular clues between adjacent views, significantly bridging perspective gaps. Additionally, we present a Label-Efficient Domain Adaptation approach that enables practical adaptation to novel targets with largely limited labels ( i.e. , 1 % and 5 %) without forgetting well-aligned source potential. Our UDGA paradigm efficiently fine-tune additional parameters leveraging significantly fewer annotations by effectively transferring from the source to target domain. In summary, our extensive experiments in various landmark datasets( e.g. , nuScenes, Lyft and Waymo) show that our novel paradigm, UDGA, provide a practical solution, outperforming current state-of-the-art models on Multi-view 3D object detection.

Acknowledgments and Disclosure of Funding

This work was primarily supported by Samsung Advanced Institute of Technology (SAIT) (85%), partially supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2019-0-00079, Artificial Intelligence Graduate School Program (Korea University), 5%), and Culture, Sports and Tourism R&D Program through the Korea Creative Content Agency grant funded by the Ministry of Culture, Sports and Tourism in 2024(International Collaborative Research and Global Talent Development for the Development of Copyright Management and Protection Technologies for Generative AI, RS-202400345025, 5%; Research on neural watermark technology for copyright protection of generative AI 3D content, RS-2024-00348469, 5%).

References

Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai. Transfusion: Robust lidar-camera fusion for 3d object detection with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1090-1099, 2022.

Gyusam Chang, Wonseok Roh, Sujin Jang, Dongwook Lee, Daehyun Ji, Gyeongrok Oh, Jinsun Park, Jinkyu Kim, and Sangpil Kim. Cmda: Cross-modal and domain adversarial adaptation for lidar-based 3d object detection. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 972-980, 2024.

Tingting Liang, Hongwei Xie, Kaicheng Yu, Zhongyu Xia, Zhiwei Lin, Yongtao Wang, Tao Tang, Bing Wang, and Zhi Tang. Bevfusion: A simple and robust lidar-camera fusion framework. Advances in Neural Information Processing Systems , 35:10421-10434, 2022.

Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela L Rus, and Song Han. Bevfusion: Multi-task multi-sensor fusion with unified bird's-eye view representation. In 2023 IEEE international conference on robotics and automation (ICRA) , pages 2774-2781. IEEE, 2023.

Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2446-2454, 2020.

Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11621-11631, 2020.

John Houston, Guido Zuidhof, Luca Bergamini, Yawei Ye, Long Chen, Ashesh Jain, Sammy Omari, Vladimir Iglovikov, and Peter Ondruska. One thousand and one hours: Self-driving motion prediction dataset. In Conference on Robot Learning , pages 409-418. PMLR, 2021.

Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In Conference on Robot Learning , pages 180-191. PMLR, 2022.

Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird's-eye-view representation from multi-camera images via spatiotemporal transformers. In European conference on computer vision , pages 1-18. Springer, 2022.

Wonseok Roh, Gyusam Chang, Seokha Moon, Giljoo Nam, Chanyoung Kim, Younghyun Kim, Jinkyu Kim, and Sangpil Kim. Ora3d: Overlap region aware multi-view 3d object detection. arXiv preprint arXiv:2207.00865 , 2022.

Junjie Huang, Guan Huang, Zheng Zhu, Yun Ye, and Dalong Du. Bevdet: High-performance multi-camera 3d object detection in bird-eye-view. arXiv preprint arXiv:2112.11790 , 2021.

Yinhao Li, Zheng Ge, Guanyi Yu, Jinrong Yang, Zengran Wang, Yukang Shi, Jianjian Sun, and Zeming Li. Bevdepth: Acquisition of reliable depth for multi-view 3d object detection. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 1477-1485, 2023.

Jose M Facil, Benjamin Ummenhofer, Huizhong Zhou, Luis Montesano, Thomas Brox, and Javier Civera. Cam-convs: Camera-aware multi-scale convolutions for single-view depth. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11826-11835, 2019.

Shuo Wang, Xinhai Zhao, Hai-Ming Xu, Zehui Chen, Dameng Yu, Jiahao Chang, Zhen Yang, and Feng Zhao. Towards domain generalization for multi-view 3d object detection in bird-eyeview. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13333-13342, 2023.

Hao Lu, Yunpeng Zhang, Qing Lian, Dalong Du, and Yingcong Chen. Towards generalizable multi-camera 3d object detection via perspective debiasing. arXiv preprint arXiv:2310.11346 , 2023.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021.

Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691 , 2021.

Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602 , 2021.

Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors , 18(10):3337, 2018.

Dennis Park, Rares Ambrus, Vitor Guizilini, Jie Li, and Adrien Gaidon. Is pseudo-lidar needed for monocular 3d object detection? In IEEE/CVF International Conference on Computer Vision (ICCV) , 2021.

Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin. Probabilistic and Geometric Depth: Detecting objects in perspective. In Conference on Robot Learning (CoRL) 2021 , 2021.

Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin. Fcos3d: Fully convolutional one-stage monocular 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 913-922, 2021.

Garrick Brazil and Xiaoming Liu. M3d-rpn: Monocular 3d region proposal network for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9287-9296, 2019.

Zechen Liu, Zizhang Wu, and Roland Tóth. Smoke: Single-stage monocular 3d object detection via keypoint estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops , pages 996-997, 2020.

Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4490-4499, 2018.

Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12697-12705, 2019.

Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16 , pages 194-210. Springer, 2020.

Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position embedding transformation for multi-view 3d object detection. In European Conference on Computer Vision , pages 531-548. Springer, 2022.

A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems , 2017.

Jinhyung Park, Chenfeng Xu, Shijia Yang, Kurt Keutzer, Kris M Kitani, Masayoshi Tomizuka, and Wei Zhan. Time will tell: New outlooks and a baseline for temporal multi-view 3d object detection. In The Eleventh International Conference on Learning Representations , 2022.

Chenyu Yang, Yuntao Chen, Hao Tian, Chenxin Tao, Xizhou Zhu, Zhaoxiang Zhang, Gao Huang, Hongyang Li, Yu Qiao, Lewei Lu, et al. Bevformer v2: Adapting modern image backbones to bird's-eye-view recognition via perspective supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17830-17839, 2023.

Yu Hong, Hang Dai, and Yong Ding. Cross-modality knowledge distillation network for monocular 3d object detection. In ECCV , Lecture Notes in Computer Science. Springer, 2022.

Peixiang Huang, Li Liu, Renrui Zhang, Song Zhang, Xinli Xu, Baichao Wang, and Guoyi Liu. Tig-bev: Multi-view bev 3d object detection via target inner-geometry learning. arXiv preprint arXiv:2212.13979 , 2022.

Zehui Chen, Zhenyu Li, Shiquan Zhang, Liangji Fang, Qinhong Jiang, and Feng Zhao. Bevdistill: Cross-modal bev distillation for multi-view 3d object detection. arXiv preprint arXiv:2211.09386 , 2022.

Sujin Jang, Dae Ung Jo, Sung Ju Hwang, Dongwook Lee, and Daehyun Ji. Stxd: Structural and temporal cross-modal distillation for multi-view 3d object detection. Advances in Neural Information Processing Systems , 36, 2024.

Yan Wang, Xiangyu Chen, Yurong You, Li Erran Li, Bharath Hariharan, Mark Campbell, Kilian Q Weinberger, and Wei-Lun Chao. Train in germany, test in the usa: Making 3d object detectors generalize. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11713-11723, 2020.

Jihan Yang, Shaoshuai Shi, Zhe Wang, Hongsheng Li, and Xiaojuan Qi. St3d: Self-training for unsupervised domain adaptation on 3d object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10368-10378, 2021.

Qiangeng Xu, Yin Zhou, Weiyue Wang, Charles R Qi, and Dragomir Anguelov. Spg: Unsupervised domain adaptation for 3d object detection via semantic point generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15446-15456, 2021.

Zeng Yihan, Chunwei Wang, Yunbo Wang, Hang Xu, Chaoqiang Ye, Zhen Yang, and Chao Ma. Learning transferable features for point cloud detection via 3d contrastive co-training. Advances in Neural Information Processing Systems , 34:21493-21504, 2021.

Yi Wei, Zibu Wei, Yongming Rao, Jiaxin Li, Jie Zhou, and Jiwen Lu. Lidar distillation: Bridging the beam-induced domain gap for 3d object detection. In European Conference on Computer Vision , pages 179-195. Springer, 2022.

Yanan Zhang, Chao Zhou, and Di Huang. Stal3d: Unsupervised domain adaptation for 3d object detection via collaborating self-training and adversarial learning. IEEE Transactions on Intelligent Vehicles , 2024.

Qianjiang Hu, Daizong Liu, and Wei Hu. Density-insensitive unsupervised domain adaption on 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17556-17566, 2023.

Vidit Vidit, Martin Engilberge, and Mathieu Salzmann. Clip the gap: A single domain generalization approach for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3219-3229, 2023.

Aming Wu and Cheng Deng. Single-domain generalized object detection in urban scene via cyclic-disentangled self-distillation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition , pages 847-856, 2022.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16000-16009, 2022.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020.

Zhenyu Li, Zehui Chen, Ang Li, Liangji Fang, Qinhong Jiang, Xianming Liu, and Junjun Jiang. Unsupervised domain adaptation for monocular 3d object detection via self-training. In European conference on computer vision , pages 245-262. Springer, 2022.

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning , pages 2790-2799. PMLR, 2019.

Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Scaling & shifting your features: A new baseline for efficient model tuning. Advances in Neural Information Processing Systems , 35:109-123, 2022.

Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems , 35:1950-1965, 2022.

Yue-Jiang Dong, Yuan-Chen Guo, Ying-Tian Liu, Fang-Lue Zhang, and Song-Hai Zhang. Ppea-depth: Progressive parameter-efficient adaptation for self-supervised monocular depth estimation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 1609-1617, 2024.

Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In Proceedings of the 1st Annual Conference on Robot Learning , pages 1-16, 2017.

Qiqi Gu, Qianyu Zhou, Minghao Xu, Zhengyang Feng, Guangliang Cheng, Xuequan Lu, Jianping Shi, and Lizhuang Ma. Pit: Position-invariant transform for cross-fov domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8761-8770, 2021.

Tzofi Klinghoffer, Jonah Philion, Wenzheng Chen, Or Litany, Zan Gojcic, Jungseock Joo, Ramesh Raskar, Sanja Fidler, and Jose M Alvarez. Towards viewpoint robustness in bird's eye view segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8515-8524, 2023.

Yunhan Zhao, Shu Kong, and Charless Fowlkes. Camera pose matters: Improving depth prediction by mitigating pose distribution bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15759-15768, 2021.

Ke Wang, Bin Fang, Jiye Qian, Su Yang, Xin Zhou, and Jie Zhou. Perspective transformation data augmentation for object detection. IEEE Access , 8:4935-4943, 2019.

Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J Brostow. Digging into self-supervised monocular depth estimation. In Proceedings of the IEEE/CVF international conference on computer vision , pages 3828-3838, 2019.

Wenjing Bian, Zirui Wang, Kejie Li, Jia-Wang Bian, and Victor Adrian Prisacariu. Nopenerf: Optimising neural radiance field with no pose prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4160-4169, 2023.

Xiaoyang Lyu, Liang Liu, Mengmeng Wang, Xin Kong, Lina Liu, Yong Liu, Xinxin Chen, and Yi Yuan. Hr-depth: High resolution self-supervised monocular depth estimation. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pages 2294-2301, 2021.

Hang Zhou, David Greenwood, and Sarah Taylor. Self-supervised monocular depth estimation with internal feature fusion. arXiv preprint arXiv:2110.09482 , 2021.

Yi Wei, Linqing Zhao, Wenzhao Zheng, Zheng Zhu, Yongming Rao, Guan Huang, Jiwen Lu, and Jie Zhou. Surrounddepth: Entangling surrounding views for self-supervised multi-camera depth estimation. arXiv preprint arXiv:2204.03636 , 2022.

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing , 13(4): 600-612, 2004.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming Li, and Gang Yu. Class-balanced grouping and sampling for point cloud 3d object detection. 2019.

Appendices

Table of Contents

A Datasets

We evaluate overall performance on landmark datasets for 3D Object Detection: Waymo [5], Lyft [7], and nuScenes [6]. The three datasets have different point cloud ranges and specifications. Hence, we convert them to a unified range and coordinates for accurate comparison. We also adopt only seven parameters to achieve consistent training results under the same conditions: center locations ( x, y, z ) , box size ( l, w, h ) , and heading angle θ . Additionally, to estimate practical degradation due to changes in camera positioning, we conducted a proof of concept by generating data similar to the nuScenes using the CARLA simulation. The details are as follows:

Waymo The Waymo dataset [5] consists of high-quality and large-scale data with 230 K frames from all 1,150 scenes using multiple LiDAR scanners and cameras. Furthermore, for the generalization purpose, Waymo is recorded at diverse cities, weather conditions, and times. For object detection in 2D or 3D, Waymo provides point cloud-annotated 3D bounding boxes as 3D data pairs and RGB image-annotated 2D bounding boxes as 2D data pairs.

nuScenes The nuScenes dataset [6] uses 6 cameras that cover a full 360-degree range of view and a single LiDAR sensor to obtain 40 K frames from 20-second-long 1,000 video sequences, which are fully annotated with 3D bounding boxes for 10 object classes. The nuScenes dataset covers 28k annotated samples for training. Also, validation and test contain 6k scenes each. The nuScenes frames are captured in the same manner as Waymo dataset for the data diversity. But unlike Waymo, nuScenes provides labels only for the point cloud data with 23 classes of 3D bounding boxes.

Lyft The Lyft dataset [7] is motivated by the impact of large-scale datasets on Machine Learning and consists of over 1,000 hours of data. This was collected by a fleet of 20 autonomous vehicles along a fixed route in Palo Alto, California, over a four-month period. It consists of 170,000 scenes (each scene is 25 seconds long) and contains 3D bounding boxes with the precise positions of nearby vehicles, cyclists, and pedestrians over time. In addition, the Lyft dataset includes a high-definition semantic map with 15,242 labelled elements and a high-definition aerial view over the area.

CARLA To quantify the performance drop resulting from camera shifts, we employed an autonomous driving simulation powered by CARLA [52] 0.9.14 and Unreal Engine 4.26. We collected 24K frames for training and 1K frames for each evaluation, driving through Town10 under cloudless weather conditions between sunrise and sunset times. This dataset includes over 100 vehicles and 30 pedestrians in random locations. In Fig. 6, the Source utilizes 6 nuScenes-like cameras and 6 LiDARs, while the Target has perturbed sensors. From the Source sensors, the Height increases by 0.65m and the Pitch increases by 5 degrees. The All synthetically moves the x, y, z-coordinates by -0.12m, 0.65m, and -0.2m/+0.2m, respectively, and rotates the yaw by -5/+5 degrees, depending on their directions. Each target sets is collected simultaneously with the Source.

B Implementation Details

To validate the effectiveness of our proposed methods, we adopt BEVDepth [12] and BEVFormer [9] as our base detectors. Both detectors utilize ResNet50 [63] backbone that initialized from ImageNet1K. Also, we construct BEV representations within a perception range of [-50.0m, 50.0m] for both the X and Y axes. In BEVDepth, we reshape multi-view input image resolutions as follow: [256 , 704] for nuScenes, [384 , 704] for Lyft, [320 , 704] for Waymo. As following DG-BEV [14], we train 24 epochs with AdamW optimizer by learning rate 2e-4 in pre-training phase. The training takes approximately 18 hours using one A100 GPU. In fine-tuning phase, we conduct an extensive grid search to determine the optimal learning rate proportional to the number of learnable parameters. Note that we extensively augment various image conditions as detailed in [14].

C Additional Experiments

In this appendix, we present additional experiments to validate the effectiveness of our proposed paradigm. First, Tab. 6 summarizes the overall results of our work from the perspective of domain shift. We also analyze how changes in camera positioning worsen the performance and evaluate whether existing augmentation methods can mitigate the deterioration. Additionally, we conduct ablation studies to enhance the LEDA structure, including comparisons with formal adapters. Finally, we present the comparison results with the transformer-based detector. The qualitative analysis of the multi-view results from our proposed paradigm is included towards the end of this chapter.

Performance across domains. In this section, we compare our proposed UDGA with existing solutions ( i.e. , DG, UDA) in various cross-domain conditions (see Tab. 6). We aim to practically mitigate perspective shifts without hindering well-defined source knowledge. Our DG branch achieves top performance, surpassing Direct Transfer in the Source domain. The UDGA, which follows DG, improves Target accuracy without compromising Source performance. Especially, we advocate that UDGA enables efficient adaptation with significantly down-scaled data split ( i.e. , 1 % and 5 % ). Also, it is noteworthy that UDGA do not forget previously learned potentials, fully transferring to target domains (up to +14.2 % NDS gain in Lyft → nuScenes). Overall, UDGA provide a practical solution to address perspective view changes, efficiently adapting with only tiny split.

Practical domain shift analysis. We analyze the impact of changes in camera geometry on 3D object estimation. The experimental model is trained using only the source dataset on ResNet50-based BEVDet and then evaluated on three sets of (source, target) to analyze performance differences. In Tab. 7, the performance of the source is similar in all test sets. On the other hand, the performance of the target decreases significantly in all cases. Since this experiment is conducted in the same environment with the same camera sensors, it demonstrates how much performance degradation is caused by the position of the camera. The set with the largest performance drop in the target is Height , where the mATE value increased significantly. The target All exhibits the worst mASE and mAOE, while the other measures also deteriorated by a similar amount as Height .

Conventional augmentation methods enhance the robustness of the model. We evaluate some of them in Tab. 8. GT sampling and CBGS [64] are techniques designed to balance ground truths. 2D augmentation directly augment multi-view inputs (i.e., image resize, crop and paste, contrast and brightness distortion). 3D and extrinsic methods are global augmentations that address both input and ground truths, and ground truths only, respectively. These methods enhance geometric understanding from input noises. However, in dynamic view changes (i.e., cross-domain), they still suffer from geometric inconsistency and show poor generalization capability. Moreover, various 2D approaches do not guarantee geometric alignments between 2D images and 3D ground truths and relevant studies have not been explored well, as reported in [14] and [55].

Searching adapter structures. We explore various modules and structures to find a suitable adapter architecture. Tab. 9, 10 show which structures and locations affects the model's performance. For adapter locations, performance is optimal when adapters are attached to all modules, gradually

improving with the addition of more. Exceptionally, attaching adapters only at the Detection Head leads to a decline in Lyft → nuScenes. In addition, Tab. 10 represents the performance of various adapter structures. The combination of Convolution and Linear layer respectively for Project Down and Up shows the best performance in both tasks. Note that training with fewer parameters(8.8M) is more effective. However, we suggest that large-scale parameters may require a larger dataset or more training, as we only trained on 10 % of the target dataset for less than 20 epochs in this experiment.

Comparison of UDGA on BEVFormer. To demonstrate the validation of UDGA, we further compare performance on BEVFormer-small (33.5M parameters) with Full FT. For accurate comparison, we provide Oracle , and Direct Transfer in nuScenes → Lyft task.

BEVFormer adopt Query-based view transformation modules V as follow Eq. 10:

where q , k and v represents query, value and key in Transformer, and then P xyz denotes pre-defined anchor BEV positions by K , and T . Here, Query-based module benefits from CrossAttn with sparse query sets, implicitly learning geometric information. Thus, we reconstruct our UDGA paradigm without explicit depth constraints. First, we adopt linear-based bottleneck structures with Layer Normalization in Eq. 11. ϕ up and ϕ down denote the projection up and down layer.

where ffn denotes feed-forward networks, and LN represents Layer Normalization. We conduct experiments by plugging these extra modules, which accounts for 36 % of the total parameters, into BEVFormer. As a result, we achieve significant adaptation performance with the 50 % data split. Notably, we demonstrate effectiveness, achieving parity with Full FT in the 100 % data split.

Additional qualitative analysis. In this section, we further visualize our depth quality in various scenarios ( i.e. , Lyft, and nuScenes). Not only our overlap depth constraint significantly improve depth

Front Left

Source

Pitch

Front

Front Right

Back

Back Right

Back Left

Source

All

Source

Height

consistency in occluded regions, but also show better spatial understanding for hard samples ( e.g. , far and low distinguishable objects). Additionally, we note that our method effectively complement insufficient contextual recognition caused by sparse depth gt in Fig. 7 (b). Overall, we stably deploy Multi-View 3DOD by leveraging effective association between adjacent views.

D Broader Impacts.

Our framework is a practical AI algorithm that enhances its generalization ability to handle domain changes robustly, enabling us to effectively reduce data costs and computing resources required for adaptation. Practically, our method makes it suitable for deployment in mass-produced vehicles, where the algorithm can inherit the knowledge of well-trained pretrained weights while self-learning to adapt to each fleet environment. The adaptation learning process is also simplified, making it easier to transfer improved pretrained networks. Furthermore, by demonstrating superior performance compared to previous methods that relied on LiDAR for auxiliary depth networks, our approach reduces the dependency on lidar modality. This suggests the feasibility of excluding expensive LiDAR sensors from future autonomous vehicles.

Front Left

GT

BEVDepth

Front

Front Right

Back

Back Right

Back Left

Ours

GT

BEVDepth

Ours

(a) Lyft

(b) nuScenes

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have summarized the contributions of our work at the end of Sec. 1 (Introduction) as well as in the abstract.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed and presented limitations in Sec. 5 (Conclusion).

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided detailed information about the implementations in Sec. 4 (Experiment) and in Appendix A, B.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We plan to release source codes upon acceptance. At current phase, we have provided details of implementations and necessary references to prior works in Sec. 4 (Experiment) and in Appendix A, B.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have provided detailed information about the implementations in Sec. 4 (Experiment) and in Appendix A, B.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have provided details of computing resources in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We conform and follow the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed both impacts in Sec. 5 (Conclusion).

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We discussed the potential risks of the proposed method, which we judge to pose no notable risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have provided credits for the benchmark dataset and cited the baseline models.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an Unified Domain Generalization and Adaptation(UDGA) scheme for multi-view 3D object detection. The main components of the proposed method are (1) depth inconsistency-based constraints from multi-view and (2) an efficient domain adaptation scheme (LEDA). The multi-view depth inconsistency constraints improve the domain gap caused by geometric gaps when it comes to perspective view changes. LEDA offers adaptation with fewer amounts of labels while preserving the knowledge from the source domain.\n\nSTRENGTHS:\nThe paper clearly addresses which problem it focuses on, and each proposed module has solid goals to address existing limitations. The experimental results from multiple datasets also seem convincing.\n\nWEAKNESSES:\nI have a few concerns about the demonstration of the proposed method as well as addressing existing works.\n\n(1) The method part needs to be improved. For instance, specifying every input and output tensor dimension (i.e., depth estimation network, etc.) would significantly help readers understand what each module is doing more quickly. \n\n(2) Using the depth inconsistency seems similar to DETR3D's[1] inconsistency constraint on RGB features. In DETR3D, it is even mentioned that using the RGB feature is better than having explicit depth estimation.\nBased on this, the depth estimation network can introduce additional parameters for optimization while not being very helpful. I cannot see experiments or comparisons with DETR3D. \n\n(3) More related works need to be addressed. Currently, the paper focuses on multi-view-based domain generalization for 3D detection. However, there is also another active line of research with LiDAR-based domain adaptation for 3D detection, pioneered by ST3D[2]. Clearly addressing the difference between those researches will make the reader recognize the line of research that the paper focuses on. \n\n(4) The effectiveness of LEDA is not well demonstrated, as claimed in the paper, compared to existing approaches. \n\n[1] Detr3d: 3d object detection from multi-view images via 3d-to-2d queries, Wang, Yue, et al. CORL, 2022\n\n[2] St3d: Self-training for unsupervised domain adaptation on 3d object detection, Yang, Jihan. et al., CVPR 2021.\n\nQUESTIONS:\nI have a few questions that correspond to the weakness that I mentioned.\n\n(1) Is there a justifiable reason why using Depth inconsistency constraint is superior to using deep RGB feature-based inconsistency ( as in DETR3D)?\n\n(2) As one of the main proposed contributions claimed, a detailed analysis of LEDA seems missing. How much does LEDA improve the proposed system compared to existing approaches in terms of efficiency and accuracy?\n\nAddressing those two questions properly will improve the quality of the paper and, subsequently, the reviewer's rating.\n\nLIMITATIONS:\nThe limitations are addressed", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is about Unified Domain Generalization and Adaptation for outdoor 3D object detection. To address the geometric misalignment between the source and target domains, Multi-view Overlap Depth Constraint that leveraging the strong association between multi-views and Label-Efficient Domain Adaptation are proposed. Experiments are done with several cross-dataset settings.\n\nSTRENGTHS:\n1. The domain generalization and adaption is important for 3D object detection.\n2. The writing is easy to follow.\n\nWEAKNESSES:\n1.\tIn the past works, the comparative methods, such as DG-BEV, and PD-BEV, reported the performance on both source and target domains. Yet it seems that the reported metrics in Table 1 do not align with the reported results in the two papers. What is the reason for such changes?\n2.\tIt is noteworthy that the NDS is specific for nuScenes and different for other datasets. It is important to discriminate such differences.\n3.\tThe authors claim that this work is toward 3D universal detection, yet there are other works toward 3D universal detection, such as [1][2]. It is better to illustrate the differences over these methods, both in discussion and experimental comparison.  Besides, there are other works devoted to multi-dataset training for 3D detection for generalization, also missed in the paper. Can the proposed method be applied in multi-source training and multi-target testing?\n4.\tThe Domain Adapter design seems simple, yet the motivation and potential of such design is missed. It is important since it  is  one of the two components of this work.\n[1] Towards universal LiDAR-based 3D object detection by multi-domain knowledge transfer. ICCV 2023.\n[2] Cross-Dataset Sensor Alignment: Making Visual 3D Object Detector Generalizable. PMLR 2023.\n\nQUESTIONS:\nThe experimental comparison and the novelty, importance of the work.\n\nLIMITATIONS:\nThe limitation part has been addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an adaptation of 3D object detectors to varying target environments using two major strategies. The proposed multi-view overlap depth constraint leverages associations across views to learn view-invariant features. Additionally, a LORA-like structure is designed for parameter-efficient adaptation, accommodating scenarios with limited target data. Experiments on three benchmark datasets demonstrate the effectiveness of the proposed approach, with minimal modification of parameters.\n\nSTRENGTHS:\n+ The paper addresses a significant and practical issue in 3D object detection, aiding in the development of robust models for dynamically changing testing environments.\n+ The flowchart clearly illustrates the core components, making it easy for readers to grasp the main idea.\n+ Extensive experiments have been conducted, and the proposed strategy significantly outperforms the pre-trained source model and even surpasses full fine-tuning strategies.\n\nWEAKNESSES:\n- The two proposed strategies appear somewhat disconnected. While the multi-view idea is interesting, the technical details are confusing. Equation (5) includes three different objectives, but their importance or sensitivity is not discussed. The latter adaptation strategy resembles existing works and lacks specific design for the 3D detection task. This is also not correlated with the view-transform. The isolated adaptation modules seem more like a stacking of existing techniques rather than a cohesive addition.\n- More discussion and comparisons with previous multi-view augmentation strategies would help clarify the merits and innovations of the proposed approach. The current claim that existing strategies are poorly generalizable is somewhat unconvincing.\n\nQUESTIONS:\nPlease refer to the weakness section.\n\nLIMITATIONS:\nThe authors have not provided a discussion on the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-14"}

comments: SUMMARY:
This paper focuses on multi-view 3D object detection. The authors proposed a unified domain generalization and adaptation-based detection method. To enhance the detection model for unseen datasets and address the geometric misalignment problem, the authors proposed a multi-view overlap depth constraint module and a label-efficient domain adaptation technique. The comprehensive experiments on lar-scale datasets, including nuScenes, Lyfy and Waymo have demonstrate the effectiveness of the proposed method.

STRENGTHS:
- The storyline is clear, the authors provided detailed motivation analysis in the introduction and present their innovation clearly.
- The performance is strong. The method shows SOTA performance in variance benchmarks with fast speed.
- Most of the figures are clear. This paper is also well-written.

WEAKNESSES:
- There is no source code for review.

QUESTIONS:
This paper seems well-shaped to me. However, I'm not an expert in this field, so I'm open to discussion with other reviewers if they hold opponent opinions to me.

LIMITATIONS:
Please refer to the above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-16