Uni$^2$Det: Unified and Universal Framework for Prompt-Guided Multi-dataset 3D Detection
Authors: Yubin Wang
Zhikang Zou
Xiaoqing Ye
Xiao Tan
Errui Ding
Cairong Zhao
Venue: ICLR 2025 Poster (2025)
Keywords: Automatic Driving
3D Object Detection
Multi-Dataset Training
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We present Uni$^2$Det for unified and universal multi-dataset training on 3D detection across diverse domains and generalization to unseen domains.

Abstract: We present Uni$^2$Det, a brand new framework for unified and universal multi-dataset training on 3D detection, enabling robust performance across diverse domains and generalization to unseen domains. Due to substantial disparities in data distribution and variations in taxonomy across diverse domains, training such a detector by simply merging datasets poses a significant challenge. Motivated by this observation, we introduce multi-stage prompting modules for multi-dataset 3D detection, which leverages prompts based on the characteristics of corresponding datasets to mitigate existing differences. This elegant design facilitates seamless plug-and-play integration within various advanced 3D detection frameworks in a unified manner, while also allowing straightforward adaptation for universal applicability across datasets. Experiments are conducted across multiple dataset consolidation scenarios involving KITTI, Waymo, and nuScenes, demonstrating that our Uni$^2$Det outperforms existing methods by a large margin in multi-dataset training. Notably, results on zero-shot cross-dataset transfer validate the generalization capability of our proposed method. Our code is available at https://github.com/ThomasWangY/Uni2Det.

Published as a conference paper at ICLR 2025
UNI UNIFIED AND UNIVERSAL FRAMEWORK FOR PROMPT-GUIDED MULTI-DATASET 3D DETECTION
Yubin Wang" Zhikang Zou²*, Xiaoqing Ye², Xiao Tan", Errui Ding", Cairong Zhao 1 1 School of Computer Science and Technology. Tongji University, Baidu Inc.
ABSTRACT
We present Uni a brand new framework for unified and universal multi- dataset training on 3D detection, enabling robust performance across diverse domains and generalization to unseen domains. Due to substantial disparities in data distribution and variations in taxonomy across diverse domains, train- ing such a detector by simply merging datasets poses a significant challenge. Motivated by this observation, we introduce multi-stage prompting modules for multi- 3D detection, which leverages prompts based on the characteristics of corresponding datasets mitigate existing differences. This elegant design facilitates seamless plug-and-play integration within various advanced 3D detection frameworks in unified manner, while also allowing straightforward adaptation for universal applicability across datasets. Experiments are conducted across mul- tiple dataset consolidation scenarios involving KITTI, Waymo, and nuScenes, demonstrating that our UnièDet outperforms existing methods by a large margin in multi-dataset training. Notably, results on zero-shot cross-dataset transfer validate the generalization capability of our proposed method. Our code is available at wwt.tttw..ocmmmmoaoaaaooeeeoeeee hww.:i/two......omtaaaagnnng///
INTRODUCTION
With the ability to capture precise geometric information of entire scenes, LiDAR has become an essential sensor for most autonomous vehicles. Due the rapid development of large-scale annotated 3D LiDAR datasets such as Waymo (Sun et al. (2200), nuScenes (Caesar et al. 2020). and
KITTI (Geiger et al.. 2012). LiDAR-based models play significant role in various critical perception
tasks for autonomous vehicles, particularly in 3D object detection. Recent studies (Lang et al.
2019
Deng et al. 2021 Shi et al., 2020a| 2023 Chen et al.. 2017] Wei et al., 2022 Yin et al. [021 Wang
al. 2022) have made significant advancements in 3D detection using large-scale benchmarks and
have demonstrated superior performance by leveraging precise 3D geometric information extracted
from point clouds. However, despite these breakthroughs, current LiDAR-based models typically adhere to a paradigm of training and testing within a single dataset, which limits the source data to narrow domain, as shown in Figure T().. Deploying dataset-specifie models directly onto other datasets equipped with different LiDAR systems often leads to significant performance degradation due to substantial domain shifts YYang et al. 2021 2022). Consequently, the single-dataset paradigm fails to produce robust and generalizable perception model, leading to poor performance on different datasets and further impairing the generalization ability.
The availability of vast training data in 2D vision (Goyal et al. 2021 Kirillov et al. 2023 Wang
al. 2023) has facilitated research into joint training of unified detectors for 2D perception tasks. However, 3D vision has not fully benefited from these advancements due to significant cross- dataset discrepancies. Thus, further exploration of multi-dataset training strategies in 3D perception tasks, particularly 3D object detection, is urgently needed. A direct approach to designing a unified 31) object detection framework for achieving multi-dataset training (MDT) involves merging multiple datasets and retraining the baseline detector on the merged dataset. However, significant domain gaps exist between 3D datasets, and directly combining multiple data sources can result negative transfer. Some efforts (Zhang et al. 2023) focused on 3D multi-dataset object detection have offered
"Equal Contribution. This work was done during Yubin Wang's internship at Baidu Inc 1 "orresponniig Author. Email: zhaocairong cerroong@ttogggii @tongi.edd..cn

---

Published as a conference paper at ICLR 2025
Dutasel
Detecter A
Hoad
Dutasel A
Detecter A
Head
Dutasel A
Point Range Alggnment
Ronge Alggnment
Partially Shored
Unifled Detector Head
Dataset B
Dataset
Delectio
Head
Dataset
(a) Single-Datase Training
(b) Naive Mult-DDaazset Training
(c) Unified and Universal Training (Ours)
Figure/ 1: Illustration of different training paradigms. Single-dataset training leverages separate detectors and heads for different datasets. Naive multi-dataset training conducts point range alignment and partially shares the parameters within detectors, but still with dataset-specific heads. We propose unified and universal training, where detectors and heads for different datasets are fully shared.
solutions for building unified training paradigm for point cloud data from different domains. As shown in Figure the overall framework is designed in datasct-specific manner, sharing certain backbone parameters while employing separate normalization and head layers for different datasets. Despite alleviating the unavoidable data-level differences to some extent, this independent paradigm suffers from two challenges: (i) this paradigm inhibits the full mutual utilization of each dataset's unique features, thereby constraining the further enhancement of the model's 's capabilities; (ii) the capacity for generalization to unseen domains is constrained due to the customization of certain network parameters specific to the trained dataset. There is scarcity of research on both refining the unified multi-dataset training paradigm and improving its generalization to other 3D datasets in more real-world scenarios. Our main goals include effectivelyy unifying the processing of diverse larger-scale point cloud data and ensuring robust generalization of the trained model to unseen domains. Compared to other works, we focus developing universal techniques for managing out-of-domain datasets. Accordingly, we define our universality as the ability of model to be jointly trained on of specific datasets and to perform zero-shot detection on new datasets using corresponding dataset attributes as prompts, without the need for re-training.
To achieve these goals, we propose Unified and Universal framework for 3D Detection (Uni2Det), which integrates multi-stage prompting modules applicable to any LiDAR dataset and various 3D object detection baselines used in autonomous driving. As shown Figure Ie), our approach enhances performance by learning parameters shared across datasets, referred to as unified and universal training. utilizes diverse prompts, such as intrinsic dataset attributes that are easily obtained from target datasets but cannot be automatically learned. Due to inherent discrepancies in large-scale 3D datasets, we perform point distribution correction during voxelization to learn unified point and voxel representations across datasets, centered mean- mean--hifted batch normalization. Furthermore, handling data with varying statistical distributions within the backbone remains a challenging problem. To mitigate variations in data distribution, particularly from the perspective of point range, we introduce BEV-based range masking that acts on BEV features. This approach provides prior signals for the 2D convolutional network, enabling it effectively handle point clouds from different datasets in a unified manner. Additionally, we observe that the same category exhibits statistical differences across datasets, which hinders the effectiveness of universal detection head to some extent. To this end, we learn object-con citect-condiiional residuals as prompts acting on each Rol feature, integrating features from pre-trae trained heads with new knowledge about the target domain. Our method does not simply combine disparate modules but should be regarded as an integrated system. allows different components of the 3D detection framework to leverage dataset-specific prior prompts, compensating for inter-dataset variations without requiring multiple branches. Benefiting from it, models can fully utilize diverse datasets for joint training, thereby improving in-domain detection performance. At the same time, the prior characteristies of unseen datasets can also be leveraged within a unified network as encoded prompts, enabling better out-of-domain generalization. Furthermore, this framework facilitates seamless plug-ant -od--nddplay integration within various advanced 3D detection frameworks while allowing direct adaptation for universal applicability across datasets.
Our main contributions consist of three parts:
2

---

Published as a conference paper at ICLR 2025
We introduce a novel training paradigm for 3D object detection which focuses on unified and universal
training, aiming at enhancing the performance in MDT settings.
We present Dni novel framework on 3D detection with multi-stage stage prompting modules for prompting various components in a detector including voxelization, backbone and head, enabling robust performance across diverse domains and generalization to unseen domains.
Experiments conducted across multiple dataset consolidation scenarios involving KITTI, Waymo, and nuScenes demonstrate that Dni significantly outperforms existing methods in multi-dataset training, especially on the generalization capability of the model.
2
RELATED WORKS
2.1 LIDAR- -BSED 3D OBJECT DETECTION
LiDAR-based 3D object detection aims to produce collection of 3D bounding boxes along with their associated object categories using LiDAR point cloud. Current LiDAR -baedd 3D object detection
research (Lang et al. 2019 Shi et al. 2020a 2019 2023 Yan et al. 2018 Shi et al.. 2020b can be broadly categorized into ppint-based methods, voxel- methods, and hybrid point- woxel- methods. Point- methods, such as Point-RCNy (Shi et al. (019) and 3DSSD Yang et al. 2020) generate feature maps directly from raw point clouds, thereby leveraging more accurate geometry information compared to previous methods. Unlike point-based methods, Voxel-based methods like VoxeINet (Zhou & Tuzel 2018) and SECOND (Yan et al., 2018 initially voxelize the input point cloud, transforming irregular LiDAR points into ordered voxels, and then extract features using 3D convolutions. PointPillars (Lang et al. 2019) encodes the input point cloud into pillars and employs 2D convolutions for feature extraction. Voxel-RCNN (Deng et al., 1021] analyzes the advantages of voxel features and explores a balanced trade-off between detection accuracy and inference speed. Additionally, some studies attempt to merge the advantages of point-based and voxel-based representations. PV-RCNN (Shi et al., 2020a) and PV-RCNN+ (Shi et al.) (023)) leverage both multi-scale 3D voxel CNN features and PointNet-basse features, consolidating them into a concise set of keypoints using newly proposed voxel set abstraction layer. Nevertheless, all the aforementioned detectors are trained and evaluated using separate 3D datasets, leading to significant degradation in detection accuracy when applied to other different datasets.
2.2 MULTI-DATASET TRAINING
In recent years, training on multiple diverse datasets has emerged as an effective strategy for en- hancing model robustness. Multi-dataset training has been previously investigated in the image domain, particularly tasks such as object detection (Zhou et al.] 2022] Wang et al. 2019) and image segmentation (Lambert et al., 2020). For perception tasks Dai et al.) 202 l Gong al., 2021; Zhao et al., 2020), dataset unification involves consolidating various semantic concepts. Early studies (Lambert et al. 2020; Zhao et al., 2020; Xu et al. 2020) have focused on merging taxonomy information and training models on a unified label space. MSeg (Lambert et al. 2020) manually unified the taxonomies of different semantic segmentation datasets and resolved inconsistent anno- tations between them. SiiverssllRCNN (Xu et al., 2020) trains a partitioned detector on multiple large datasets and modeled class relations using inter-dataset attention module. To reduce the annotation cost associated with unifying the label space, recent studies (Zhou et al., 2022 Wang al. 2019) have explored the use of dataset-specific supervision. Although joint training of unified detector has been studied in 2D perception tasks, further exploration in 3D perception tasks, such as 3D object detection, remains urgently needed. Uni3D (Zhang et al., 2023) attempts design a framework in dataset-specific manner, sharing certain backbone parameters while employing separate normalization and head layers for different datasets. PPT (Wu et al., 2024) proposes a novel framework for multi-dataset synergistic learning in 3D representation learning that supports multiple pre-training paradigms with prompt-driven normalization. However, its learning of dataset-specific prompts relies on an automatic approach that does not incorporate intrinsic characteristics of the dataset, which are not sufficiently universal since it requires new parameters for each new dataset. OneDet3D (Wang et al. 2024) seeks to achieve universality by integrating indoor and outdoor datasets into a unified framework, employing an open vpccbbocabllary strategy for object detection. Despite its merits, OneDer3D's experiments on outdoor datasets are limited exclusively to the car category rather
3

---

Published as a conference paper at ICLR 2025
Dataset
Multi-Stage Prompting Modules
ID
Datuset
Point Distribution
BFV-hased Range Masking
Objer-ccndiitooall Residual Learming
Correction
Discrimination Loss
Point Range A Algument
Voxelization
Delection Loss
Hoael Froture
BAY Feanure
Raf Featue
Poler
Backhone
Backbone
Head
Channel Concatemation
Flement-wise Addition
Dataset B
Figure 2: Illustration of the overall framework of Uni Det. The multi-stage prompting modules are employed as the core component to make the detection more unified and universal.
than utilizing a broader range of categories for both training and inference. To address these issues, we propose Det for 3D detection, which integrates multi-stage prompting modules applicable to any LiDAR dataset and various 3D detection baselines, enabling robust performance across domains and generalization to unseen domains. Our method leverages the dataset's 's inherent properties, such as point cloud distribution and range, which are easily obtainable other datasets and inherently more generalizable than other methods.
METHOD
The overall framework is shown in Figure We first describe our problem setting and the multi- dataset evaluation method in Sec. 3.1 Next, we introduce our multi-stage prompt learning modules for multi-dataset 3D detection, from various components in detectors including Voxelization in Sec. Backbone in Sec. 3.3 and Head in Sec. 3.4
3.2
3.1 PRELIMINARY
In the realm of 3D object detection, the task involves analyzing an input frame LiDAR points to predict associated labels, including categories and orientated bounding boxes. Training object detection model JF with its parameter θ on a single dataset typically involves straightforward approach: minimizing the 3D detection loss over set of point clouds and its corresponding ground truth y from the dataset D:
min (I
(I)
Suppose that dataset is characterized by joint probability distribution Pxy over the input point cloud and label space x y. In the scope of multi-dataset training (MDT), we possess N datasets {D. originating from diverse domains. Each D; is linked to a distinct data distribution P The goal of MDT is to utilize multiple labeled datasets for training a unified model F X V. various domains. One straightforward strategy entails merging all datasets into substantially larger one, denoted as Dmerge= D, "D2 U...U Dr.. While datasets may feature distinct label spaces, training and evaluation are limited to categories relevant to autonomous driving scenarios: vehicle, pedestrian, and cyclist. Consequently, the label space y can be shared across various domains. This approach optimizes the same loss function over the expanded dataset DD Deerge:
aiming for increased generalizability and minimized prediction errors
min (x.y)
[[(F(x;; 0),
In the following sections, we present the design of our Uni Det and show how to train a 3D perception model that performs well seen datasets and generalizes unseen datasets.
4

---

Published as a conference paper at ICLR 2025
3.2 PROMPT FOR VOXELIZATION; POINT DISTRIBUTION CORRECTION
The discrepancy in point distribution among different datasets poses a significant challenge in multi- dataset training. We use KITTI as an example. Since point features are typically related to coordinates in the ego-coordinate system, the feature channel corresponding to the x-axis consistently has values greater than in KITTI. As result, differences between KITTI and other datasets cause variations in the point cloud feature distribution, which in turn influences the training process across multiple datasets. To address data-level discrepancies in large-scale annotated 3D LiDAR datasets, we aim to develop simple modules during voxelization. These modules will enable existing 3D detectors to learn universal point and voxel representations across diverse datasets, as shown in TigureBaa)
Point representation learning Instead of relying on coordinates as point features, certain studies have explored effective methods of fusing information from various viewpoints. This is achieved through a learnable network incorporating a linear layer and batch normalization. However, in this approach, the batch normalization process does not account for MDT training scheme, where points within the batch come from frames in various datasets having large statistic differences. To address this, we introduce a new normalization approach termed "Mean-shifted batch normalization" to perform instance-level feature correction. Compatible with any 3D detectors, this method can alleviate statistical differences in features extracted by standard 2D or 3D backbones.
Mean-shifted batch normalization After the linear layer, we obtain a batch of point features represented as P pp....,, ......... feature, and N; represents the total number of points in the i-th frame. Conventional BN carries normalization across all frames (instances) under the assumption that all data follows the same distribution. However, two frames may exhibit disparate point ranges due to the different sensors in use and even if they come from the same sensor, the distribution of points may still be highly random. To address this, under the MDT setting, we argue that instance-level statistics are also crucial and introduce mhan-shifted batch normalization. Subsequently, samples from each dataset are regularized using the basic mean a with an adjustment from the current instance-specine mean p², as follows:
PNa. from M frames. Here, p denotes the j-th point
f D2J ap (1 a)j
Vo-E
where yi and a denote the channel-wise mean and variance of the feature set P. which are employed for conventional channel-wise feature normalization ensure the input data conforms to zero-mean and unit-variance, and is added to ensure numerical stability. Here we maintain the sharing of variance σ. a E [0, 1] is a balancing ratio for the shifted mean. When a O, it is equivalent to performing the regular BN operation, while a = 1. the normalization procedure disregards the basic mean and relies solely on instance-level statistics. The subsequent transformation step for pi remains the same as in conventional batch normalization. This approach allows us to learn universal point and voxel representations across diverse datasets with instance-level statistics as regularization. Compared with PPT (Wu et al.) 2024b and Uni3D (Zhang et al., 2023 3, our approach operates only at the voxelization stage, using levelevel point cloud distribution as a dataset attribute to prompt and regularize features. The advantage of feature uniformity ensures that the training of subsequent model parts is not negatively affected by feature differences between datasets.
3.3 PROMPT FOR BACKBONE: BEV-BASED RANGE MASKING
In the realm of modeling point clouds in the MDT setting, learning unified features from diverse sources and domains poses significant challenge due to variations the point range and data distribution across different datasets. To address this challenge, we introduce BEV-based range masking acting on BEV features to effectively handle point clouds from different datasets, as shown in Figure b).
Given the preconfigured point range (I1, 8/1, F2, V2) for producing BEV features with an aligned infer a binary mask for each dataset based on its
coordinate system, where I1 < x2, 1/1 < y2, point range. The purpose of this mask is to explicitly indicate whether the regions or grids on the BEV plane are inside the point range of the frame. Suppose H and W are the spatial shape of the
5

---

Published as a conference paper at ICLR 2025
Pronpp-eshenndd Frenee
Fester
roidaab
Mapping
Biowey Moi
Cancat
Promgicchanccl
Pater Foomrry
Cons
Faalore
Prong--cmaance
Batch-lndd mone
MT Franao
Pronghoskaweod
BET Fenture
(a) Point Distribution Conediin
(b) BEV-based Range Masking
(c) Ohject-codiiional Resicul Leaming
Figure 3: Illustration of multi-stage prompting modules, including three modules for prompting
different components of the detector.
BEV plane, and (#1- xg. v5) is the point range of the i-th dataset. We naturally map this point range the BEV plane based on the preconfigured point range according to the following equation:
1 (x] x1) HJ,9i
(vi 31) WJ,
(02 - H V222
(3) 91) W7.
vi
(4)
1/2 y1
32 I1
1/2 211
Having the mapped point range on the BEV plane (a vi' w0' w/ we can obtain mask M E R for the i-th dataset by:
M mnn = {
={
if S S Vi' S n S
(5)
0
if others.
Given frame of point clouds from the i-th dataset, our approach concatenates BEV features with the corresponding masks M' along the feature dimension before each 2D convolutional layer. Using this prior signal, the network can effectively adapt to point clouds from various datasets, avoiding excessive focus on the area outside the relevant regions, thereby maintaining the integrity of crucial information. This integration provides a solution for the unified backbone to model features across datasets, which not only preserves sppeset- information but also enhances the robustness and adaptability of feature modeling.
3.4 PROMPT FOR HEAD: OBJECT-CONDITIONA. RESIDUAL LEARNING
The prompting modules in previous stages facilitate the framework become more 'unified'. On the other hand, learning a general detection head is crucial in designing a 'universal' framework. Since previous works use multiple dataset-specific detection heads for prediction, such designs cannot be directly transferred to new datasets and therefore cannot be considered universal. In this section, we explore the potential of a universal detection head for predicting point clouds from diverse domains without dataset-specific branches. However, directly training detection head on multiple datasets poses challenges. As noted previous works, the same category exhibits statistical differences across datasets, motivating to design prompts to mitigate the distribution gap. Consequently, we introduce object-level residual learning, inspired by ( YYu et al.) 2023), on Rol features, integrating them from pre-trained heads with new knowledge about the target domain, shown in Figure 3(c)). Instead of learning set of object-agnostic task residuals, we argue that learning object-conditional residuals is more effective and transferable unseen domains prompts.
Given a batch of RoI features = { from frames of different datasets and their labels Y= = {y, | where y/1 = j if feature x; is from a frame of the j-th dataset, we feed each Rol feature z; into residual function f to obtain object-con residual r,. The generation process is formulated as r; fSG(zz))) where SG indicates the stop- noo-graddentt operation to prevent hindering the regular learning of Rol features. Since the generated residuals should be relevant to the domain

---

Published as a conference paper at ICLR 2025
or dataset to which the feature belongs, design discriminator D, implemented by an MLP, to distinguish these residuals, using the dataset ID as a prior label. The discrimination process is formulated as i D(r;). We use the cross-entropy loss to measure the discrimination loss Lass between the predicted label set Ý = { and the ground truth label set Y, which further added to the regular detection loss Ldet as the final loss. By learning such object-conditioaa residuals, we can enhance the original Rol feature with prior Kataset-specific characteristies by 2 2; + rss and models will tend to make predictions according to specific distribution, thus mitigating the influence of statistical differences in taxonomy across datasets.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets. Our experiments are conducted on three commonly used autonomous driving datasets: Waymo (Sun et al., 2020j, nuScenes (Caesar et al. 2020), and KITTI (Geiger et al. (012).
Waymo (Sun et al. 2020) stands out as the largest dataset with over 230,000 annotated 64-m
LiDAR frames gathered from six US cities. nuScenes (Caesar et al. (020)) comprises 28, 130 training
samples and 6,019 validation samples collected using 32-beam LiDAR. KITTI (Geiger al. 2012)
includes 7,481 annotated LiDAR frames collected via (4-beam LiDAR. These datasets exhibit varia- tions in data-level distributions arising from disparities in LiDAR types, geographic location of data acquisition, and variations in the definition of categorical annotations.
Implementation details The experiments are conducted using OpenPCDet (Team al. 2020 Particularly, we note that differences in point cloud range significantly degrade cross-dataset detection accuracy. Therefore, we align the point cloud range of all datasets to [75.2, 75. 75.2Jm for the X and Y axes and [2, 4]m for the Z-axis. In all experimental settings, we follow Uni3D (Zhang et al. 2023) and employ the standard optimization techniques utilized by PV-RCNN (Shi et al. 2020a) and VoxeIRCNN (Deng et al. 202T). For the balancing ratio in our proposed mean-shified batch normalization, we set a = 0.1 for VoxcIRCNN and a = 0.5 for PV-RCNN. This involves using the Adam optimizer with initial learning rate of 0.01 and implementing the OneCycle learning rate decay strategy. The network trained across 8 NVIDIA A800 GPUs, with a total training epoch set to 30. For the experiments is set to 0.01, while for Waymo-nuScenes consolidation, it is set to 0.001. We utilize only 20% of the uniformly sampled frames on Waymo dataset for model training.
Waymo-KITT) KITTI and nuScenes KITTI consolidations, the weight decay
Evaluation metric. We utilize the official tools to evaluate the performance of all baselines and our method, following (Zhagg et al., 2023). For Waymo, we use Average Precision (AP) and Average Precision re- ee-eeihttdd by Heading (APH) for each class, based on the LEVEL 1 metric. For KITTI and nuScenes, we report Average Precision (AP) in both Bird's Eye View (BEV) and 3D over 40 recall positions, with moderate case results for KITTI. AP is evaluated with an IoU threshold of 0.7 for the car category (Vehicle on Waymo) and 0.5 for pedestrian and cyclist categories. All experimental results presented in this paper reported on the official validation set.
4.2 RESULTS OF ZERO-SHOT EVALUATION ON UNSEEN DATASETS
To comprehensively assess the universal design of our framework, we perform a zero-shot evaluation on several unseen 3D datasets using different detection models. Our approach. Uni is compared against two main baselines. First, we benchmark it against the "Source Only" model, which only uses the source domain for training, and against a robust generalization method, SN (Wang et al.) 2020), within the single-source generalization setting. Additionally, we extend the comparison to the dual-source generalization scenario, evaluating Uni against simple data-merging strategies and Uni3D, along with its variants. We also attempt to integrate SN into our method with extra statistical supervision on the target domain. As shown Table our Uni is proved to achieve more generalized representations SN. This is because our universal framework effectively utilizes the prior information associated with target datasets as perform better adaptation. For the dual-source generalization, discover the trend that using a single detection head in Uni3D yields superior generalization performance compared to employing multiple detection heads. This suggests that training the detection head
single dataset as well, further enhancing performance based on
7

---

Published as a conference paper at ICLR 2025
Table Results of zero-shot evaluation on unseen datasets. We report the results of the Car (Vehicle on Waymo) category under the IoU threshold of 0.7 and utilize A AEBev/AP3D over 40 recall positions on KITTI. Source Only denotes that the model trained on the source domain and directly tested on the target domain. S.H. for Uni3D (Zhang et al. (223) indicates using single head instead of dataset-specific heads as variant.
Single-Source Generalization
Dual-Source Generalization
Waymo KITTI
Waymo + nuScenes KITTI
Method
Method
Detector
mAP
Detector
mAP
Source Only
PV-RCNN
61.18/ / 22.01
Data Merging
PV-RCNN
69.07 36,17
SN
PV-RCNN
09.92/60.J1
Data Merging (w/ SN)
PV-RCNN
72.43 79.99
Uni-Det (w/SN)
PV-RCNN
72.41 /33,96
Uni3D
PV-RCNN
37.83
Source Only
Voxel-RCNN 64.88 / 19.90
Uni3D (w/ M.H)
PV-RCNN
$1.79/38.88
SN
VoxcI-RCNN
55.83/55.50
Uni3D (w/ S.H., SN)
PV-RCNN
73.48 60.51
Uni Det (w/ SN)
VoxeI-RCNN
76.34/57.85
Uni Det
PV-RCNN
72.39/40.12
nuScenes
+ KITTI
Uni Det (W/SN)
PV-RCNN
75.57 / 64.09
Method
Detector
mAP
Data Merging
VoxcI-RCNN
69.02 / 36.57
Source Only
PV-RCNN
68.15 37.17
Data Merging SN)
VoNeI-RCNN
72.32 52,94
SN
PV-RCNN
60.48 / 49.47 447
Uni3D
VoNeI-RCNN
72.68 / 39.65
Uni Det (w/ SN)
PV-RCNN
66.75 / 55.43
Uni3D (w/ M.H)
VoxcI-RCNN
73.12/
Source Only
VoxcI-RCNN
69.41 /3.48
Uni3D (w/ S.H., SN)
VoxcI-RCNN
75.69) /3344
SN
VoxeI-RCNN
67.05 / /8.06
Dni
VoXel-RCNN
74.07 / 43.76
Uni Det (w/ SN)
VoxeI-RCNN 71.02 50.94
Uni Det (w/ SN)
VoxeI-RCNN
78.63 58.24
across multiple domains enhances its generalization ability and mitigates overfitting to any single domain. Building on this insight, our Uni framework employs multi-stage prompts enable more unified and universal training, which not only boosts zero-shot generalization performance but also maintains competitive edge in in-domain tasks. By comparing the overall results, we confirm that Det significantly improves generalization performance when incorporating new datasets, demonstrating its potential for robust 3D detection across varying domains.
4.3 RESULTS OF MULTI- DATASE 3D OBJECT DETECTION
To evaluate the unified design of our framework, we conduct experiments on the two-dataset combi- nation of three widely-used autonomous driving datasets: Waymo (Sun et al. 2020), KITTI (Geiger ct al. 22012), and nuScenes (Caesar et al.) 2020), and report results in Table 2 with comparison to basclines demonstrated in (Zhang et al. 2023). We summarize our findings in three points.
Dni Firstly, performance improvement from multi-dataset training is guaranteed for Uni Det. In some cases, the previous state-of state-of-theeart Uni3D shows worse results under multi-dataset training than when trained single dataset (e.g., results on Waymo under Waymo-nuScene consolidation), indicating that Uni3D did not fully and effectively utilize data from multiple datasets for training. In contrast, 2 Det avoids this issue and ensures improved performance with additional datasets for training, demonstrating excellent scalability.
Secondly, dataset-agnostic head is feasible instead of the dataset-specific head. Although the improved results of Uni3D compared to simply merging datasets demonstrate the benefits of learning dataset-specific detection heads, our work addresses the poor performance issue with single detection head through unified paradigm, proving the feasibility of learning aggostt- detection head. Using more training data from different datasets to train a single detection head in our unified paradigm is more likely to enhance detection performance.
Lastly, Uni Det considered more robust and unified framework for multi-dataset training. Across all dataset combinations, our Dni consistently outperforms Uni3D, demonstrating the effective- ness of our approach in a multi-dataset setting. This result also indicates that our unified training paradigm is stable and robust, capable of converting point cloud data from any source domains or datasets into a more unified distribution through prompts for better prediction. However, our method shows lower AP when inferring some categories (e.g., results for Car on nuScenes under KITTI-nuScenes consolidation). Despite this, considering the boost from other categories within the dataset, it maintains overall improvement across each dataset.
8

---

Published as a conference paper at ICLR 2025
Table 2: Results of joint training on different dataset consolidations. Following Uni3D Zhang et al. 2023), we report the car (Vehicle on Waymo), pedestrian, and cyclist results under the IoU threshold of 0.7, 0.5, and 0.5, respectively, and utilize AP/APH of LEVEL metric on Waymo, and AP over 40 recall positions on nuScenes and KITTI. P.T. indicates pre-training the baseline detector on the other dataset, and fine-tune the detector on the current dataset. D.M. stands for simply merging data from different datasets as the training data. The best detection results are marked using bold. Here we adopt Voxel-RCNN as the baseline detector. Due to the page limitation, present the experimental results based on PV-RCNN in Appendix.
Wayme-nnSennes Consolidation
Trained on
Method
Tosted
Waymo
on
Vehicle
Pedestrian
Cyclist
Car
Podestrian
Cyclist
Waymo
w/ P.T.
95,46/74.99
74.58/68.06
65,92/64.98
71.99/69.34
34.34/21.55
2.84/1.57
0.09/0:02
12.42/7.85
nuSicenes
w/ P.T.
6.11/5.90
0.73/0.56
0.01/0.01
2.30/2.16
55.23/39.14 14
23.65/16.47
8.51/5.80
13/20.44
D.M.
66.67/66.23
60.36/54.08
52.03/51.25
59.69/57119
51,40336688
15:04/999
5,40/3.87
23.95/15.18
Both W&N
Uni3D
75. 75.26/74.77
75.46/68775 75
65.02/63.12
71.91/68888
60. 18/42.23
30,08/24.37
14.60/1.32
34.95/26.31
Det
76.13/75.66
77,27/71.44
66.40/65.46
73.27/70.99
60.26/41844
31.17/25.31
17.17/14.42
36.20/27.19
KITTI-nuSeenes Consolidation
Tested on KITTI
Tested on nuScenes
Trained
Method
Car
Pedestrian
Cyclist
mAP
Car
Pedestrran
Cyclist
mAP
KITTI
w/ PT.
89.90/81
59,49156.17
54,35/54.15
67,9863.88
12,89/5.50
0,24/0.18
0,08/0.03
4,39/1.91
nuSoenes
71.61/40.64
39.67/29.99
7,29/6.88
39.52/25.88
53,57/39.65
24.9321117
11,42/9:55
29.97/23.59
D.M.
89.24/73.72
61,03/54.55
62.71/59.92
70,99/62.73
48.88/20448
12,58/8.3
1,77/0.97
18.74/9.92
Both K.N.N
Uni3D
10.09/83.10
62.99/58.30
70.20/68.10
74.43/69.83 83
59.25/41.51
29.12/23.18
15.16/1.116
34.51/25.95
Det
90.60/84.16
68.40/64.47
68.74/65.68
75.91/71.44
58.09/39.68
31,1025.83
20.56/17.53
36.58/2..88
KITTI- Waymo Consolidation
Trained on
Method
Tested on KITTI
Tested
Waymo
Cur
Pedestrian
Cyclist
mAP
Vehicle
Pedestrian
Cyclist
mAP
KITTI Wayme
w/ P.T.
89.51/81.41
60.30/57.10
55.53/51.34
68.45/63.28
8.70/8.62
19.14/16.01
21.87/20.83
16.57/15.15
w/ P.T.
64.84/19.99
62.58/59.01
56.44/49.43
61,29/4281
72.76/72.26
72.42/64.94
63,27/62.23
69.48/66.4 48
D.M.
74.53/32.11
60.11/54.85
59, 59.69/55.94
61.78847.63
74.35/73.85
74.80668.39 39
61.87/63.95
71.34/68.73
Both K&W
Uni3D
90.03/82.39
62.51/57.01
69.52/66.30
74.02/68.57
74.83/74.33
74.79168.24
66.83/65.82
72.15/69 .46
Det
90.30/84.23
64.30/61.03
71.15/69.18
75.25/71.48
75.35/74.77
76.64/71.22
67.03/65.73
73.01/70.57
Table 3: Results jointly training the Voxel- RCNN on three datasets.
Table 4: Ablation study of prompts in different
stages of Uni'Det based on VoxeI-RCNN.
Tosted
Method
Vexelization
Backhone
Head
KITTI
Waymo
Trained on
KITTI
NaSveens
Waymo
Avg.
Baselinc
72.73/69 94
71.0969.11 12
KITTI
70.01/66.09
3.84/1.58
12.69/1.44
28.36/26.38
73.84/70.56
71.71/89.33
muScenes
32.64/17.70
28.99/2.20
14.63/1414
42/17.98
73.43/70.19
71.52/69.43
Waymo
64,0044..77
12,58/634
71. 71.84/99.25
49.41/40.28
Ours
73.95/70.43
71,73/69.82
UniSD
72. 22.9967.46
25.06/26.48
71.9889.28
59.73/54.41
74.96/70.95
72.29/69. 93
Unf Det
76.04/72.61
34.03/25.44
72.45770.20
60.84/56.1N
75.25/7L.48
73.01/70.57
4.4 FURTHER ANALYSIS
Results on Waymo-KITTI-nuSeee Consolidations. TaalleBshows the results of jointly training VoxeI-RCNN on Mayyo--KITT-nnuseeeee consolidations. We report average AP over all categories within the dataset. Our Uni2 demonstrates high detection results on KITTI and Waymo, on which the results of Uni3D do not differ much from those on the single dataset. Overall, more balanced and consistent boosts on different datasets can be observed in U on average AP BEV and AP3D-
Ablation on prompts in different stages. We investigate the influence of prompts in different stages of Uni"Det, including voxelization, backbone, and head. The evaluation setting follows the KITTI- Waymo consolidation in Table 2 As shown in Table 4 the prompt modules implemented at each stage enhance performance compared to the baseline. Notably, the point distribution correction at the voxelization stage shows the most significant improvement among all stages. This demonstrates that learning unified low-level point features is crucial for MDT and serves as foundation for subsequent prompting stages. We also observe that gradually adding prompts wageeby-sttage results in noticeable performance gains for each stage, reflecting the complementarity between the prompting modules. At last, using prompts from all stages together achieves a significant improvement compared to the baseline.

---

Published as a conference paper at ICLR 2025
Width
Length
Height
2.4-
1.8-
2.2
1.6
1.8
1.6
1.4
1.2
Ground Truth Ours twie OCRLI Ours Ow OCRLI
Ground Truth Ours (wio OCRLI Ours twi OCRL)
Ground Iruth
OCNL) Ours (nd OCHL)
Figure 4: Illustration of statistical distribution differences of object size (length, width, and height) in KITTI between the ground truth and the predictions of Uniè with and without object-conditional residual learning (OCRL) module.
Ablation on object-conditioal residual learning. We investigate the statistical distribution differences of object size in KITTI between the ground truth and the predictions of Uni2 with and without object-conditions evident that our method. particularly with the OCRL module, more closely approximates the ground truth distribution, especially in terms of mean and variance, thereby reducing statistical differences across datasets and mitigating potential performance degradation. However, this module may also misidentify the discriminative domain for samples from to ambiguous residuals that may cause inaccurate bounding box predictions.
residual learning (OCRL) module. As shown in Figure 4 it is
scenes in certain cases, leading
Ablation on normalization strategies. We conduct abla- tion studies of normalization strategies in PPT (Wu al.] 2024). Uni3D) (Zhang et al. 2023) and our Uni on the Soyyoo-KITII consolidation based on VoxeI-RCNN, as shown in Table 5 There are no other well-designed modules and only with single detection head for fair comparison. Our proposed mean-shifted batch normalization is more ad- vantageous, especially on KITTI where only the front view is provided, and it is important to note that the other two com- pared methods are not conducive to generalization scenarios, which further highlights strengths.
Table 5: Ablation normalization strategies based on VoxeI-RCNN.
Method
KITTI
Waymo
Baseline
72.73/69.94
71.09/69.1
Uni3D
73.01/70.12
71.45/69.23
PPT
73.25/70.09
71.73/69.46
Uni
73.84/70.56
71.71/69.73
CONCLUSION
We introduce Det, novel framework designed for unified and universal multi-datase training in 3D detection which utilizes multi-stage prompting modules to harmonize differences between datasets by leveraging sppesit- characteristics, ensuring robust performance across various domains and effective generalization to new domains. Our work is promising to enhance performance across various domains and facilitate effective generalization to new ones 3D detection, which has the potential advance fields like autonomous driving.
Limitation and discussion. We identify a limitation in our approach, which relies on the assumption of identical set of categories across different datasets. This limitation impedes the detection of diverse label spaces with more varied categories. We propose that refining categories by subdividing objects based on size to create unified label space or mapping unseen classes existing categories using prior knowledge (e.g.. object size) both facilitate zero-shot generalization. Additionally, open- vocabulary 3D detection, enhanced by textual prompts, can expand the range of detectable categories. We expect that future research building on our work will contribute to the development of unified, universal 3D detection systems, Moreover, our method has not yet been applied to indoor datasets, and we aim to make it more versatile and effective across diverse applications.
ACKNOWLEDGMENT
This work supported by National Natural Science Fund of China (No.62473286).
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11621-11631, 2020.
Xiaozi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pp. 1907-1915, 2017.
Xiyang Dai, Yinpeng Chen, Bin Xiao, Dongdong Chen, Mengchen Liu, Lu Yuan, and Lei Zhang. Dynamic head: Unifying object detection heads with attentions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7373-7382, 2021.
Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li. Voxel r-cnn: Towards high performance voxel-based 3d object detection. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 1201-1209, 2021.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference computer vision and pattern recognition, pp. 3354-3361. IEEE, 2012.
Rui Gong,
Dengxin Dai, Yuhua Chen, Wen Li, and Luc Van Gool. mdalu: Multi-source domain adaptation and label unification with partial datasets. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8876-8885, 2021.
Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengehao Wang, Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, et al. Self-supervised pretraining of visual features in the wild. arXiv preprint arXiv:2103.01983 2021.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015 2023.
John Lambert, Zhuang Liu, Ozan Sener, James Hays, and Vladlen Koltun. Mseg: A composite dataset for multi-domain semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2879-2888, 2020.
Alex H Lang. Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Point- pillars: Fast encoders for object detection from point clouds. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12697-12705, 2019.
Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrenn: 3d object proposal generation and detection from point cloud. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 770-779, 2019.
Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang. Jianping Shi, Xiaogang Wang. and Hongsheng
Li. Pv-renn: Point-voxel feature set abstraction for 3d object detection. In Proceedingy of the IEFF/CVF conference on computer vision and patlern recognition, pp. 10529-10538, 2020a.
Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. From points parts: 3d object detection from point cloud with part-aware and part-aggregation network. IEEE transactions pattern analysis and machine intelligence, 43(8))2647-2664, 2020b.
Shaoshuai Shi, Li Jiang, Jiajun Deng, Zhe Wang, Chaoxu Guo, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-renn++: Point-voxel feature set abstraction with local vector representation for 3d object detection. International Journal of Computer Vision, 131(2)531-551. 2023.
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2446-2454, 2020.
11

---

Published as a conference paper at ICLR 2025
OD Team et al. Openpedet: An open-source toolbox for 3d object detection from point clouds, 2020.
Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6830-6839, 2023.
Xudong Wang, Zhaowei Cai, Dashan Gao, and Nuno Vasconcelos. Towards universal object detection by domain attention. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7289-7298, 2019.
Yan Wang, Xiangyu Chen, Yurong You, Li Erran Li, Bharath Hariharan, Mark Camphell, Kilian Q Weinberger, and Wei-Lun Chao. Train in germany, test in the usa: Making 3d object detectors gen- eralize. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11713-11723, 2020.
Yue Wang. Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In Conference on Robot Learning, pp. 181-191. PMLR, 2022.
Zhenyu Wang, Yali Li. Hengshuang Zhao, and Shengjin Wang. One for all: Multi-domain joint training for point cloud based 3d object detection. arXiv preprint vrXiv:2411.01558 2024.
Yi Wei, Zibu Wei, Yongming Rao, Jiaxin Li, Zhou, and Jiwen Lu. Lidar distillation: Bridging the beam-induced domain gap for 3d object detection. In European Conference on Computer Vision, pp. 195- Springer, 2022.
Xiaoyang Wu, Zhuotao Tian, Xin Wen, Bohao Peng, Xihui Liu, Kaicheng Yu, and Hengshuang Zhao. Towards large-scale 3d representation learning with multi-dataset point prompt training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19551-19562, 2024.
Hang Xu, Linpu Fang, Xiaodan Liang, Wenxiong Kang, and Zhenguo Li. Universal-renn: Universal object detector via transferable graph r-cnn. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 12492-12499, 2020.
Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18 (10):3337, 2018
Jihan Yang. Shaoshuai Shi, Zhe Wang, Hongsheng Li, and Xiaojuan Qi. St3d: Self-training for unsu-
pervised domain adaptation on 3d object detection. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 10368- 2021.
Jihan Yang, Shaoshuai Shi, Zhe Wang, Hongsheng Li, and Xiaojuan Qi. St3d++: Denoised self- training for unsupervised domain adaptation on 3d object detection. IEEE transactions on pattern analysis and machine intelligence, 45(5):6354-6377 63711 2022.
Zetong Yang. Yanan Sun, Shu Liu, and Jiaya Jia. 3dssd: Point-based 3d single stage object detector. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 11040-11048, 2020.
Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11784-11799 1193, 2021.
Tao Yu, Zhihe Lu, Xin Jin, Zhibo Chen, and Xinchao Wang. Task residual for tuning dision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10899-10909, 2023.
Bo Zhang, Jiakang Yuan, Botian Shi, Tao Chen, Yikang Li, and Yu Qiao. Uni3d: A unified bascline for multi-dataset 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9253-9262, 2023.
12

---

Published as a conference paper at ICLR 2025
Xiangyun Zhao, Samuel Schulter, Gaurav Sharma, Yi-Hsuan Tsai, Manmohan Chandraker, and Ying Wu. Object detection with unified label space from multiple datasets. Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part XIV 16, pp. 178-193. Springer, 2020.
Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Simple multi-dataset detection. In Proceed- ings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 7571-7580, 2022.
Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4490-4499, 2018.
13

---

Published as a conference paper at ICLR 2025
APPENDIX
A
EXTRA EXPERIMENTS
Results of multi-dataset 3D object detection based on PV-RCNN. We present the experimental results for the two-dataset combination of three autonomous driving datasets using PV-RCNN, as shown in T The conclusions are consistent with those in Table further demonstrating that Uni is a more robust and unified framework for multi-dataset training.
Table 6: Results of joint training on different dataset consolidations based on PV-RCNN. The experiment and evaluation settings follow Table
Waymo-mmSeenes Consolidation
Trained on
Method
Tosted on Waymo
Tested on muSocacs
Vohiclo
Podestrian
Cyclist
mAP
Car
Podestrian
Cyclist
mAP
Waymo
w/ WTP..
74.77/74.26
73,32/66.31
64.06/63.05
70.72/67.87
33.86/17.47
2.88/1.53
0,040.011
12.26/6.34
ouSeenes
w/ PT.
44.59/44.24
7.67/6.33
8.77/8.58
20.34/19.72
57.92/41.53
24.32/17.31
11,52/9.19
31.25/22.68
D.M.
66.22/65.75
55,41449.29
56,5055.48
59.38/56.44
48.67/30.43
12.66/8.J
1.67/L04
21.00/13,20
Both W&N
Uni3D
74.12/66.90
63,28/62.12
70.98/67.97
60.77/42.66
27.44/2188
33.90/25446
Det
76.03/5.53
76.24/70.29
64,97/63.95
72.41/69.92
61.38/42.76
28.60/22.49
15.10/12.90
35.03/26005
KIT KITTe-nuSenns Consolidation
Trained on
Method
Tested
KITTI
Tested nuScenes
Car
Pedestrian
Cyclist
mAP
Car
Pedesttian
Cyclist
mAP
KITTI
w/ P.T.
89.26/83.14 14
60.56555.90
63.6066.88
14/6733
13.43/5.61
0.69/0.22 27
0.04/0.00
4.72/1.96
nuSccmess
w/ P.T.
09.40038.25
33,24/24.88
1.68/1.61
34.77/21.58
53.24/36.72
20.65/17.09
8.95/7.58
27.61/20.46
D.M.
87.79/77.95
55.32/48.22 29
59.15/55.10
67.49/60.5 45
41 41.29/21.57
10.21/7.00 .08
1.23/1.15
17,58/9.93
Both K&N
Uni3D
89.77/85.49
60.03/5.558
69.03/66.10
72.94/69.09
59.08/41.67
25.27/19:26
12.2K/10.83
32,20/23.92
Det
90.52/85.36
61.73/58.53
71.76/69.29
74.67/71.06
58.30X41.21
29.11/24.00
12.62/10.93
33.34/25.38
KITTI-Waymo Consolidation
Trained on
Method
on
Waymo
Podestrian
Cyclist
Podestrian
Cyclist
KITTI
w/ P.T.
3.240083.42
62, 022,69/58.00
59.9K/59.43
70.08/67.24
8.75/8.64
12.12/9.90
9.200% 76
10.02%6.10
Waymo
w/ P.T.
00.25/25.00
10,16/55.00
56.09/50.50
10/40/4..11
71.08/70.58 54
70.12/62.91
62.37/61
67. 77886/64.95
D.M.
87.49/68.35
62.84/60.06
68.09/65.75
72.81/64.72
50.68/50.31
58.76/52.59
15,14/54,17
54,86/52.36
Both K.W.W
Uni3D
89.42/83.15
60.85/57.49
71.61/65.88
73.96/68.84
75.07/74.54
72.95/66.08
63.8 63,80/62.92
70.61/67.85
Dni
90.70/84.65
61,02/58.33
72.86/71.26
74.86/71.41
75.43/74.92
74.96/69.20
65,57/64.49
71.99/69.54
Ablation the balancing ratio. We conduct ablation studies regarding the hyperparameter balancing ratio a based on different 3D detectors, based on AP3D metrics of KITTI on the Waymo- KITTI consolidation,
shown in Figure [ When 0.1 for PV-RCNN and VoxeI-RCNN, the performance can be optimal. Since VoxeI-RCNN mainly relies on voxelized point cloud data and is more sensitive the distribution of point clouds, the balancing ratio for point feature correction needs to be larger.
0.5 for
PV-RIN
75.8
75.5
Balancing
Figure 5: Ablation the balancing ratio based PV-RCNN and Voxel-RCNN, based on AP3 metrics of KITTI on Waymo-KITTI consolidation.
14


Reviews:
[{"review_id": "QF3vYYLLn8", "reviewer": "Reviewer_AxYi", "summary": "This paper addresses a critical challenge in autonomous driving research: integrating multiple datasets collected across different locations and using varied sensor types, especially LiDAR-based detection datasets. Due to differences in distribution, such datasets cannot be directly combined for training without significant performance trade-offs. The authors propose a novel, multi-stage prompting approach to effectively leverage the unique characteristics of each dataset and mitigate modality disparities, ultimately enabling a more robust multi-dataset training framework. This multi-stage approach is designed as a plug-and-play module that enhances several advanced 3D detection frameworks by targeting three stages: voxelization, BEV feature processing, and the detection head. Experimental results substantiate that the proposed multi-dataset training technique significantly improves cross-domain performance relative to baseline models, showcasing its effectiveness in harnessing diverse datasets for training.", "questions": "Check weaknesses", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper tackles a practical issue in autonomous driving, where dataset diversity can restrict the ability to scale training. By addressing how to integrate datasets with different characteristics, the research directly benefits the community by opening up avenues for broader dataset utilization, potentially enhancing the generalization of autonomous driving models.\n- The proposed approach is modular, designed as a plug-and-play component compatible with various advanced 3D detectors.\n- The use of frame-specific mean shifts for batch normalization is particularly novel. I’m curious whether this concept has been explored in existing literature or if it is an original contribution by the authors.\n- The paper includes comprehensive experiments to validate its claims, showing that: (1) the method outperforms baselines like Uni3D, (2) it can be integrated with various 3D detectors, (3) each of the three prompting stages contributes to the improved performance, and (4) OCRL improves the alignment of object size distribution with ground truth data.\n- The paper is well-organized and easy to follow, with clear figures. However, providing a list of all abbreviations in figure captions would enhance clarity.", "weaknesses": "While the paper is methodologically sound and well-supported by experiments, there are a few areas that could benefit from additional clarity and detail:\n- For mean-shifted BN: \n1) The authors claim that mean-shifted BN introduces dataset-specific characteristics, yet the parameter \\alpha appears to capture only frame-specific characteristics rather than broader dataset-level traits. A more intuitive approach might involve defining dataset-specific means and variances to capture the unique properties of each dataset more accurately. I suggest the authors discuss this alternative and clarify how frame-level mean shifts contribute to dataset-level adaptability.\n2) It remains unclear how the parameter \\alpha is specified within the model. Further explanation of its determination, whether it is learned, fixed, or computed dynamically, would help readers understand its role in adapting the batch normalization to different datasets.\n- For the BEV-based range masking component, the authors choose to concatenate the mask as an additional channel rather than directly masking out unwanted regions in the BEV representation. It would be valuable for the authors to provide a rationale for this choice. Specifically, how does the inclusion of the mask as a separate channel improve model performance or facilitate feature extraction compared to direct masking? A discussion on the impact of this approach on model interpretability and cross-domain generalization would also be insightful.\n- More discussion about the limitation when applying the approach to the real-world applications will be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0o9yEauo7c", "reviewer": "Reviewer_sQWH", "summary": "The paper introduces Uni2Det, a framework designed to tackle the challenges of 3D object detection across multiple datasets. By leveraging multi-stage prompting modules, Uni2Det aims to overcome data distribution discrepancies and enhance generalization capabilities. The framework is evaluated on datasets such as KITTI, Waymo, and nuScenes, demonstrating superior performance compared to existing methods.", "questions": "(1) Sensor Variability: \n\nHow does Uni2Det perform when dealing with datasets that utilize sensors with significantly different specifications? Understanding this could provide insights into its robustness across various sensor setups. For example, how is the performance on indoor datasets like Scannet [2] or SUN-RGBD [1]. \n\n(2) Extension to Diverse Categories: \n\nIs it possible to extend the framework to handle datasets with richer categories, like some open-vocabulary 3d object detection works [3, 4]? This would enhance its applicability to a wider range of scenarios. Please provide discussions in the paper.\n\n(3) Efficiency Considerations: \n\nWhat are the computational overhead and latency introduced by the multi-stage prompting modules? Providing metrics on efficiency would be beneficial for assessing practical deployment scenarios.\n\n(4) Failure Case Analysis: \n\nCan the authors provide examples or analysis of scenarios where the framework might underperform? This would help in understanding its limitations and potential areas for improvement. Please include failure cases in the paper.\n\n[1] Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In CVPR, 2015.\n\n[2] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017.\n\n[3] Y. Lu, C. Xu, X. Wei, X. Xie, M. Tomizuka, K. Keutzer, and S. Zhang, “Open-vocabulary point-cloud object detection without 3d annotation,” CVPR, 2023.\n\n[4] Y. Cao, Y. Zeng, H. Xu, and D. Xu, “Coda: Collaborative novel box discovery and cross-modal alignment for open-vocabulary 3d object detection,” in NeurIPS, 2023.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) Innovative Techniques: \n\nThe introduction of multi-stage prompting modules is a novel approach to address the differences in data distribution among various datasets, potentially setting a new standard for unified 3D detection frameworks.\n\n\n(2) Comprehensive Experiments: \n\nThe authors conducted extensive experiments across multiple prominent datasets, showing significant improvements in both multi-dataset training scenarios and zero-shot transfer performance. This thorough evaluation strengthens the validity of the proposed method.\n\n(3) Robust Generalization: \n\nThe framework shows impressive zero-shot capabilities, effectively adapting to unseen datasets without additional retraining. This highlights its potential for real-world applications where models need to perform well across diverse environments.\n\n(4) Scalability: \n\nUni2Det demonstrates improved scalability by effectively utilizing additional datasets for training, ensuring performance boosts without the pitfalls of negative transfer that other methods might suffer from.", "weaknesses": "(1) Complexity and Implementation: \n\nThe multi-stage prompting modules, while innovative, may introduce additional complexity to the framework. This could potentially affect the ease of implementation and deployment in practical scenarios, requiring more resources and expertise.\n\n(2) Assumption of Identical Categories: \n\nThe framework relies on the assumption that datasets share identical categories, which limits its applicability to datasets with diverse or non-overlapping label spaces. This constraint could hinder the adoption of Uni2Det in broader applications.\n\n(3) Limited Component Analysis: \n\nThe paper could benefit from a more detailed analysis of how each component of the framework contributes to overall performance. Understanding the impact of each module could provide deeper insights and guide future improvements.\n\n(4) Computational Overhead: \n\nThe potential increase in computational requirements due to the additional modules is not thoroughly discussed. This could be a concern for deploying the framework in resource-constrained environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cy31YK7XXA", "reviewer": "Reviewer_C7SQ", "summary": "This paper proposes a multi-stage prompting pipeline to use multiple data sources to train one 3D detector. Specifically, they performed mean-shifted batch normalization, applied BEV-based masking, and trained a discriminator on object-conditioned residual using classification loss. The object-conditioned residual is added to the RoI feature for prediction.", "questions": "1. In mean-shifted batch normalization, it would be helpful to further explain how to calculate the instance-specific mean. Additionally, it will be wonderful to compare the proposed method with an alternate approach of first de-mean data using instance-specific mean, following batch-wise demeaning without alpha weighting. Further more, what is the influence of completely discarding this stage, only using the backbone and the head?\n2. In object-conditional residual learning, how is the dataset id encoded in training? Also, during testing on zero-shot datasets, does this pipeline obtain object-conditional residual before prediction? \n3. In testing, it will be interesting to see how the model performs on classes that do not intersect with existing class vocabulary. At the same time, applying this method to datasets with various scenes, i.e., indoor 3D detection datasets, could provide a broader perspective on its robustness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper is easy to understand and has a clear structure.\n2. De-meaning data across batches and within one instance is an interesting way to improve generalization. Using dataset-relevant information as prompts is also very interesting.\n3. The performance seems good.", "weaknesses": "There are too few classes for evaluation, and the class domain is limited to autonomous driving scenarios. The authors mentioned this in section 3.1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AZrAoaC1ma", "reviewer": "Reviewer_dmVZ", "summary": "In this work, the authors propose Uni$^2$Det, a prompt-guided multi-dataset 3D object detection framework. In detail, the authors handle multi-dataset 3DOD from three perspectives (*i.e.*, voxelization, bev features, and detection head). First, the designed point distribution correctness uses instance- and batch-level normalization to deal with the distribution shift in different datasets. Further, the bev-based range masking can select features of corresponding areas of the different datasets. Finally, the authors propose object-conditioned residual learning to inject dataset-specific features into each instance to further decode the bounding boxes. Experiments show that the proposed method can achieve higher results on both in-domain and out-of-domain tasks.", "questions": "- I know MDF may represent multi-dataset fusion. However, the authors use MDT in the preliminary and then use MDF in the following section. This needs to be modified to avoid confusion.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Experiments show that the proposed Uni$^2$Det can achieve SOTA performance on both cross-domain and multi-dataset training settings.\n- The authors treat dataset-specific attributes as prompts which is quite interesting.", "weaknesses": "1. My first question is about the proposed BEV-based range masking. I wonder about the masking strategy during the inference phase. Do we need to know the dataset-specific point cloud range before evaluation? If so, what will happen when we do not know the point cloud range? Besides, in the domain generalization experiments, if we use the point cloud range of the target domain, it seems unfair since the domain-specific knowledge has already been injected into the model.\n2. Further, I wonder about the performance using more recent 3D detection models. I fully understand that the authors chose PV-RCNN and Voxel R-CNN for a fair comparison. However, they have been proposed for several years.\n3. Questions about experimental settings, especially hyperparameters. I need to know how the authors define the value of hyperparameters such as $\\alpha$, there seem no ablation studies about this.\n4. The motivation of point mean-shift normalization and object-conditional residual learning is similar to the data-level correction and semantic-level coupling and recoupling. Both of them first handle data-level shifts and then strengthen domain-specific features. The authors need to explain the advantages of the proposed methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)