Uni$^2$Det: Unified and Universal Framework for Prompt-Guided Multi-dataset 3D Detection

## Abstract

Abstract We present Uni 2 Det, a brand new framework for unified and universal multi-dataset training on 3D detection, enabling robust performance across diverse domains and generalization to unseen domains.
Due to substantial disparities in data distribution and variations in taxonomy across diverse domains, training such a detector by simply merging datasets poses a significant challenge.
Motivated by this observation, we introduce multi-stage prompting modules for multi-dataset 3D detection, which leverages prompts based on the characteristics of corresponding datasets to mitigate existing differences.
This elegant design facilitates seamless plug-and-play integration within various advanced 3D detection frameworks in a unified manner, while also allowing straightforward adaptation for universal applicability across datasets.
Experiments are conducted across multiple dataset consolidation scenarios involving KITTI, Waymo, and nuScenes, demonstrating that our Uni 2 Det outperforms existing methods by a large margin in multi-dataset training.
Notably, results on zero-shot cross-dataset transfer validate the generalization capability of our proposed method.

## 1 Introduction

With the ability to capture precise geometric information of entire scenes, LiDAR has become an essential sensor for most autonomous vehicles.
Due to the rapid development of large-scale annotated 3D LiDAR datasets such as Waymo (Sun et al., ), nuScenes (Caesar et al., ), and KITTI (Geiger et al., ), LiDAR-based models play a significant role in various critical perception tasks for autonomous vehicles, particularly in 3D object detection.
Recent studies (Lang et al., ; Deng et al., ; Shi et al., ; ; Chen et al., ; Wei et al., ; Yin et al., ; Wang et al., ) have made significant advancements in 3D detection using large-scale benchmarks and have demonstrated superior performance by leveraging precise 3D geometric information extracted from point clouds.
However, despite these breakthroughs, current LiDAR-based models typically adhere to a paradigm of training and testing within a single dataset, which limits the source data to a narrow domain, as shown in Figure 1(a).
Deploying dataset-specific models directly onto other datasets equipped with different LiDAR systems often leads to significant performance degradation due to substantial domain shifts (Yang et al., ; ).
Consequently, the single-dataset paradigm fails to produce a robust and generalizable perception model, leading to poor performance on different datasets and further impairing the generalization ability.

The availability of vast training data in 2D vision (Goyal et al., ; Kirillov et al., ; Wang et al., ) has facilitated research into joint training of unified detectors for 2D perception tasks. However, 3D vision has not fully benefited from these advancements due to significant cross-dataset discrepancies. Thus, further exploration of multi-dataset training strategies in 3D perception tasks, particularly 3D object detection, is urgently needed.
A direct approach to designing a unified 3D object detection framework for achieving multi-dataset training (MDT) involves merging multiple datasets and retraining the baseline detector on the merged dataset.
However, significant domain gaps exist between 3D datasets, and directly combining multiple data sources can result in negative transfer.
Some efforts (Zhang et al., ) focused on 3D multi-dataset object detection have offered solutions for building a unified training paradigm for point cloud data from different domains. As shown in Figure 1(b), the overall framework is designed in a dataset-specific manner, sharing certain backbone parameters while employing separate normalization and head layers for different datasets. Despite alleviating the unavoidable data-level differences to some extent, this independent paradigm suffers from two challenges: (i) this paradigm inhibits the full mutual utilization of each dataset’s unique features, thereby constraining the further enhancement of the model’s capabilities; (ii) the capacity for generalization to unseen domains is constrained due to the customization of certain network parameters specific to the trained dataset.
There is a scarcity of research on both refining the unified multi-dataset training paradigm and improving its generalization to other 3D datasets in more real-world scenarios.
Our main goals include effectively unifying the processing of diverse larger-scale point cloud data and ensuring robust generalization of the trained model to unseen domains. Compared to other works, we focus on developing universal techniques for managing out-of-domain datasets. Accordingly, we define our universality as the ability of a model to be jointly trained on a set of specific datasets and to perform zero-shot detection on new datasets using corresponding dataset attributes as prompts, without the need for re-training.

Figure: Figure 1: Illustration of different training paradigms. Single-dataset training leverages separate detectors and heads for different datasets. Naive multi-dataset training conducts point range alignment and partially shares the parameters within detectors, but still with dataset-specific heads. We propose unified and universal training, where detectors and heads for different datasets are fully shared.
Refer to caption: x1.png

To achieve these goals, we propose Unified and Universal framework for 3D Detection (Uni^2Det), which integrates multi-stage prompting modules applicable to any LiDAR dataset and various 3D object detection baselines used in autonomous driving.
As shown in Figure 1(c), our approach enhances performance by learning parameters shared across datasets, referred to as unified and universal training. It utilizes diverse prompts, such as intrinsic dataset attributes that are easily obtained from target datasets but cannot be automatically learned.
Due to inherent discrepancies in large-scale 3D datasets, we perform point distribution correction during voxelization to learn unified point and voxel representations across datasets, centered on mean-shifted batch normalization. Furthermore, handling data with varying statistical distributions within the backbone remains a challenging problem. To mitigate variations in data distribution, particularly from the perspective of point range, we introduce BEV-based range masking that acts on BEV features. This approach provides prior signals for the 2D convolutional network, enabling it to effectively handle point clouds from different datasets in a unified manner. Additionally, we observe that the same category exhibits statistical differences across datasets, which hinders the effectiveness of a universal detection head to some extent. To this end, we learn object-conditional residuals as prompts acting on each RoI feature, integrating features from pre-trained heads with new knowledge about the target domain.
Our method does not simply combine disparate modules but should be regarded as an integrated system. It allows different components of the 3D detection framework to leverage dataset-specific prior prompts, compensating for inter-dataset variations without requiring multiple branches.
Benefiting from it, models can fully utilize diverse datasets for joint training, thereby improving in-domain detection performance.
At the same time, the prior characteristics of unseen datasets can also be leveraged within a unified network as encoded prompts, enabling better out-of-domain generalization. Furthermore, this framework facilitates seamless plug-and-play integration within various advanced 3D detection frameworks while allowing direct adaptation for universal applicability across datasets.

Our main contributions consist of three parts:

- •
We introduce a novel training paradigm for 3D object detection which focuses on unified and universal multi-dataset training, aiming at enhancing the performance in MDT settings.
- •
We present Uni^2Det, a novel framework on 3D detection with multi-stage prompting modules for prompting various components in a detector including voxelization, backbone and head, enabling robust performance across diverse domains and generalization to unseen domains.
- •
Experiments conducted across multiple dataset consolidation scenarios involving KITTI, Waymo, and nuScenes demonstrate that Uni^2Det significantly outperforms existing methods in multi-dataset training, especially on the generalization capability of the model.

## 2 Related works

### 2.1 LiDAR-based 3D object detection

LiDAR-based 3D object detection aims to produce a collection of 3D bounding boxes along with their associated object categories using a LiDAR point cloud.
Current LiDAR-based 3D object detection research (Lang et al., ; Shi et al., ; ; ; Yan et al., ; Shi et al., ) can be broadly categorized into point-based methods, voxel-based methods, and hybrid point-voxel-based methods.
Point-based methods, such as Point-RCNN (Shi et al., ) and 3DSSD (Yang et al., ) generate feature maps directly from raw point clouds, thereby leveraging more accurate geometry information compared to previous methods.
Unlike point-based methods, Voxel-based methods like VoxelNet (Zhou & Tuzel, ) and SECOND (Yan et al., ) initially voxelize the input point cloud, transforming irregular LiDAR points into ordered voxels, and then extract features using 3D convolutions.
PointPillars (Lang et al., ) encodes the input point cloud into pillars and employs 2D convolutions for feature extraction.
Voxel-RCNN (Deng et al., ) analyzes the advantages of voxel features and explores a balanced trade-off between detection accuracy and inference speed.
Additionally, some studies attempt to merge the advantages of point-based and voxel-based representations.
PV-RCNN (Shi et al., ) and PV-RCNN++ (Shi et al., ) leverage both multi-scale 3D voxel CNN features and PointNet-based features, consolidating them into a concise set of keypoints using a newly proposed voxel set abstraction layer.
Nevertheless, all the aforementioned detectors are trained and evaluated using separate 3D datasets, leading to significant degradation in detection accuracy when applied to other different datasets.

### 2.2 Multi-dataset training

In recent years, training on multiple diverse datasets has emerged as an effective strategy for enhancing model robustness. Multi-dataset training has been previously investigated in the image domain, particularly in tasks such as object detection (Zhou et al., ; Wang et al., ) and image segmentation (Lambert et al., ).
For perception tasks (Dai et al., ; Gong et al., ; Zhao et al., ), dataset unification involves consolidating various semantic concepts.
Early studies (Lambert et al., ; Zhao et al., ; Xu et al., ) have focused on merging taxonomy information and training models on a unified label space.
MSeg (Lambert et al., ) manually unified the taxonomies of different semantic segmentation datasets and resolved inconsistent annotations between them.
Universal-RCNN (Xu et al., ) trains a partitioned detector on multiple large datasets and modeled class relations using an inter-dataset attention module.
To reduce the annotation cost associated with unifying the label space, recent studies (Zhou et al., ; Wang et al., ) have explored the use of dataset-specific supervision.
Although joint training of a unified detector has been studied in 2D perception tasks, further exploration in 3D perception tasks, such as 3D object detection, remains urgently needed.
Uni3D (Zhang et al., ) attempts to design a framework in a dataset-specific manner, sharing certain backbone parameters while employing separate normalization and head layers for different datasets. PPT (Wu et al., ) proposes a novel framework for multi-dataset synergistic learning in 3D representation learning that supports multiple pre-training paradigms with prompt-driven normalization.
However, its learning of dataset-specific prompts relies on an automatic approach that does not incorporate intrinsic characteristics of the dataset, which are not sufficiently universal since it requires new parameters for each new dataset.
To address these issues, we propose Uni^2Det for 3D detection, which integrates multi-stage prompting modules applicable to any LiDAR dataset and various 3D detection baselines, enabling robust performance across domains and generalization to unseen domains.
Our method leverages the dataset’s inherent properties, such as point cloud distribution and range, which are easily obtainable in other datasets and is inherently more generalizable than other methods.

## 3 Method

The overall framework is shown in Figure 2. We first describe our problem setting and the multi-dataset evaluation method in Sec. 3.1. Next, we introduce our multi-stage prompt learning modules for multi-dataset 3D detection, from various components in detectors including Voxelization in Sec. 3.2, Backbone in Sec. 3.3 and Head in Sec. 3.4.

### 3.1 Preliminary

In the realm of 3D object detection, the task involves analyzing an input frame of LiDAR points to predict associated labels, including categories and orientated bounding boxes. Training an object detection model $\mathcal{F}$ with its parameter $\Theta$ on a single dataset typically involves a straightforward approach: minimizing the 3D detection loss $\ell$ over a set of point clouds $\mathbf{x}$ and its corresponding ground truth $y$ from the dataset $\mathcal{D}$:

$$ $\min_{\Theta}\mathbb{E}_{(\mathbf{x},y)\in D}\left[\ell(\mathcal{F}(\mathbf{x} ;\Theta),y)\right].$ (1) $$

Suppose that a dataset is characterized by a joint probability distribution $P_{XY}$ over the input point cloud and label space $\mathcal{X}\times\mathcal{Y}$.
In the scope of multi-dataset training (MDT), we possess $N$ datasets $\{\mathcal{D}_{i}\}_{i=1}^{N}$ originating from diverse domains.
Each $\mathcal{D}_{i}$ is linked to a distinct data distribution $P_{XY}^{i}$.
The goal of MDT is to utilize multiple labeled datasets for training a unified model $\mathcal{F}:\mathcal{X}\rightarrow\mathcal{Y}$, aiming for increased generalizability and minimized prediction errors across various domains.
One straightforward strategy entails merging all datasets into a substantially larger one, denoted as $\mathcal{D}_{merge}=\mathcal{D}_{1}\cup\mathcal{D}_{2}\cup\cdots\cup\mathcal{D
}_{N}$.
While datasets may feature distinct label spaces, our training and evaluation are limited to categories relevant to autonomous driving scenarios: vehicle, pedestrian, and cyclist.
Consequently, the label space $\mathcal{Y}$ can be shared across various domains.
This approach optimizes the same loss function over the expanded dataset $\mathcal{D}_{merge}$:

$$ $\min_{\Theta}\mathbb{E}_{(\mathbf{x},y)\in\mathcal{D}_{merge}}\left[\ell( \mathcal{F}(\mathbf{x};\Theta),y)\right]$ (2) $$

In the following sections, we present the design of our Uni^2Det and show how to train a 3D perception model that performs well on seen datasets and generalizes to unseen datasets.

Figure: Figure 2: Illustration of the overall framework of Uni^2Det. The multi-stage prompting modules are employed as the core component to make the detection more unified and universal.
Refer to caption: x2.png

### 3.2 Prompt for voxelization: point distribution correction

The discrepancy in point distribution among different datasets poses a significant challenge in multi-dataset training. We use KITTI as an example. Since point features are typically related to coordinates in the ego-coordinate system, the feature channel corresponding to the x-axis consistently has values greater than 0 in KITTI. As a result, differences between KITTI and other datasets cause variations in the point cloud feature distribution, which in turn influences the training process across multiple datasets. To address data-level discrepancies in large-scale annotated 3D LiDAR datasets, we aim to develop simple modules during voxelization. These modules will enable existing 3D detectors to learn universal point and voxel representations across diverse datasets, as shown in Figure 3(a).

#### Point representation learning

Instead of relying on coordinates as point features, certain studies have explored effective methods of fusing information from various viewpoints. This is achieved through a learnable network incorporating a linear layer and batch normalization.
However, in this approach, the batch normalization process does not account for MDF training scheme, where points within the batch come from frames in various datasets having large statistic differences.
To address this, we introduce a new normalization approach termed “Mean-shifted batch normalization” to perform instance-level feature correction.
Compatible with any 3D detectors, this method can alleviate statistical differences in features extracted by standard 2D or 3D backbones.

#### Mean-shifted batch normalization

After the linear layer, we obtain a batch of point features represented as $P=\{p_{1}^{1},p_{2}^{1},...,p_{j}^{i},...,p_{N_{M}}^{M}\}$ from $M$ frames. Here, $p_{j}^{i}$ denotes the $j$-th point feature, and $N_{i}$ represents the total number of points in the $i$-th frame.
Conventional BN carries out normalization across all frames (instances) under the assumption that all data follows the same distribution.
However, two frames may exhibit disparate point ranges due to the different sensors in use and even if they come from the same sensor, the distribution of points may still be highly random.
To address this, under the MDF setting, we argue that instance-level statistics are also crucial and introduce mean-shifted batch normalization.
Subsequently, samples from each dataset are regularized using the basic mean $\mu$ with an adjustment from the current instance-specific mean $\mu^{i}$, as follows:

$$ $\hat{p}_{j}^{i}=\frac{p_{j}^{i}-\alpha\mu^{i}-(1-\alpha)\mu}{\sqrt{\sigma+ \epsilon}},$ (3) $$

where $\mu$ and $\sigma$ denote the channel-wise mean and variance of the feature set $P$, which are employed for conventional channel-wise feature normalization to ensure the input data conforms to zero-mean and unit-variance, and $\epsilon$ is added to ensure numerical stability. Here we maintain the sharing of variance $\sigma$. $\alpha\in[0,1]$ is a balancing ratio for the shifted mean. When $\alpha=0$, it is equivalent to performing the regular BN operation, while $\alpha=1$, the normalization procedure disregards the basic mean and relies solely on instance-level statistics. The subsequent transformation step for $\hat{p}_{j}^{i}$ remains the same as in conventional batch normalization. This approach allows us to learn universal point and voxel representations across diverse datasets with instance-level statistics as regularization. Compared with PPT (Wu et al., ) and Uni3D (Zhang et al., ), our approach operates only at the voxelization stage, using low-level point cloud distribution as a dataset attribute to prompt and regularize features. The advantage of feature uniformity ensures that the training of subsequent model parts is not negatively affected by feature differences between datasets.

Figure: Figure 3: Illustration of multi-stage prompting modules, including three modules for prompting different components of the detector.
Refer to caption: x3.png

### 3.3 Prompt for backbone: BEV-based range masking

In the realm of modeling point clouds in MDF setting, learning unified features from diverse sources and domains poses a significant challenge due to variations in the point range and data distribution across different datasets.
To address this challenge, we introduce BEV-based range masking acting on BEV features to effectively handle point clouds from different datasets, as shown in Figure 3(b).

Given the preconfigured point range $(x_{1},y_{1},x_{2},y_{2})$ for producing BEV features with an aligned coordinate system, where $x_{1}<x_{2}$, $y_{1}<y_{2}$, we can infer a binary mask for each dataset based on its point range.
The purpose of this mask is to explicitly indicate whether the regions or grids on the BEV plane are inside the point range of the frame.
Suppose $H$ and $W$ are the spatial shape of the BEV plane, and $(x_{1}^{i},y_{1}^{i},x_{2}^{i},y_{2}^{i})$ is the point range of the $i$-th dataset.
We naturally map this point range to the BEV plane based on the preconfigured point range according to the following equation:

$$ ${x_{1}^{i}}^{\prime}=\lfloor\frac{(x_{1}^{i}-x_{1})}{x_{2}-x_{1}}H\rfloor,{y_{ 1}^{i}}^{\prime}=\lfloor\frac{(y_{1}^{i}-y_{1})}{y_{2}-y_{1}}W\rfloor,{x_{2}^{ i}}^{\prime}=\lceil\frac{(x_{2}^{i}-x_{1})}{x_{2}-x_{1}}H\rceil,{y_{2}^{i}}^{ \prime}=\lceil\frac{(y_{2}^{i}-y_{1})}{y_{2}-y_{1}}W\rceil.$ (4) $$

Having the mapped point range on the BEV plane $({x_{1}^{i}}^{\prime},{y_{1}^{i}}^{\prime},{x_{2}^{i}}^{\prime},{y_{2}^{i}}^{
\prime})$, we can obtain mask $M^{i}\in\mathrm{R}^{H\times W}$ for the $i$-th dataset by:

$$ $M^{i}_{m,n}=\left\{\begin{array}[]{ll}1&\begin{aligned} &\text{if}\ {x_{1}^{i} }^{\prime}\leq m\leq{x_{2}^{i}}^{\prime},\ {y_{1}^{i}}^{\prime}\leq n\leq{y_{2 }^{i}}^{\prime}\end{aligned}\\ 0&\text{if others.}\end{array}\right.$ (5) $$

Given a frame of point clouds from the $i$-th dataset, our approach concatenates BEV features with the corresponding masks $M^{i}$ along the feature dimension before each 2D convolutional layer.
Leveraging this prior signal, the network can effectively adapt to point clouds from various datasets, which avoids excessive focus on the area outside the relevant regions, thereby maintaining the integrity of crucial information.
This integration provides a solution for the unified backbone to model features across datasets, which not only preserves dataset-specific information but also enhances the robustness and adaptability of feature modeling.

### 3.4 Prompt for head: object-conditional residual learning

The prompting modules in previous stages facilitate the framework to become more ‘unified’. On the other hand, learning a general detection head is crucial in designing a ‘universal’ framework. Since previous works use multiple dataset-specific detection heads for prediction, such designs cannot be directly transferred to new datasets and therefore cannot be considered universal. In this section, we explore the potential of a universal detection head for predicting point clouds from diverse domains without dataset-specific branches. However, directly training a detection head on multiple datasets poses challenges. As noted in previous works, the same category exhibits statistical differences across datasets, motivating us to design prompts to mitigate the distribution gap. Consequently, we introduce object-level residual learning, inspired by (Yu et al., ), on RoI features, integrating them from pre-trained heads with new knowledge about the target domain, as shown in Figure 3(c). Instead of learning a set of object-agnostic task residuals, we argue that learning object-conditional residuals is more effective and transferable to unseen domains as prompts.

Given a batch of RoI features $X=\{x_{i}\}_{i=1}^{N}$ from frames of different datasets and their labels $Y=\{y_{i}\}_{i=1}^{N}$, where $y_{i}=j$ if feature $x_{i}$ is from a frame of the $j$-th dataset, we feed each RoI feature $x_{i}$ into a residual function $f$ to obtain object-conditional residual $r_{i}$. The generation process is formulated as $r_{i}=f(\text{SG}(x_{i}))$, where SG indicates the stop-gradient operation to prevent hindering the regular learning of RoI features. Since the generated residuals should be relevant to the domain or dataset to which the feature belongs, we design a discriminator $D$, implemented by an MLP, to distinguish these residuals, using the dataset ID as a prior label. The discrimination process is formulated as $\hat{y}_{i}=D(r_{i})$. We use the cross-entropy loss to measure the discrimination loss $\mathcal{L}_{dis}$ between the predicted label set $\hat{Y}=\{\hat{y}_{i}\}_{i=1}^{N}$ and the ground truth label set $Y$, which is further added to the regular detection loss $\mathcal{L}_{det}$ as the final loss. By learning such object-conditional residuals, we can enhance the original RoI feature with prior dataset-specific characteristics by $\hat{x}_{i}=x_{i}+r_{i}$, and models will tend to make predictions according to a specific distribution, thus mitigating the influence of statistical differences in taxonomy across datasets.

## 4 Experiments

### 4.1 Experimental setup

#### Datasets.

Our experiments are conducted on three commonly used autonomous driving datasets: Waymo (Sun et al., ), nuScenes (Caesar et al., ), and KITTI (Geiger et al., ).
Waymo (Sun et al., ) stands out as the largest dataset with over 230,000 annotated 64-beam LiDAR frames gathered from six US cities.
nuScenes (Caesar et al., ) comprises 28,130 training samples and 6,019 validation samples collected using 32-beam LiDAR.
KITTI (Geiger et al., ) includes 7,481 annotated LiDAR frames collected via 64-beam LiDAR.
These datasets exhibit variations in data-level distributions arising from disparities in LiDAR types, geographic location of data acquisition, and variations in the definition of categorical annotations.

#### Implementation details

The experiments are conducted using OpenPCDet (Team et al., ).
Particularly, we note that differences in point cloud range significantly degrade cross-dataset detection accuracy.
Therefore, we align the point cloud range of all datasets to [75.2, 75.2]m for the X and Y axes and [2, 4]m for the Z-axis.
In all experimental settings, we follow Uni3D (Zhang et al., ) and employ the standard optimization techniques utilized by PV-RCNN (Shi et al., ) and VoxelRCNN (Deng et al., ).
For the balancing ratio $\alpha$ in our proposed mean-shifted batch normalization, we set $\alpha=0.1$ for VoxelRCNN and $\alpha=0.5$ for PV-RCNN.
This involves using Adam optimizer with an initial learning rate of 0.01 and implementing the OneCycle learning rate decay strategy.
The network is trained across 8 NVIDIA A800 GPUs, with a total training epoch set to 30.
For the experiments on Waymo-KITTI and nuScenes-KITTI consolidations, the weight decay is set to 0.01, while for Waymo-nuScenes consolidation, it is set to 0.001.
We utilize only 20% of the uniformly sampled frames on Waymo dataset for model training.

#### Evaluation metric.

We utilize the official tools to evaluate the performance of all baselines and our method, following (Zhang et al., ). For Waymo, we use Average Precision (AP) and Average Precision re-weighted by Heading (APH) for each class, based on the LEVEL 1 metric. For KITTI and nuScenes, we report Average Precision (AP) in both Bird’s Eye View (BEV) and 3D over 40 recall positions, with moderate case results for KITTI. AP is evaluated with an IoU threshold of 0.7 for the car category (Vehicle on Waymo) and 0.5 for pedestrian and cyclist categories. All experimental results presented in this paper are reported on the official validation set.

### 4.2 Results of zero-shot evaluation on unseen datasets

To comprehensively assess the universal design of our framework, we perform a zero-shot evaluation on several unseen 3D datasets using different detection models.
Our approach, Uni^2Det, is compared against two main baselines. First, we benchmark it against the ”Source Only” model, which only uses the source domain for training, and against a robust generalization method, SN (Wang et al., ), within the single-source generalization setting. Additionally, we extend the comparison to the dual-source generalization scenario, evaluating Uni^2Det against simple data-merging strategies and Uni3D, along with its variants.
We also attempt to integrate SN into our method with extra statistical supervision on the target domain.
As shown in Table 1, our Uni^2Det is proved to achieve more generalized representations on a single dataset as well, further enhancing performance based on SN. This is because our universal framework effectively utilizes the prior information associated with target datasets so as to perform better adaptation. For the dual-source generalization, we discover the trend that using a single detection head in Uni3D yields superior generalization performance compared to employing multiple detection heads. This suggests that training the detection head across multiple domains enhances its generalization ability and mitigates overfitting to any single domain. Building on this insight, our Uni^2Det framework employs multi-stage prompts to enable more unified and universal training, which not only boosts zero-shot generalization performance but also maintains a competitive edge in in-domain tasks. By comparing the overall results, we confirm that Uni^2Det significantly improves generalization performance when incorporating new datasets, demonstrating its potential for robust 3D detection across varying domains.

### 4.3 Results of multi-dataset 3D object detection

To evaluate the unified design of our framework, we conduct experiments on the two-dataset combination of three widely-used autonomous driving datasets: Waymo (Sun et al., ), KITTI (Geiger et al., ), and nuScenes (Caesar et al., ), and report our results in Table 2 with comparison to baselines demonstrated in (Zhang et al., ). We summarize our findings as three points.

Firstly, performance improvement from multi-dataset training is guaranteed for Uni^2Det. In some cases, the previous state-of-the-art Uni3D shows worse results under multi-dataset training than when trained on a single dataset (e.g., results on Waymo under Waymo-nuScenes consolidation), indicating that Uni3D did not fully and effectively utilize data from multiple datasets for training. In contrast, our Uni^2Det avoids this issue and ensures improved performance with additional datasets for training, demonstrating excellent scalability.

Secondly, a dataset-agnostic head is feasible instead of dataset-specific head. Although the improved results of Uni3D compared to simply merging datasets demonstrate the benefits of learning dataset-specific detection heads, our work addresses the poor performance issue with a single detection head through a unified paradigm, proving the feasibility of learning a dataset-agnostic detection head. Using more training data from different datasets to train a single detection head in our unified paradigm is more likely to enhance detection performance.

Lastly, Uni^2Det is considered a more robust and unified framework for multi-dataset training. Across all dataset combinations, our Uni^2Det consistently outperforms Uni3D, demonstrating the effectiveness of our approach in a multi-dataset setting. This result also indicates that our unified training paradigm is stable and robust, capable of converting point cloud data from any source domains or datasets into a more unified distribution through prompts for better prediction. However, our method shows lower AP when inferring some categories (e.g., results for Car on nuScenes under KITTI-nuScenes consolidation). Despite this, considering the boost from other categories within the dataset, it maintains an overall improvement across each dataset.

Table: Table 1: Results of zero-shot evaluation on unseen datasets. Source Only denotes that the model is trained on the source domain and directly tested on the target domain. S.H. for Uni3D (Zhang et al., ) indicates using a single head instead of dataset-specific heads as a variant. Results are reported for Car category.

Table: Table 2: Results of joint training on different dataset consolidations. Following Uni3D (Zhang et al., ), we report the car (Vehicle on Waymo), pedestrian, and cyclist results under IoU threshold of 0.7, 0.5, and 0.5, respectively, and utilize AP/APH of LEVEL 1 metric on Waymo, and APBEV/AP3D over 40 recall positions on nuScenes and KITTI. P.T. indicates pre-training the baseline detector on the other dataset, and fine-tune the detector on the current dataset. D.M. stands for simply merging data from different datasets as the training data. The best detection results are marked using bold. Here we adopt Voxel-RCNN as the baseline detector. Due to the page limitation, we present the experimental results based on PV-RCNN in Appendix.

**Table 3: Results of jointly training the Voxel-RCNN on three datasets.**
| Trained on | Tested on |  |  |  |
| --- | --- | --- | --- | --- |
| KITTI | NuScenes | Waymo | Avg. |  |
| KITTI | 70.04/66.09 | 3.84/1.58 | 12.69/11.47 | 28.86/26.38 |
| nuScenes | 32.64/17.70 | 28.99/22.20 | 14.63/14.04 | 25.42/17.98 |
| Waymo | 64.00/45.27 | 12.38/6.34 | 71.84/69.23 | 49.41/40.28 |
| Uni3D | 72.19/67.46 | 35.06/26.48 | 71.95/69.28 | 59.73/54.41 |
| Uni^2Det | 76.04/72.61 | 34.03/25.44 | 72.45/70.20 | 60.84/56.08 |

Figure: Figure 4: Illustration of statistical distribution differences of object size (length, width, and height) in KITTI between the ground truth and the predictions of Uni^2Det with and without object-conditional residual learning (OCRL) module.
Refer to caption: extracted/5890807/Styles/Figure/fig4.png

### 4.4 Further analysis

#### Results on Waymo-KITTI-nuScenes Consolidations.

Table 4 shows the results of jointly training Voxel-RCNN on Waymo-KITTI-nuScenes consolidations. We report average AP over all categories within the dataset. Our Uni^2Det demonstrates high detection results on KITTI and Waymo, on which the results of Uni3D do not differ much from those on the single dataset.
Overall, more balanced and consistent boosts on different datasets can be observed in Uni^2Det on average APBEV and AP3D.

#### Ablation on prompts in different stages.

We investigate the influence of prompts in different stages of Uni^2Det, including voxelization, backbone, and head. The evaluation setting follows the KITTI-Waymo consolidation in Table 2. As shown in Table 4, the prompt modules implemented at each stage enhance performance compared to the baseline. Notably, the point distribution correction at the voxelization stage shows the most significant improvement among all stages. This demonstrates that learning unified low-level point features is crucial for MDF and serves as a foundation for subsequent prompting stages. We also observe that gradually adding prompts stage-by-stage results in noticeable performance gains for each stage, reflecting the complementarity between the prompting modules. At last, using prompts from all stages together achieves a significant improvement compared to the baseline.

#### Ablation on object-conditional residual learning.

We investigate the statistical distribution differences of object size in KITTI between the ground truth and the predictions of Uni^2Det with and without object-conditional residual learning (OCRL) module. As shown in Figure 4, it is evident that our method, particularly with the OCRL module, more closely approximates the ground truth distribution, especially in terms of mean and variance, thereby reducing statistical differences across datasets and mitigating potential performance degradation.

#### Ablation on normalization strategies.

We conduct ablation studies of normalization strategies in PPT (Wu et al., ), Uni3D (Zhang et al., ) and our Uni^2Det on the Waymo-KITTI consolidation based on Voxel-RCNN, as shown in Table 5. There are no other well-designed modules and only with a single detection head for fair comparison. Our proposed mean-shifted batch normalization is more advantageous, especially on KITTI where only the front view is provided, and it is important to note that the other two compared methods are not conducive to generalization scenarios, which further highlights our strengths.

Table: Table 5: Ablation on normalization strategies based on Voxel-RCNN.

| Method | KITTI | Waymo |
| --- | --- | --- |
| Baseline | 72.73/69.94 | 71.09/69.12 |
| Uni3D | 73.01/70.12 | 71.45/69.23 |
| PPT | 73.25/70.09 | 71.73/69.46 |
| Uni^2Det | 73.84/70.56 | 71.71/69.73 |

#### Ablation on normalization strategies.

We conduct ablation studies of normalization strategies in PPT (Wu et al., ), Uni3D (Zhang et al., ) and our Uni^2Det on the Waymo-KITTI consolidation based on Voxel-RCNN, as shown in Table 5. There are no other well-designed modules and only with a single detection head for fair comparison. Our proposed mean-shifted batch normalization is more advantageous, especially on KITTI where only the front view is provided, and it is important to note that the other two compared methods are not conducive to generalization scenarios, which further highlights our strengths.

## 5 Conclusion

We introduce Uni^2Det, a novel framework designed for unified and universal multi-dataset training in 3D detection which utilizes multi-stage prompting modules to harmonize differences between datasets by leveraging dataset-specific characteristics, ensuring robust performance across various domains and effective generalization to new domains.
Our work is promising to enhance performance across various domains and facilitate effective generalization to new ones in 3D detection, which has the potential to advance fields like autonomous driving.

#### Limitation.

We reveal the limitation that our approach relies on the identical set of categories within different datasets, which hinders the detection on different label spaces with more diverse categories, and hope that future research based on our work will advance unified and universal 3D detection to be more inclusive and effective across a broader range of applications.

## Appendix

## Appendix A Extra Experiments

#### Results of multi-dataset 3D object detection based on PV-RCNN.

We present the experimental results for the two-dataset combination of three autonomous driving datasets using PV-RCNN, as shown in Table 6. The conclusions are consistent with those in Table 2, further demonstrating that Uni^2Det is a more robust and unified framework for multi-dataset training.

Table: Table 6: Results of joint training on different dataset consolidations based on PV-RCNN. The experiment and evaluation settings follow Table 2.

#### Ablation on the balancing ratio.

We conduct ablation studies regarding the hyperparameter balancing ratio based on different 3D detectors, based on AP3D metrics of KITTI on the Waymo-KITTI consolidation, as shown in Figure 5. When $\lambda=0.1$ for PV-RCNN and $\lambda=0.5$ for Voxel-RCNN, the performance can be optimal. Since Voxel-RCNN mainly relies on voxelized point cloud data and is more sensitive to the distribution of point clouds, the balancing ratio for point feature correction needs to be larger.

Figure: Figure 5: Ablation on the balancing ratio based on PV-RCNN and Voxel-RCNN, based on AP3D metrics of KITTI on the Waymo-KITTI consolidation.
Refer to caption: extracted/5890807/Styles/Figure/fig5.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: Check weaknesses
soundness: 3
strengths: - The paper tackles a practical issue in autonomous driving, where dataset diversity can restrict the ability to scale training. By addressing how to integrate datasets with different characteristics, the research directly benefits the community by opening up avenues for broader dataset utilization, potentially enhancing the generalization of autonomous driving models.
- The proposed approach is modular, designed as a plug-and-play component compatible with various advanced 3D detectors.
- The use of frame-specific mean shifts for batch normalization is particularly novel. I’m curious whether this concept has been explored in existing literature or if it is an original contribution by the authors.
- The paper includes comprehensive experiments to validate its claims, showing that: (1) the method outperforms baselines like Uni3D, (2) it can be integrated with various 3D detectors, (3) each of the three prompting stages contributes to the improved performance, and (4) OCRL improves the alignment of object size distribution with ground truth data.
- The paper is well-organized and easy to follow, with clear figures. However, providing a list of all abbreviations in figure captions would enhance clarity.
summary: This paper addresses a critical challenge in autonomous driving research: integrating multiple datasets collected across different locations and using varied sensor types, especially LiDAR-based detection datasets. Due to differences in distribution, such datasets cannot be directly combined for training without significant performance trade-offs. The authors propose a novel, multi-stage prompting approach to effectively leverage the unique characteristics of each dataset and mitigate modality disparities, ultimately enabling a more robust multi-dataset training framework. This multi-stage approach is designed as a plug-and-play module that enhances several advanced 3D detection frameworks by targeting three stages: voxelization, BEV feature processing, and the detection head. Experimental results substantiate that the proposed multi-dataset training technique significantly improves cross-domain performance relative to baseline models, showcasing its effectiveness in harnessing diverse datasets for training.
weaknesses: While the paper is methodologically sound and well-supported by experiments, there are a few areas that could benefit from additional clarity and detail:
- For mean-shifted BN: 
1) The authors claim that mean-shifted BN introduces dataset-specific characteristics, yet the parameter \alpha appears to capture only frame-specific characteristics rather than broader dataset-level traits. A more intuitive approach might involve defining dataset-specific means and variances to capture the unique properties of each dataset more accurately. I suggest the authors discuss this alternative and clarify how frame-level mean shifts contribute to dataset-level adaptability.
2) It remains unclear how the parameter \alpha is specified within the model. Further explanation of its determination, whether it is learned, fixed, or computed dynamically, would help readers understand its role in adapting the batch normalization to different datasets.
- For the BEV-based range masking component, the authors choose to concatenate the mask as an additional channel rather than directly masking out unwanted regions in the BEV representation. It would be valuable for the authors to provide a rationale for this choice. Specifically, how does the inclusion of the mask as a separate channel improve model performance or facilitate feature extraction compared to direct masking? A discussion on the impact of this approach on model interpretability and cross-domain generalization would also be insightful.
- More discussion about the limitation when applying the approach to the real-world applications will be helpful.

---

confidence: 2
contribution: 2
initial_rating: 6
presentation: 3
questions: (1) Sensor Variability: 

How does Uni2Det perform when dealing with datasets that utilize sensors with significantly different specifications? Understanding this could provide insights into its robustness across various sensor setups. For example, how is the performance on indoor datasets like Scannet [2] or SUN-RGBD [1]. 

(2) Extension to Diverse Categories: 

Is it possible to extend the framework to handle datasets with richer categories, like some open-vocabulary 3d object detection works [3, 4]? This would enhance its applicability to a wider range of scenarios. Please provide discussions in the paper.

(3) Efficiency Considerations: 

What are the computational overhead and latency introduced by the multi-stage prompting modules? Providing metrics on efficiency would be beneficial for assessing practical deployment scenarios.

(4) Failure Case Analysis: 

Can the authors provide examples or analysis of scenarios where the framework might underperform? This would help in understanding its limitations and potential areas for improvement. Please include failure cases in the paper.

[1] Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In CVPR, 2015.

[2] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017.

[3] Y. Lu, C. Xu, X. Wei, X. Xie, M. Tomizuka, K. Keutzer, and S. Zhang, “Open-vocabulary point-cloud object detection without 3d annotation,” CVPR, 2023.

[4] Y. Cao, Y. Zeng, H. Xu, and D. Xu, “Coda: Collaborative novel box discovery and cross-modal alignment for open-vocabulary 3d object detection,” in NeurIPS, 2023.
soundness: 3
strengths: (1) Innovative Techniques: 

The introduction of multi-stage prompting modules is a novel approach to address the differences in data distribution among various datasets, potentially setting a new standard for unified 3D detection frameworks.


(2) Comprehensive Experiments: 

The authors conducted extensive experiments across multiple prominent datasets, showing significant improvements in both multi-dataset training scenarios and zero-shot transfer performance. This thorough evaluation strengthens the validity of the proposed method.

(3) Robust Generalization: 

The framework shows impressive zero-shot capabilities, effectively adapting to unseen datasets without additional retraining. This highlights its potential for real-world applications where models need to perform well across diverse environments.

(4) Scalability: 

Uni2Det demonstrates improved scalability by effectively utilizing additional datasets for training, ensuring performance boosts without the pitfalls of negative transfer that other methods might suffer from.
summary: The paper introduces Uni2Det, a framework designed to tackle the challenges of 3D object detection across multiple datasets. By leveraging multi-stage prompting modules, Uni2Det aims to overcome data distribution discrepancies and enhance generalization capabilities. The framework is evaluated on datasets such as KITTI, Waymo, and nuScenes, demonstrating superior performance compared to existing methods.
weaknesses: (1) Complexity and Implementation: 

The multi-stage prompting modules, while innovative, may introduce additional complexity to the framework. This could potentially affect the ease of implementation and deployment in practical scenarios, requiring more resources and expertise.

(2) Assumption of Identical Categories: 

The framework relies on the assumption that datasets share identical categories, which limits its applicability to datasets with diverse or non-overlapping label spaces. This constraint could hinder the adoption of Uni2Det in broader applications.

(3) Limited Component Analysis: 

The paper could benefit from a more detailed analysis of how each component of the framework contributes to overall performance. Understanding the impact of each module could provide deeper insights and guide future improvements.

(4) Computational Overhead: 

The potential increase in computational requirements due to the additional modules is not thoroughly discussed. This could be a concern for deploying the framework in resource-constrained environments.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. In mean-shifted batch normalization, it would be helpful to further explain how to calculate the instance-specific mean. Additionally, it will be wonderful to compare the proposed method with an alternate approach of first de-mean data using instance-specific mean, following batch-wise demeaning without alpha weighting. Further more, what is the influence of completely discarding this stage, only using the backbone and the head?
2. In object-conditional residual learning, how is the dataset id encoded in training? Also, during testing on zero-shot datasets, does this pipeline obtain object-conditional residual before prediction? 
3. In testing, it will be interesting to see how the model performs on classes that do not intersect with existing class vocabulary. At the same time, applying this method to datasets with various scenes, i.e., indoor 3D detection datasets, could provide a broader perspective on its robustness.
soundness: 3
strengths: 1. The paper is easy to understand and has a clear structure.
2. De-meaning data across batches and within one instance is an interesting way to improve generalization. Using dataset-relevant information as prompts is also very interesting.
3. The performance seems good.
summary: This paper proposes a multi-stage prompting pipeline to use multiple data sources to train one 3D detector. Specifically, they performed mean-shifted batch normalization, applied BEV-based masking, and trained a discriminator on object-conditioned residual using classification loss. The object-conditioned residual is added to the RoI feature for prediction.
weaknesses: There are too few classes for evaluation, and the class domain is limited to autonomous driving scenarios. The authors mentioned this in section 3.1.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: - I know MDF may represent multi-dataset fusion. However, the authors use MDT in the preliminary and then use MDF in the following section. This needs to be modified to avoid confusion.
soundness: 2
strengths: - Experiments show that the proposed Uni$^2$Det can achieve SOTA performance on both cross-domain and multi-dataset training settings.
- The authors treat dataset-specific attributes as prompts which is quite interesting.
summary: In this work, the authors propose Uni$^2$Det, a prompt-guided multi-dataset 3D object detection framework. In detail, the authors handle multi-dataset 3DOD from three perspectives (*i.e.*, voxelization, bev features, and detection head). First, the designed point distribution correctness uses instance- and batch-level normalization to deal with the distribution shift in different datasets. Further, the bev-based range masking can select features of corresponding areas of the different datasets. Finally, the authors propose object-conditioned residual learning to inject dataset-specific features into each instance to further decode the bounding boxes. Experiments show that the proposed method can achieve higher results on both in-domain and out-of-domain tasks.
weaknesses: 1. My first question is about the proposed BEV-based range masking. I wonder about the masking strategy during the inference phase. Do we need to know the dataset-specific point cloud range before evaluation? If so, what will happen when we do not know the point cloud range? Besides, in the domain generalization experiments, if we use the point cloud range of the target domain, it seems unfair since the domain-specific knowledge has already been injected into the model.
2. Further, I wonder about the performance using more recent 3D detection models. I fully understand that the authors chose PV-RCNN and Voxel R-CNN for a fair comparison. However, they have been proposed for several years.
3. Questions about experimental settings, especially hyperparameters. I need to know how the authors define the value of hyperparameters such as $\alpha$, there seem no ablation studies about this.
4. The motivation of point mean-shift normalization and object-conditional residual learning is similar to the data-level correction and semantic-level coupling and recoupling. Both of them first handle data-level shifts and then strengthen domain-specific features. The authors need to explain the advantages of the proposed methods.

Meta-review: This paper presents a novel unified framework for prompt-guided multi-dataset 3D detection. The proposed method shows very significant performance on several challenging benchmarks including KITTI, Waymo, and nuScenes. All the reviewers are positive about the technical contributions made by this submission. AC also agrees with the comments of the reviewers and recommends an acceptance.

Decision: Accept (Poster)