One for All: Multi-Domain Joint Training for Point Cloud Based 3D Object Detection

Authors: zhenyu  wang
yali  li
hengshuang  zhao
shengjin  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 08:18:44

Abstract: The current trend in computer vision is to utilize one universal model to address all various tasks. Achieving such a universal model inevitably requires incorporating multi-domain data for joint training to learn across multiple problem scenarios. In point cloud based 3D object detection, however, such multi-domain joint training is highly challenging, because large domain gaps among point clouds from different datasets lead to the severe domain-interference problem. In this paper, we propose OneDet3D, a universal one-for-all model that addresses 3D detection across different domains, including diverse indoor and outdoor scenes, within the same framework and only one set of parameters. We propose the domain-aware partitioning in scatter and context, guided by a routing mechanism, to address the data interference issue, and further incorporate the text modality for a language-guided classification to unify the multi-dataset label spaces and mitigate the category interference issue. The fully sparse structure and anchor-free head further accommodate point clouds with significant scale disparities. Extensive experiments demonstrate the strong universal ability of OneDet3D to utilize only one trained model for addressing almost all 3D object detection tasks (Fig. 1). We will open-source the code for future research and applications.

Main paper: 
 related work: 
 3D object detection aims to predict category tags and oriented 3D bounding boxes for the scene. We primarily discuss methods where point clouds serve as the input. Current 3D detectors can be generally categorized into point-based and voxel-based methods. Point-based methods [25, 51, 21, 34] usually extract point-wise features, then perform clustering and classification for detection. Voxel-based methods [30, 39, 32, 9, 31, 47, 33] usually extract voxel-wise features using 3D sparse convolution, then convert them into dense 3D features for 3D box prediction. Considering the disparities in point clouds, existing 3D detection methods are also separated into indoor 3D detectors [25, 51, 30, 39] and outdoor ones [45, 35, 32, 9, 31, 5, 33] , where totally different model architectures are utilized for each. Recently, [41] proposes a unified model architecture for both indoor and outdoor 3D detection. However, these methods still follow a single-dataset training-and-testing paradigm, and cannot address 3D detection for point clouds from various domains with one set of parameters.Multi-dataset training aims to involve multiple datasets from various domains in training, so that the model can generalize in multi-domain data at the inference time. Since RGB images mainly differ in content, while the structural differences in images themselves are not significant, multi-dataset training has been widely studied in the field of 2D object detection [40, 52, 55, 24, 42] . In comparison, substantial differences inherently exist in the point clouds themselves, making multi-dataset training more challenging in the 3D object detection task. Some recent works [48, 43, 46] have studied this problem. However, they only address multi-dataset training within either indoor or outdoor scenes and cannot handle multiple datasets simultaneously from both indoor and outdoor scenes. For example, [40] and [12] deal with multi-dataset training with RGB images, [20] and [48] focus on outdoor-only multi-dataset training, where the discrepancies between different datasets are far less than those between indoor and outdoor point clouds. OneDet3D demonstrates that despite these substantial differences, 3D detection can still be addressed with a universal solution. This is a crucial advancement for generalization in the 3D domain. 
preliminary: 
 Given a point cloud x, 3D object detection aims to predict its label y, which consists of the category tags and 3D bounding boxes. Multi-domain (i.e., multi-dataset) data are utilized during training. Denote the domain as D and the total number of domains as N , the total training data can thus be denoted asD = {D n = {(x (n) , y (n) )}} N n=1 .The purpose of multi-domain joint training is to train a unified model from all these domains, which can obtain the minimum prediction error on all different domains D. The obtained 3D detector should also generalize well on new domains. [11] 64 Data-level interference. As in Tab. 1, it can be observed that sensors for collecting indoor and outdoor point clouds exhibit fundamental differences, resulting in significant disparities in the range covered by the point clouds, with differences exceeding 10 to nearly 20 times. This also leads to substantial differences in object sizes and sparsity within the scenes. Because of such scale differences, it is challenging to utilize the same point-wise clustering technique or feature map with the fixed size during joint training for point clouds from different scenes. Even among datasets that belong to the same category of indoor or outdoor point clouds, there are still slight differences in the sensors used for collection. For instance, SUN RGB-D [36] points are from RGB-D camera captures, while ScanNet points are reconstructed from RGB images. Distinction in the number of LiDAR beams also leads to differences in point cloud sparsity. We thus propose domain-aware partitioning in section 4.2, guided by the routing technique, to alleviate such data-level interference.Category-level interference. Different datasets typically possess distinct label spaces. An object classified as background in one dataset might be considered as foreground in another. Even for the same category, different datasets sometimes employ different classification and definition ways, such as the definition of the "car" category in outdoor datasets. Such dataset-specific taxonomy and annotation inconsistencies pose challenges in unifying multiple label spaces. The category-level differences thus result in the interference problem among multiple datasets during training. We propose the language-guided classification in section 4.3 to mitigate such category-level interference. 
method: 
 The overview of our OneDet3D is illustrated in Fig. 3 . We utilize 3D sparse convolution for feature extraction and anchor-free detection head for 3D box prediction. Based on it, we propose the domainaware partitioning during feature extraction to alleviate data-level interference, and propose the language-guided classification in the anchor-free head to mitigate category-level interference.Architecture. We design the architecture of OneDet3D from the feature extractor and the detection head aspects. For the feature extractor, we utilize 3D sparse convolution to extract voxel-wise features.Compared to point-wise structures, voxel-wise features are more robust to domain gaps and less sensitive to hyper-parameters, suitable for multi-domain training. Additionally, sparse convolution is not only computationally efficient but also operates solely on points, thus not relying on fixed-size feature maps. This enables to extract domain-invariant 3D features for multi-domain joint training.For the detection head, we adopt the anchor-free way, where objects are represented by their center points. It directly regards points from sparse convolutions as centers to represent objects, avoiding the need for conversion from sparse to dense feature maps. We do not employ any pruning layers [13, 30] . For classification, we use soft focal loss as in [41] . For easier optimization, we utilize the IoU in the BEV space, IoU BEV , as the soft target. Specifically, denote the binary target class label as c, the predicted class probability as p, the classification loss is:InsteadEQUATIONwhereαt = α•c•IoU BEV +(1-α)•(1-c•IoU BEV ).This classification loss can be viewed as using the soft target IoU BEV in focal loss [19] . By employing IoU BEV , the network classification focuses solely on the position on the horizontal plane, which helps improve the calibration of classification scores. Here, we discard positional information in the height direction to prevent optimization from becoming overly complex, which makes the network easier to converge during joint training.For regression, we use the 3D IoU loss [53] , optimizing with the usual 3D IoU. For both centerness and IoU prediction, we utilize the binary cross entropy loss. The IoU prediction branch is also supervised with the usual 3D IoU. Since different datasets vary in scale, we employ dataset-aware sampling during joint training: sampling datasets first and then randomly selecting samples. Denote network parameters as θ, the objective of network training can thus be formulated as:arg min θ N n=1 1 |D n | (x (n) ,y (n) )∈Dn L (n) cls + L (n) reg + L (n) centerness + L (n) iou (2)During multi-domain joint training, we first aim to mitigate the data-level interference caused by differences in the inherent structure of point clouds. We identify two primary sources of interference. First, due to significant differences between data, interference mainly arises in the normalization layers, which adjust the scatter of data to maintain their consistency. Second, convolution mainly focuses on local information, leading to interference in context learning across different domain point clouds, where scale difference mainly exists. Therefore, we partition parameters about these two aspects into domain-specific ones. We design a domain router to guide such domain-aware training.In this way, the partitioned parameters are responsible for learning domain-equivalent knowledge, while the majority of the model can avoid interference and learn domain-invariant 3D knowledge. This allows multi-domain joint training to effectively acquire universal 3D representations.Given the input point cloud x (n) , the domain router aims to guide its path for the domain-aware partitioning. We utilize a domain classifier for the routing mechanism by classifying its correct domain label n. To achieve this, we employ 3D sparse convolutions with kernel sizes of 3 and 1 for simple feature extraction, then utilize global average pooling (GAP) to obtain the feature of the whole scene. After applying softmax, we obtain the domain probability {p Scatter partitioning. The normalization layers conduct regularization for input data, thus reducing hierarchical differences in data and making the network easier to train. Then normalization layers conduct re-scaling to adjust the scatter of data. Considering the significant differences in different domains, the same re-scaling operations will lead to disparities in the output data scatters. In this situation, we partition scaling and shifting parameters after normalization for each domain data, so that data scatter in different domains can be partitioned. All other convolution layers can be shared, only scaling and shifting parameters being domain-specific and differing. For unseen domains at the inference time, we introduce the domain probability from the domain router. Specifically, we keep N sets of scaling and shifting parameters {γ(n) } N n=1 , {β (n) } N n=1 .Scatter partitioning can thus be formulated as:EQUATIONwhere we utilize x to denote the output of normalization layers. Through this, the network can apply individualized re-scaling operations from different domains, resulting in domain-specific scatter thus effectively mitigating the data-level interference. Only introducing N sets of scaling and shifting parameters almost negligibly increases the model size.Context partitioning. In addition, we separately learn global context information for different domain data to prevent interference between them in terms of global context. Specifically, for features f from the blocks in the feature extractor, we first apply a global average pooling layer to extract the feature of the whole scene, then utilize a 3D sparse convolution to learn its context information.According to previous work [41] , global information mainly matters in indoor scenes. We thus only impose context learning for indoor domains. The process of context partitioning can thus be formulated as:EQUATIONwhere we utilize f to denote the updated features with partitioned domain-aware global context.We then aim to alleviate category-level interference among domains caused by label conflicts. Different datasets inherently possess different label spaces, which leads to the problem of annotation inconsistency. Moreover, at the inference time, unseen domains may involve a label space that is different from those seen during training. Such a category-level difference results in different definitions of the same object, leading to the conflict and interference problems during training.To address this, we utilize language vocabulary embeddings from CLIP [28] for classification. Specifically, we use the prompt "a photo of {name}" to extract language embeddings of the category names from different datasets using CLIP. These language embeddings are then used as parameters of the fully connected layer to perform the final classification, and are kept frozen during training. Each dataset utilizes its own language embeddings, effectively mitigating such interference.Due to the fully convolution architecture and the anchor-free head, the final classification is typically achieved through 3D sparse convolutions. To incorporate language embeddings, we convert the sparse features of points into dense features, and then use language embeddings for classification through fully connected layers. However, this conversion from sparse to dense features, together with the frozen language embeddings, poses obstacles to gradient backpropagation, making it difficult for the network to converge. To address this, we introduce a class-agnostic classification branch that performs only foreground-background binary classification. This branch is shared across different datasets and implemented using 3D sparse convolution. In this way, a part of classification can be addressed through 3D sparse convolution, making it easier to converge. The classification probabilities from both branches are multiplied finally. The utilization of such shared class-agnostic classification across domains also facilitates the model in learning general category knowledge in the 3D domain.Open-vocabulary extension. Introducing language embeddings into classification enables our OneDet3D to be easily extended to the open-vocabulary setting, benefiting from the generalization ability of text to unseen categories. To further ensure category scalability, we follow [23] by first performing large-scale vocabulary inference on 2D images using a pre-trained 2D open-vocabulary detector [54] , then projecting the obtained 2D boxes into 3D space to obtain 3D pseudo labels with an expanded vocabulary. With such large-vocabulary pseudo labels for multi-domain joint training, the  
experiments: 
 In this section, we demonstrate the one-for-all ability of our OneDet3D through extensive experiments.Close-vocabulary and open-vocabulary 3D object detection experiments are both conducted. We mainly conduct multi-dataset joint training on SUN RGB-D [36] , ScanNet [8] , KITTI [11] , and nuScenes [2] datasets, and utilize S3DIS [1] and Waymo [37] for unseen domains in cross-dataset experiments. We implement OneDet3D with mmdetection3D [7] , and train it with the AdamW [22] optimizer. We use the 0.01m voxel size for indoor datasets and the 0.05m voxel size for outdoor ones. Besides this, other architecture-related hyper-parameters are all the same for different datasets.During multi-dataset training, the attribute channel size is set to the least common multiple of the attribute dimensions from the different datasets, which is 6-dim. The attributes of the point clouds from different datasets are repeated accordingly to match this unified channel size.We first conduct multi-dataset joint training on all the above mentioned four datasets, and perform closed-vocabulary inference. Specifically, for the SUN RGB-D dataset, we perform 3D detection on 10 classes, for ScanNet it's 18 classes, while for the KITTI and nuScenes datasets, we focus on the performance of the "car" category. The results are listed in Tab. Comparison with more recent methods. We further compare with some more recent 3D detectors and list the comparison in Tab. 3. As can be seen, these recent methods target at specific 3D scenes. They may outperform OneDet3D in those particular datasets, but AP tends to drop when the scene changes, especially when switching from outdoor to indoor. After multi-dataset training, due to the dataset-aware interference, AP on all datasets degrade severely. In such multi-dataset scenarios, OneDet3D still achieves the best. Besides, compared with Uni3D [48] , which has provided a unified model for outdoor point clouds, OneDet3D provides a universal solution for all point clouds. As can be seen, even compared with these recent methods, OneDet3D is still the first universal 3D detector that can generalize across various point clouds. The main reason is that the specific designs of these detection heads, such as vote-based methods or BEV detection, are influenced by the structure and content of point clouds and thus are only applicable to outdoor scenes. Additionally, these methods lack designs to address multi-dataset interference, resulting in performance degradation across all datasets during multi-dataset joint training. In contrast, the anchor-free detection head of our OneDet3D is more versatile for both indoor and outdoor scenes. Furthermore, domain-aware partitioning and language-guided classification can alleviate multi-dataset interference. Therefore, our approach provides a more universal solution for 3D detection. We then conduct open-vocabulary 3D object detection experiments with our OneDet3D. In our experiments, we refer to the setting in CoDA, where SUN RGB-D involves 46 classes and ScanNet involves 60 classes in total. Their top 10 classes are used as base categories. For multi-dataset joint training, we combine the base categories from both datasets to form a union, resulting in a total of 16 base categories, with the rest as novel categories. We reproduce the results of existing methods under this new category division and list the comparison in Tab. 4. It's worth noting that for a fair comparison, we used exactly the same setting as CoDA, which utilizes a single-view image setting in ScanNet, slightly different from the above closed-vocabulary setting. It can be seen that the superiority of our method is more obvious here. On the SUN RGB-D dataset, we achieve the AP novel improvement of over 5.94% compared to CoDA. On the ScanNet dataset, we achieve the 15.52% AP novel , surpassing CoDA by even more than 9%. This strongly demonstrates that our OneDet3D not only generalizes well across domains but also exhibits strong generalization ability at the category level. Compared to single-dataset training, multi-dataset training brings the over 1% improvement. This is because the utilization of multiple datasets enables to learn richer knowledge about categories. This experiment validates the universal ability of OneDet3D in terms of categories, which demonstrates its basic capability for universal 3D object detection.We further conduct cross-domain 3D detection experiments, with S3DIS and Waymo as new domains for inference. The comparison results are listed in Tab. 5 and Tab. 6. It can be observed that on S3DIS, in single-dataset training, our method already outperforms existing methods. This is because our language-guided classification better alleviates category conflicts. The performance is slightly better when trained on ScanNet because the ScanNet domain is more similar to S3DIS. After training on both datasets, the cross-domain AP on S3DIS improves by more than 4%, indicating the model ability to integrate information from both domain datasets. Furthermore, with the introduction of two outdoor datasets, AP 25 improves by 0.9%, with AP 50 remaining stable. This demonstrates that our OneDet3D can learn from such highly different domain point clouds for enhancement in cross-domain 3D detection. In outdoor point clouds, this is even more pronounced. KITTI is relatively similar to Waymo but only contains small-scale point clouds, while nuScenes is larger-scale but exhibits a larger domain gap. Training separately on these two datasets thus only yields limited cross-dataset AP 3D on Waymo. In comparison, through multi-dataset training, the model can utilize the characteristics of both, resulting in a substantial 23.1% improvement. This demonstrates the generalization capability of our method to unseen domains and further validates the necessity of multi-dataset training.We finally conduct ablation study in this subsection and list the results in Tab. 7 to evaluate our designs. Here we conduct multi-dataset training on the SUN RGB-D and KITTI datasets, and utilize S3DIS here for cross-domain evaluation. We also list the single-dataset training results as a reference. As can be seen, although our model architecture allows for multi-dataset joint training, directly multi-dataset training results in decreased AP on both seen and unseen domains, because of the interference problem. After introducing scatter partitioning, which alleviates interference among multiple domains during regularization, the model performance essentially matches and slightly exceeds that of single-dataset training. Then, through context partitioning, the detector can Finally, with languageguided classification, the effect of category conflict can be mitigated. Since the category conflict problem between SUN RGB-D and KITTI is not severe, the performance improvement on these two datasets is relatively moderate. The about 0.5% AP improvement here is primarily because of the common class-agnostic classification. In cross-dataset experiments on S3DIS, language embeddings contribute to a more AP increase, more than 2% improvement. This is mainly because of the more substantial overlap in categories between SUN RGB-D and S3DIS, making language embeddings more effective for these two domains. Such the ablation study thus demonstrates the necessity of these designs to address the interference issues for multi-dataset joint training.We provide visualized results from our OneDet3D in Fig. 4 . It can be seen that no matter for indoor or outdoor point clouds from different domains, OneDet3D can perform 3D detection effectively using only one set of parameters. This further demonstrates its effectiveness and universal ability. 
conclusion: 
 In this paper, we propose OneDet3D, a universal point cloud based 3D object detector that can generalize across various domains, categories, and scenes with only one set of parameters. The fully sparse structure and the anchor-free detection head serve as the basic model architecture. With partitioning in scatter and context, together with the language-guided classification, the interference caused by point clouds and categories can be alleviated. Extensive experiments demonstrate the strong one-for-all ability of OneDet3D. For the first time, we implement various scenarios and requirements of 3D object detection within a unified framework. This demonstrates that our OneDet3D has learned general 3D representations through multi-domain joint training, thus basically realizing the demands of universal 3D object detection and 3D foundation models. We believe that our research will stimulate following research along the universal computer vision direction in the future. Context partitioning. We discuss the design details about partitioning the learning of global context and list such ablation study in Tab. 8. As can be seen, without the partitioning, the model can already achieve the performance through multi-dataset training that is better than the single-dataset training baseline, partially due to our design of scatter partitioning. If we apply context partitioning for both indoor and outdoor point clouds, the 3D detection AP decreases on all datasets. Especially for the outdoor KITTI dataset, AP m decreases by 3.8%. This is because global information in outdoor scenes tends to be disruptive, given the excessive background points and the fact that foreground objects occupy only a small portion of the scene. In comparison, introducing context partitioning only for indoor point clouds can lead to a more than 1% AP improvement, and the cross-dataset AP improvement is 2.5%. This is because indoor scenes are smaller and the object distribution is more crowded, making global context relatively more important thus contributing to performance improvement. Therefore, the rationality of our context partitioning design is validated.Language-guided classification. We then discuss the design details about our language-guided classification and list the related 3D detection results in Tab. 9. The anchor-free detection head typically employs 3D sparse convolution for final classification. However, this approach struggles to address the category conflict issue among different domains. Especially during cross-dataset evaluation, differences in category definitions across datasets and the potential new categories in new domains can all restrict the model performance. Utilizing language embeddings as the classification layer can help mitigate this issue, due to the generalization ability of the text modality. However, directly using CLIP embeddings for classification results in a decrease in detection AP across all datasets. This is because a frozen fully connected layer impedes gradient backpropagation in the fully sparse convolution structure. Instead, when CLIP embeddings are trainable instead of frozen, the achieved AP can be comparable to or slightly surpass that of 3D sparse convolution. However, in this way, CLIP embeddings are not available at the inference time, and the model thus cannot generalize to new domains, especially new categories during inference.In comparison, what we utilize is the combination of 3D sparse convolution and frozen CLIP embeddings. 3D sparse convolution is utilized for class-agnostic classification, and is shared among all domains. The frozen CLIP embeddings are utilized for class-specific classification, and can benefit the model with the help of the text modality. As a result, both detection AP from the indoor and outdoor domains can be boosted. The cross-dataset AP on S3DIS increases the most, a 2% AP 25 improvement, because the category conflict problem can be alleviated. The effectiveness of our designs in language-guided classification is thus demonstrated. Classification loss. We then analyze the choice of the loss function for classification, and list the comparative results in Tab. 10. It can be observed that if we utilize focal loss for classification, which is widely utilized in the anchor-free detection head, the model cannot converge. This problem appears after the introduction of the language-guided classification. Since the class-specific classification is implemented by CLIP embeddings, which are frozen during training, the model requires a stronger optimization way for learning. The usual used focal loss thus becomes insufficient in this task. Utilizing soft focal loss, with IoU as the soft target can introduce positional information in classification, thus alleviating this problem. When using IoU 3D as the soft target, as IoU 3D is relatively hard to optimize, especially coupled with the classification task, the performance is still limited. The use of decoupled IoU proposed in [41] can alleviate this problem, by decoupling IoU calculation in the xy plane and the z axis. However, this decoupling is still not enough in our task, because the frozen CLIP embeddings make optimization more challenging. Therefore, we directly disregard the z direction here, and utilize IoU BEV here. The positional information in the xy plane can already provide sufficient supervision signals for classification, and disregarding the z direction also makes optimization easier. Ultimately, the model can conduct supervised learning with the language-guided classification, and achieve satisfying results on both indoor and outdoor point clouds. We further provide more visualized results on the indoor SUN RGB-D dataset (Fig. 5 ), indoor ScanNet dataset (Fig. 6 ), outdoor KITTI dataset (Fig. 7 ), and outdoor nuScenes dataset (Fig. 8 ). OneDet3D obtains satisfying detection results on all these four datasets, with only one set of parameters. Our method can accurately detect objects in various domain point clouds, ranging from indoor crowded scenes with overlapping objects to outdoor scenes with small-sized objects. This further demonstrates its effectiveness and universality. 
: 
 Figure 1 : The high-level overview comparing the multi-domain joint training performance of 10 existing 3D detectors and our OneDet3D. These models are jointly training on the indoor datasets SUN RGB-D (SUN), ScanNet, and outdoor datasets KITTI, nuScenes (nuS). We also evaluate the cross-domain performance on the indoor S3DIS and outdoor Waymo datasets. The center of the circle means that the corresponding metric is less than 10%, and the outermost means 90%. Existing indoor detectors are plotted in red, outdoor detectors are in green, and detectors that aim for different scenes are in orange. Our model has the remarkable capacity to generalize across a wide range of diverse 3D scenes (a larger polygon area) with only one set of parameters and the same architecture.The current trend in computer vision is to utilize one universal model to address all various tasks. Achieving such a universal model inevitably requires incorporating multi-domain data for joint training to learn across multiple problem scenarios. In point cloud based 3D object detection, however, such multi-domain joint training is highly challenging, because large domain gaps among point clouds from different datasets lead to the severe domain-interference problem. In this paper, we propose OneDet3D, a universal one-for-all model that addresses 3D detection across different domains, including diverse indoor and outdoor scenes, within the same framework and only one set of parameters. We propose the domain-aware partitioning in scatter and context, guided by a routing mechanism, to address the data interference issue, and further incorporate the text modality for a language-guided classification to unify the multi-dataset label spaces and mitigate the category interference issue. The fully sparse structure and anchor-free head further accommodate point clouds with significant scale disparities. Extensive experiments demonstrate the strong universal ability of OneDet3D to utilize only one trained model for addressing almost all 3D object detection tasks (Fig. 1 ). 3D point cloud based object detection aims to predict the oriented 3D bounding boxes and the corresponding semantic category tags for the real scenes given a point set. Unlike mature 2D detectors [29, 14, 38, 4] , which once trained, can generally conduct inference on different types of images in various scenes and environments, current 3D detectors still follow a single-dataset training-andtesting paradigm, i.e., point clouds used during inference should be from the totally same domain as that used during training. Whether indoor [25, 51, 44, 39] or outdoor [45, 16, 35, 32] , existing point cloud based 3D detectors can only be trained on datasets from one specific domain, then be tested on the same domain data. Such restriction of training and testing on a single dataset severely hampers the generalization ability of 3D detectors, resulting in a significant lag in the progress of 3D detection compared to 2D in terms of universality.To address this issue, multi-domain joint training (i.e., multi-dataset joint training) should be introduced into point cloud based 3D object detection, to allow 3D detectors to learn from point clouds of different domains through large-scale joint training. In this way, a 3D detector, once trained, can well generalize across various domains of point clouds. The ultimate goal is to obtain a 3D detector that can support unified 3D object detection across different domains with only one set of parameters, thereby achieving the target of universal 3D object detection.The motivation of multi-domain joint training is to learn universal 3D knowledge by leveraging point clouds from different sources and domains, thereby establishing a general representation from 3D data to 3D spatial positions. Through this, a model independent of point cloud source, collection, and domain can be achieved. With common 3D knowledge from diverse point clouds, it can effectively serve as a universal 3D detector and function as a 3D foundation model. However, achieving this is highly challenging and difficult. As can be seen in Fig. 1 , due to the significant domain gaps (e.g., point cloud ranges, scenes, object sizes, sparsity, etc.), existing 3D detectors fail to support this. Specifically, current 3D detectors can be generally divided into point-based and voxel-based ones. For point-based 3D detectors (the upper part of Fig. 2a ) [25, 51, 21] , it is difficult to apply the same sampling and grouping technique for different domain data. For voxel-based 3D detectors (the lower part of Fig. 2a ) [45, 47, 17, 41] , which usually require converting from sparse to dense features for 3D box prediction, the scale differences between indoor and outdoor point clouds make it difficult to represent them using dense features of the same size. This consequently limits existing models to learning domain-specific knowledge, restricting their ability to acquire generalized 3D knowledge.In this paper, we propose OneDet3D, a unified point cloud based 3D detector with only one set of parameters through multi-domain joint training. As in Fig. 2b , we employ 3D sparse convolution for feature extraction, which is more robust to domain gaps compared to point-based feature extractors [26, 27] , making it well-suited for adapting to point clouds from different domains. Subsequently, we utilize an anchor-free detection head, where objects are represented by center points [38, 47, 30] , enabling direct compatibility with sparse convolution and avoiding the constraints of fixed-size dense features. Such a fully sparse structure, together with the anchor-free detection head using center point representation, provides an effective architecture for multi-domain joint training.Based on the model architecture, during multi-domain joint training, the domain-interference issue should be further addressed. This issue primarily comprises two aspects: data-level interference caused by differences in point clouds themselves, and category-level interference caused by label conflict among categories across different domains. To mitigate the data-level interference, we employ domain-aware partitioning, which partitions parameters where the interference problem mainly exists to be domain-specific and keeps the vast majority shared among different domains. The data-level interference can thus be effectively prevented without increasing the model complexity too much. Specifically, we partition re-scaling in normalization layers to maintain the consistency of the data scatter, and parameters about context learning for reducing the effect of range disparities. They are guided by a domain router implemented by a domain classifier. To alleviate category-level interference, we employ language-guided classification, leveraging the text modality to alleviate conflict issues. We utilize a combination of fully connected layers and sparse convolution for class-specific and class-agnostic classification to ensure compatibility with the anchor-free head.Our main contributions can be summarized as follows:• We propose OneDet3D, a multi-domain point cloud joint training model for universal 3D object detection. To the best of our knowledge, this is the first 3D detector that supports point clouds from domains in both indoor and outdoor simultaneously with only one set of parameters. Extensive experiments demonstrate the one-for-all ability of our OneDet3D. OneDet3D possesses the strong generalization ability in both category and scene, thus effectively achieving the goal of universal 3D object detection. In the close-vocabulary setting, it achieves comparable performance using only one set of parameters. In the open-vocabulary setting, it obtains more than 7% performance. 
 
 Appendix: Limitation and Potential Negative Social Impacts. Our approach addresses the multi-domain joint training issue in point clouds, aiming to achieve a universal 3D detector. Currently, it focuses on supervised learning. Its utilized point clouds should be fully-annotated, with both categories and 3D boxes. In the future, we plan to incorporate weakly-labeled or unlabeled data to enable the model to conduct larger-scale learning. Additionally, since our model can utilize various sources of point clouds for training, the misuse of collected data could potentially lead to negative social impacts. 


Decision: The paper receives two positive and one negative ratings. Overall, the main concerns from the reviewers are some technical clarifications (e.g., how to handle unseen domains, dataset sampling/arrangement) and more comparisons with the prior work. After the rebuttal, two reviewers find that the major concerns are addressed well, where one reviewer upgraded the rating to borderline accept. For reviewer PVw6 who did not update the comment, the AC took a close look and find that all the major comments are well addressed in the rebuttal, including the technical comparisons with the prior work, more comparisons with 3D object detectors and cross-dataset setting with unseen domain. After reading the paper and reviews, the AC finds the merit of paper solving an important task for multi-domain 3D object detection, with well supported technical contributions and extensive experiments. Therefore, the AC recommends the acceptance decision and encourages the authors to incorporate the suggestion from the reviewers in the final version, as well as releasing the implementation to foster the development of this field.

Reviews: review: paper_summary: paper_summary: This manuscript introduces OneDet3D, a universal point cloud-based 3D object detector designed to address the challenges of multi-domain joint training. The primary motivation is to overcome the limitations of existing 3D detectors, which are typically trained and tested on single datasets, restricting their generalization across different domains. OneDet3D aims to provide a unified solution that can handle diverse indoor and outdoor scenes using a single set of parameters.

The authors claimed the following contributions of OneDet3D:

- Domain-Aware Partitioning: This technique aims to address data-level interference caused by differences in point cloud structures across domains. The parameters related to data scatter and context learning are partitioned and guided by a domain router, allowing the model to learn domain-specific features without increasing complexity.

- Language-Guided Classification: By incorporating text modality through CLIP embeddings, OneDet3D mitigates category-level interference among different datasets. This approach allows for better generalization to new domains and categories.

- Fully Sparse Architecture: The use of 3D sparse convolution and an anchor-free detection head makes the model robust to domain gaps and efficient in handling point clouds from various domains.

Experiments on datasets like SUN RGB-D, ScanNet, KITTI, and nuScenes demonstrate the effectiveness of OneDet3D. The model achieves promising performance in both closed-vocabulary and open-vocabulary settings, showing improvements over existing methods and strong generalization abilities.
strength_weakness: strength_weakness: (-) The overall OneDet3D framework is a combination of several previous approaches, which might not demonstrate a strong novelty in the related area:
- The domain router and context partitioning from Sec. 3.2 is similar to what [R1] and [R2] did for reducing domain differences. 
- The scatter partitioning method in Sec. 3.2 is closely related to Uni3D [R3] (see their Sec. 3.3). 
- The language-guided classification in Sec. 3.3 shares the same intuition with [R4] and [R2].

(-) The experimental comparisons could benefit from involving more recent 3D object detectors. For example, the latest model in Tab. 2 (UVTR) is from two years ago; while most of the other models are from 2020 (or even earlier).

(-) The overall elaboration of this manuscript deserves further improvements. Several claims are made without supporting evidence or references. Besides, most of the technical details are given in the text; having more graphical illustrations or algorithm flows could reduce the redundancy in the elaboration and help readers better understand the proposed method.

---
### References:
- [R1] Towards Universal Object Detection by Domain Attention. CVPR, 2019.
- [R2] Multi-Space Alignments Towards Universal LiDAR Segmentation. CVPR, 2024.
- [R3] Uni3D: A Unified Baseline for Multi-Dataset 3D Object Detection. CVPR, 2023.
- [R4] DaTaSeg: Taming A Universal Multi-Dataset Multi-Task Segmentation Model. NeurIPS, 2023.
questions: questions: - **Q1:** As mentioned in Weakness 1, it is recommended to make a clearer comparison and have additional discussions with closely related works from existing literature. Putting the Related Work section behind the Introduction section and adding more detailed analyses and discussions with existing works can be beneficial. Including experimental comparisons and ablation studies with existing works, such as Uni3D, could further justify the effectiveness of the proposed OneDet3D.

- **Q2:** As mentioned in Weakness 2, supplementing the results with more recent single-dataset training approaches could further improve the comprehensiveness of the benchmark studies.

- **Q3:** As mentioned in Weakness 3, the manuscript could benefit from having more graphical illustrations or algorithm flows, instead of just plain text descriptions. This is also in line with the style of ML conferences.

- **Q4:** For the scatter partitioning method in Sec. 3.2: Since the dataset-specific statistics are used, how does OneDet3D handle a new point cloud with unknown statistics during inference?

- **Q5:** How does OneDet3D handle the class mapping differences using the language-guided classification in Sec. 3.3? For example, how to handle cases like the different definitions of `bicycle` and `bicyclist` (in KITTI and Waymo)?
limitations: limitations: The authors acknowledge several limitations in their approach: The current focus on supervised learning with fully annotated point clouds limits scalability. Future work should explore weakly labeled or unlabeled data to reduce reliance on extensive annotations.

Additionally, the manuscript lacks a detailed analysis of the computational cost associated with the proposed methods, which is crucial for assessing real-time application feasibility. What is more, there is a risk of overfitting to seen domains during multi-domain training. More experiments on unseen domains would strengthen the claims of generalization. The scalability and generalizability of OneDet3D to other types of sensors and different environmental settings have not been extensively discussed. Further research is needed to assess the model's robustness in diverse real-world scenarios.
paper_summary: paper_summary: This paper proposes OneDet3D, which is a multi-domain jointly trained point cloud object detector for universal 3D object detection. It is the first 3D detector that supports point clouds from both indoor and outdoor scenes simultaneously with only one set of parameters. The experiments are conducted on multiple indoor and outdoor benchmarks.
strength_weakness: strength_weakness: - The performance in Table 2 for the nuScenes dataset is strange, where AP is not aligned with the numbers reported by other methods.

- The sizes of the adopted datasets are quite different. Are any resampling strategies adopted? How to mix them up? The training scheme and implementation details should be added, such as training schedule, data augmentation, and so on.

- The attributes of point clouds are different. For example, indoor datasets may contain RGB information and outdoor data such as Waymo may contain elongation and timestamp information. How to deal with the different channel sizes? A lot of details are ignored by the authors.

- There are many basic grammar issues. For example, "is joint training on" should be "is jointly trained on".

- I notice the authors claim and adopt fully sparse architecture. However, there is no comparison or discussion with previous work with fully sparse structures such as [1 - 4]. What are the differences between the proposed architecture and theirs? Do the issues they addressed still exist? 

- Sec. 3.3 is hard to follow and should be rewritten with more details and sufficient explanation. (1) The description of "we utilize language vocabulary embeddings from CLIP" should be more detailed. Does the network predict CLIP embedding? How to supervise?  (2) Why is it necessary to convert sparse point features to dense features in Sec. 3.3?  (3) What is the specific definition of the dense features?  (4) Why is the back propagation unfeasible? (5) How many classification branches does it have? What does the "both branches" in L215 mean? 

- Lack of comparison with other similar methods such as [5,6] in Table 4 & 5. \

[1] Fully Sparse 3d object detection \
[2] Voxelnext: Fully sparse voxelnet for 3d object detection and tracking \
[3] FSD V2: Improving Fully Sparse 3D Object Detection with Virtual Voxels \
[4] SAFDNet: A Simple and Effective Network for Fully Sparse 3D Object Detection \
[5] ST3D: Self-training for Unsupervised Domain Adaptation on 3D Object Detection \
[6] Uni3D: A Unified Baseline for Multi-dataset 3D Object Detection
questions: questions: See the weakness part. I would like to increase the score if my concerns are well addressed.
limitations: limitations: The authors do not discuss the limitations and social impacts, which should be added.
paper_summary: paper_summary: This paper proposes OneDet3D, a universal one-for-all model that addresses 3D detection across various domains, including both indoor and outdoor scenes. It tackles two primary issues: data-level interference caused by differences in the point clouds themselves and category-level interference caused by label conflicts among categories across different domains.
strength_weakness: strength_weakness: 1.	In lines 60-62, the authors claim that 3D sparse convolution is better than point-based feature extractors due to its robustness to domain gaps. Could the authors elaborate on this in detail?
2.	Regarding the domain router, how is the domain label n obtained? Does data from the same dataset share the same domain label?
3.	In Table 1, different datasets have different views. I wonder if any design could be implemented to tackle this difference for better generalization ability.
4.	In Equation 1, experiments regarding the hyperparameters α and c are missing. The values of these hyperparameters are not mentioned.
5.	In Table 2, the best performance achieved by existing methods is not highlighted in bold.
6.	Table 4 underlines the second-best performance, while Table 5 does not. Table 4 underlines the second-best performance, while Table 5 does not. It would be beneficial to maintain consistency.
questions: questions: Please refer to the weakness section.
limitations: limitations: N/A

Comments: title: Looking forward to Feedback as Discussion Deadline Approaches
comment: Thanks for your thorough reviews, which are very helpful to improving the quality of our paper. We apologize for any inconvenience caused, but as the deadline for discussion (Aug 13 11:59 pm AoE) draws near, we would like to provide an update on our progress.

If you need further clarification or have additional questions, please don't hesitate to contact us. Again, we sincerely thank you for your time and effort in reviewing our paper.

Thanks
Dear Reviewer PVw6:

We thank you for the precious review time and valuable comments. We have provided corresponding responses and results, which we believe have covered your concerns. We hope to further discuss with you whether or not your concerns have been addressed. Please let us know if you still have any unclear parts of our work.
Thank you for providing the additional experiments and detailed explanations in your rebuttal. It addressed my concerns effectively.
Thanks so much for your response. It addressed most of my concerns. I appreciate the efforts to implement fully sparse methods such as voxelnext and fsdv2 in the indoor datasets, which are supposed to be added into the main paper. I would like to increase my score.