Towards 3D Objectness Learning in an Open World
Authors: Taichi Liu
Zhenyu Wang
Ruofeng Liu
Guang Wang
Desheng Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Open world
3D Object Detection
Primary area: applications

Abstract: Recent advancements in 3D object detection and novel category detection have made significant progress, yet research on learning generalized 3D objectness remains insufficient. In this paper, we delve into learning open-world 3D objectness, which focuses on detecting all objects in a 3D scene, including novel objects unseen during training. Traditional closed-set 3D detectors struggle to generalize to open-world scenarios, while directly incorporating 3D open-vocabulary models for open-world ability struggles with vocabulary expansion and semantic overlap. To achieve generalized 3D object discovery, We propose OP3Det, a class-agnostic Open-World Prompt-free 3D Detector to detect any objects within 3D scenes without relying on hand-crafted text prompts. We introduce the strong generalization and zero-shot capabilities of 2D foundation models, utilizing both 2D semantic priors and 3D geometric priors for class-agnostic proposals to broaden 3D object discovery. Then, by integrating complementary information from point cloud and RGB image in the cross-modal mixture of experts, OP3Det dynamically routes uni-modal and multi-modal features to learn generalized 3D objectness. Extensive experiments demonstrate the extraordinary performance of OP3Det, which significantly surpasses existing open-world 3D detectors by up to 16.0% in AR and achieves a 13.5% improvement compared to closed-world 3D detectors.

Towards 3D Objectness Learning in an Open World
Taichi Liu', Zhenyu Wang", Ruofeng Liu², Guang Wang', Desheng Zhang'
Rutgers University "Tsinghua University "Michigan State University *Florida State University taichi liu@rutgers edu, wangzy20@tsinghua. edu.
liuruofedmsu.een edu, guang wang@fsu edu, deshengâcs rutgers. edu op3det.github.io
Abstract
Recent advancements in 3D object detection and novel category detection have made significant progress, yet research on learning generalized 3D objectness remains insufficient. In this paper, delve into learning open-world 3D objectness, which focuses on detecting all objects in a 3D scene, including novel objects unseen during training. Traditional closed-set 3D detectors struggle to generalize to open- world scenarios, while directly incorporating 3D open-vocabulary models for open- world ability struggles with vocabulary expansion and semantic overlap. To achieve generalized 3D object discovery, we propose OP3Det, glass-agnostic Open- World Prompt-free 3D Detector to detect any objects within 3D scenes without relying on hand cradfed text prompts. We introduce the strong generalization and zero-shot capabilities of 2D foundation models, utilizing both 2D semantic priors and 3D geometric priors for class-agnostic proposals to broaden 3D object discovery. Then, by integrating complementary information from point cloud and RGB image in the cross-modal mixture of experts, OP3Det dynamically routes uni-modal and multi-modal features to learn generalized 3D objectness. Extensive experiments demonstrate the extraordinary performance of OP3Det, which significantly surpasses existing open-world 3D detectors by up to 16.0% AR and achieves 13.5% improvement compared to closed-wor world 3D detectors.
1 Introduction
In 3D perception systems, especially in real- world environments such as autonomous driving and robotics, object categories of interest may change dynamically. This has led to increasing attention challenging tasks like distrf-diittibution 3D detection [1, 2], open-world 3D detection [3, 4] and open- vocabulary 3D detection [5, 6, 7, 8, 9, 10]. improving generalization beyond closed-set set assumptions. core challenge across these tasks is the ability to localize all objects, which lies in understanding how objects structured in 3ID scenes to distinguish them from the background. While extensive efforts have been made to identify unknown or novel objects in the 2D domain [11, 12, 13] known as class-agnostic object detection (OD) [14], such exploration the 3D domain remains limited. To bridge this gap, we introduce learning 3D objectness in a class-agnostic paradigm, enabling models to detect and discover objects beyond known categories. Therefore, our goal is to achieve class-agnostic 3D object detection, where objects are identified and localized based on their intrinsic properties rather than pre-defined semantic labels, thus supporting open-world world perception.
In a class class- aggostic manner, ensuring a high recall rate is essential, as it ensures that the majority of objects in the scene detected regardless of their semantic categories. This serves as foundation for accurate category assignment and significantly contributes to object detection for categories of interest [7, 15]. Although current point-cloud-basee
3ID detectors [16, 17, 18, 19, 20] have
achieved significant success in 3D benchmark datasets [21, 22, 23, 24], simply shifting from class-
specific to class-agnostic classification is ineffective. This is primarily because 3D point cloud
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Pre-detined catpories/
Prompt-Fre
Test promppi
30 pain douds
3D plontccluuss clouds
Model
OP3Det
RGB mages
RGB mages
(a) existing 3D object detectors
(b) OP3Det (ours)
Figure I: Illustration for Prompt-free 3D objectness learning. (a) Closed-world detectors can only recognize pre-defined categories in the training dataset (yellow boxes). Although some 3D detectors can detect novel classes via pre-defined prompts (green boxes), they still cannot detect "all" when the given vocabularies are limited. (b) In comparison, our OP3Det can detect rare categories (red boxes) and better discover 3D objects without the requirement of any semantic labels and text prompts.
data are extremely limited in both the scale of data and annotated categories. Moreover, directly employing open-vocabulary 3D models for class-agnostic detection f'aces significant challenges due to vocabulary expansion and semantic overlap [25] in hand-crafted text prompts, making them ineffective for novel object discovery and preventing the learning of open-world 3D objectness, as can be seen in Fig Therefore, learning 3D open- world objectness and achieving strong localization generalization is highly challenging. In contrast, the 2D domain is far more resource-rich in both models and data. Plenty of pre-trained foundation models [26. 27] and the detectors trained on extensive vocabularies [28, 29, 30] with broad classes [31, 32, 33] demonstrate strong generalization capability. Our intuition is to transfer strong zero-shot abilities from 2D pre-trained models to the 3D domain, exploring its generalization ability for 3D object discovery and 3D objectness learning.
We propose OP3Det, class-agnostic Open-Worte World Prompt-frce 3D Detector, which exploits extensive 2D semantic knowledge to learn open- open-world 3D objectness. Here, prompt-free means that our method requires no text prompts or any semantic priors as inputs at inference time, making semantic prompt-free the model directly learns 3D objectness from geometric and visual cues. More specifically, we primarily use the large 2D foundation model Segment Anything Model (SAM) [27], to extract abundant and generalizable class-agnostic object masks in a scene. However, SAM often produces fragmented masks or partial object masks, which will severely hinder the learning of whole objectness [34, 35]. To address this, we adopt multi-scale point sampling strategy that considers 3D spatial proximities to refine the uniformly distributed point prompts provided to SAM, enabling more accurate extraction of class-agnostic object bounding boxes. Through semantic and geometric cues, a greater variety of novel objects can be discovered effectively, which are subsequently projected into the 3D space for 3D object discovery in point clouds prior training.
To better learn 3D objectness during the training phase, we further leverage 2D semantic knowledge and integrate both point cloud and RGB image modalities for multi-modal training. Prior works have explored fusion at various levels, including peint-level [36, 37], feature-level [38, 40), and object-level [41, 42, 43, 44]. Although these methods have shown strong performance, they often rely heavily on fused features, while overlooking the importance of preserving modality-specife informative cues. We thus propose the cross-modal mixture of experts (MoE) to effectively connect both intra-modal and inter-modal information. Specifically, we use the self-attention structure to encode uni-modal and multi-modal features. Through a multi-modal router and modal-specific experts, OP3Det dynamically fuses uni-modal or multi-modal features, ensuring that the most relevant information can be adopted. The model can adapt its strategy according to the specific demands of each scenario, whether it requires a stronger reliance on 2D semantic information from images, 3D geometric cues from point clouds, balanced integration of both modalities.
Our main contributions can be summarized as follows:
We introduce a novel and practical problem setting, class-agnostic open-world 3D object detection, which aims to detect all objects in 3D scene and reflect real open-world environments. To the best of our knowledge, we are the first to formally define and address this problem in the 3D domain.
We propose OP3Det, multi-modal 3D detector for learning open-world 3D objectness. A multi- scale point sampling strategy is designed to enhance 2D-3D association and reveal broader range of object instances for effective oprn-world 3D object discovery.

---

3D Object Discovery
Multi-modal Imput
Cross-modal Mixture of Experts Fusion
Class -gn-stic OD
F
Mult-V/ddl
Bouder
Encoder
Trandiiinn
munti scale sompiing
Reg Regresdon
RtB image
Export
Mod Mooeecce Sum
Festure
F
SAM
Semantic
class
Eroodar
Verel
Fp
Object Clasificatin
Attention
Geometricc
Expert
Troiniing only
Point
Figure 2: The overview of OP3Det. We apply SAM to introduce abundant 2D semantic knowledge for 3D object discovery. Multi-scale point sampling is utilized in this process. The cross-modal MoE is then employed to guide data pathways for uni-modal and multi-modal features, allowing the model dynamically adapt its reliance on unimodal or cross-modal information according to the scenario.
We design cross-modal miixtuee-ff-expeee (MoE) module guided by a multi- router, which dynamically selects between uni-modal and multi-moda pathways to adaptively learn 3D objectness under diverse open- world scenarios.
Extensive experiments demonstrate the ability of our OP3Det to detect in the open world. OP3Det possesses a strong generalization ability in both cross-category and cross-dataset settings. It achieves 27% improvement for novel class discovery compared to the baseline method. The adaptability of our method also makes it casily extendable outdoor scenes, class-specific detection or the 2D domain.
2 Related Work
Open-world 3D scene understanding, Open-world 3D learning aims identify and detect 3D objects from an arbitrary set of categories, allowing models to generalize to novel object categories that are not present in the training data. Recent methods [45, 46, 47, 48, 49, 50] have conducted spen-world learning for 3D segmentation. However, these methods rely on precise mask-level annotations for geometric information learning. Open-vocabulary 3D detectors [5, 6. 7, 9, 10] usually use RGB images and pre- treitri 2D models [26, 27] to enrich semantic information for recognizing novel categories. Despite their success, these methods rely on a pre-defined vocabulary as input for detection, rather than truly learning objectness. When the vocabulary is incomplete or mismatched with the scene, they still fail to detect all objects, resulting classes. In comparison, we formally explore open-world 3D objectness learning in a class-agnostic way, aiming to detect all salient objects in a scene without relying on a fixed label set or text prompts.
low recall in novel
Applications of SAM in 3D scenes. The strong zero-shot generalization capabilities of SAM have motivated its adoption in 3D scenes. Previous works leverage SAM to generate fine-graines 3D masks for 3D segmentation. SAM3D [46] and Segment3D [51] both used a bottom-up framework that applied SAM to RGB-D images to obtain 2D masks, which are then projected into 3D space for su- pervised training. In contrast, methods such as REAL [52], SAM- [53] and OpenMask3D [54] adopt a top-down strategy. They utilize projected 3D labels as prompts guide SAM in generating more accurate 2D masks, which are subsequently back-projected to produce dense or diverse 3D annotations. However, the segments from SAM are not solely focused on objects. Directly using SAM will introduce noise into the generated 3D labels. Our method apply SAM for 3D object detection in open-world learning. By climinating the need for 2D text and 3D labels as prompts, we enable scalable training and generalization to unseen objects in an open world.
2D and 3D Feature Fusion. Existing multi-modal fusion methods can be generally divided into point-level, feature-level and object-level categories. Point-level fusion introduces 2D features directly in the 3D domain. PointPainting [36] and PointAugmenting [37] enhance LiDAR- based 3D object detection by enriching point cloud features with image semantics. Feature-level fusion integrates multi-mode features using shared representation spaces or attention-based modules, like BEVFusion [38. 39] projecting LiDAR and image features into the BEV space. Object-level fusion integrates codallit-sspccfffe information the instance level. SparseFusion [41] fuses instance-level sparse features from both 2D and 3D inputs, and ObjectFusion [42] uses heatmap- proposal
3

---

generator to align object-centric features. However, these methods often rely heavily on fused features, while overlooking the need to preserve modality-speciiff critical cues. OP3Det adaptively filters irrelevant cross-mddal features while preserving and enhancing informative intra-modal signals.
3
Method
We formulate the class-agnostic open-world 3D object detection task in Sec. 3.1. Fig. 2 shows the overall architecture of the proposed OP3Det. To achieve open- world and class-agnostic 3D detection, OP3Det learns 3D objectness through two key components: (i) a 3D Object Discovery strategy (Sec. 3.2) that expands the set of potential 3D objects and (ii) cross-modal MoE module (Sec. 3.3) that dynamically fuses semantic and geometric representations for robust 3D objectness learning.
3.1 Problem Formulation
In our work, 3D objectness denotes the likelihood that a spatial region corresponds to physically discrete object, distinguishable from background structures or noise, regardless of semantic category. Formally, let (-) denote a learnable model that maps input features Vimput to an objectness confidence score. The 3D objectness learning can be expressed as: I [o ( Pimput) > r] where T is confidence threshold for classifyyng a spatial region as foreground, and I[-] is the indicator function (1 denotes a valid object region and O denotes background). The model ø is trained to approximate this decision function, assigning high confidence to true object regions and low confidence to background or noise.
Given a point cloud Xp and corresponding RGB images X,, the training data contain annotated 3D bounding boxes {(, bb,)) ) where C and bi: are the objectness label and 3D bounding box of the i-th object, M is the number of 3D boxes. Our goal is to leverage the paired multi-modal input (Xp, Xr) as input features and localizing all object instances during inference, including novel and unseen categories.
with the b6; learn detector capable of discovering
For multi-modal training, the 3D point cloud features Fp XxX are extracted through the voxel-based backbone and the image backbone separately, Fj is then projected into the 3D voxel space for the image features in the voxel space F/ E R X Denote the camera intrinsic matrix as K and the extrinsic matrix as R. then the corresponding positions in the 2D image
and 2D image features F3
be obtained by projecting 3D positions in the 3D voxel space through KR, We concatenate these two features to obtain the multi-modal features: FM = [Fp, Fj). For multi-modal fusion, we propose the rross-modal MoE fuse Fp, Fy, FM. in order to integrate 2D semantic, 3D geometric, and multi-modal information in the training phase.
3.2 3D Object Discovery
3D object discovery enables the discovery of novel objects prior to training. To achieve this, we leverage cross-priors from both the 2D and 3D space. In terms of the 2D domain, we utilize the Segment Anything Model (SAM) [27], which is trained on extensive 2D datasets and thus demonstrates strong zero-shot generalization performance across various scenarios. We apply SAM inference, obtaining series of class-
directly on RGB images X, to conduct regment-anythiin agnostic masks. Due to the rich semantic information from SAM, these masks often cover a broader range of objects, thus significantly addressing the limitations of object information in 3D datasets. The segment-anything inference process employs regular 64x64 grid of points {(x: v)} as non-semantic point prompts, which serve as inputs to SAM to obtain segmentation results.
However, SAM, as segmentation model, often leads to fragmented outputs or object parts and sub-parts in the final mask outputs. As illustrated in Fig. 3a, this produces large number of chaotic masks, introducing significant noise into the final annotations. This severely impedes class-agnostic detection, which generally targets entire objects at global level. To address this issue, we propose a multi-scale sampling strategy, guided by per- per-point object prior probabilities. We begin by selecting a source point (I,, ys) from the point set that is most likely to be related the object, according to the IoU score from SAM and attention values from suff-supervised model as 2D object priors. Then we filter neighboring points whose 3D distances to the selected point are within a threshold ensuring that local geometric consistency is preserved-a property that cannot be reliably derived from 2D image alone. Specifically, to obtain the 3D distance between the point (x,, y) and the source point (I., ( Va), we project all 3D points onto the 2D image plane through K R.. Then, we select

---

(a) SAM result
(b) Point sampling,
(c) Point sampling,
(d) Multi-scale sampling class-agnostic detector
0.2
5
Figure 3: Visualization of point sampling trategy. The segmentation masks from SAM contain many small fragments and object parts. By using multi-scale point sampling, these noisy masks can be mitigated. Post-processing with 2D class-agnostic detector further improves the quality.
the 3D points (z/, 1/ z²) and (I', V4 =({) whose projected points are closest to (I, y) and (Ia, Ua) respectively. The 3D distance between (x, y) and (I.: ys) is actually the 3D distance between (z', 2') and (z's V., The iteration continues until no further points selected. In this way, points that are too close to each other and with low object prior values are filtered out, enabling the elimination of many overly small object masks object parts, thus reducing noise, which can be seen in Fig. 3b and Fig. 3c.
The choice of 5 is a crucial parameter, as it affects the scale of the final masks. When ō is too small, filtering of object parts may be insufficient, while a large å may lead to the exclusion of useful objects. To address this, we use series of 8 values ranging from small to large, (0.2,0.5,1,2) in our experiments specifically, and combine their results through NMS. Such a multi-scale point sampling strategy captures the advantages of different scales, yielding more reliable segmentation results.
Finally, further filter out the remaining small noise masks, such as tiny fragments left when is small, and to enhance object localization, we pass the segmentation results through pre-trained class-agnostic 2D detector [55]. Since this 2D detector is trained in a class-agnostic way, it focuses on the localization information and is sensitive to complete object boundaries. Thus, it can effectively help determine whether each mask represents a whole object. The object masks and bounding boxes will also be adjusted according to the bounding box regression of such 2D detector. For objectness prediction, we multiply the IoU prediction scores from SAM and the objectness scores from the class-agnostic 2D detector to obtain the updated scores. We then filter low-score object masks based on such updated scores to ultimately reduce noisy masks, as is illustrated in Fig. 3d. These 2D boxes ultimately projected into the 3D space through K R, for 3D object discovery.
3.3 Cross-Modal MoE
In the previous subsection, we primarily focus on 3D object discovery in point clouds prior to training. Further, during the training process, we continue to exploit semantic knowledge, integrating geometric information from 3D point clouds facilitate 3D objectness learning. Therefore, we employ the multi-modal training approach, using both point clouds and RGB images for 3D object detection.
In the closed-world setting, directly using multi-modal features FM for detection can already lead to performance gains [56, 38). However, this does not work in the open-world setting. This is because in the class-agnostic binary classification mode, recognizing different objects also heavily relies on geometric information, which is widely present in point cloud features Fp. Furthermore, certain multi-modal scenes may be dominated by a single modality, leading to incomplete spatial understanding under occlusions or restricted viewpoints, making it essential to incorporate effective modal interactions to fully exploit the strengths of each individual modality. To address this, we propose a cross-modal Mixture-of offture-of-EExprrts (MoE) module that selectively guide the data pathways of 2D semantic features, 3D geometric features, and mudai- fused features, achieving dynamic multi-modal fusion to boost 3D objectness learning.
We first utilize the self-attention [57] module uni-modal and multi-modal features, respectively, on its spatial dimensions. This enables the detector to concentrate on the important spatial regions in
The operation is that we project 3D points into the 2D space using K Rs, finding points within the 2D box, then clustering them to obtain the 3D box (466, 312 ) This be viewed as 2D-+3D projection.
5

---

the features for the subsequent 3D detection, thus extracting important features for each modality: Fp SelfAttn(Fr). and Fr. FM are defined and obtained in the same way.
Then, we utilize the multi-modal router to obtain the routing probability pr, Pr,Pi, Pr, for different modality features, guiding the data pathways for each modality. This router consists of 31) convolution layer, a global average pooling layer, a fully connected layer, and the final softmax. Denote the router as R. this process can be denoted as:
Dr,,Pi,Pe PMr) R(FM)
Guided by these routing probabilities, we finally apply semantic expert E₁, geometric expert Ep, and fused expert EM. We implement experts through three 3D convolution layers with kernel sizes of 1, 3 and 1 sequentially. The specific process is as follows:
F =
E Pr (.(FF
(E(PJI,M
Finally, F is fed into the detection head, where we adopt the 3D detection transformer [58, 15], As F represents a synthesis of Fp, F5 and FMr the model can dynamically adjust the data pathways based on the input data for dynamic multi-modal fusion, ensuring that the most suitable features can be utilized for the final detection. This thus enhances multi-modal class-agnostic detection.
3.4 Training and Inference
Training. We use RGB images and point clouds pairs to guide the training of our class- agnossic 3D network. For each image, 3D object discovery is performed using the SAM [27], selected for its large-scale open-world world training, remarkable zero-shot generalization to unseen objects, and class- agnostic design. With both annotated and discovered 3D bounding boxes enriched by corresponding RGB images, we then employ the Cross-Modal MoE to train a multimodal 3D detector capable of learning class-agnostic objectness across modalities. Ultimately, the learning loss function of OP3Det primarily follows the loss function in [9]. To better suit our task, the classification loss is formulated as class-agnostic binary classification loss. For 3D scenes with multi-view images, we extract features from each view and project them into the voxel space using their respective projection matrices. The projected features are then aggregated to the multi-modal representation.
During training, our model utilizes 2D images to discover potential objects and provide semantic supervision for 3D objectness learning. During inference, it performs detection directly on point cloud-image pairs, requiring no additional stages or external modules beyond a standard multi-modal 3D detector. The learned cross- MoE further enables class-agnostic 3D objectness inference in a fully prompt-free manner.
Inference.
Experiments
Datasets. For indoor scenes, we utilize SUN RGB-D [21] and ScanNet V2 [22] datasets. SUN RGB-D contains 46 classes, while ScanNet V2 contains 200 categories in total [59]. We mainly follow the setting of [7] for category splitting. Specifically, for SUN RGB-D, the categories with the top 10 most training samples are selected as base (seen) categories, while the remaining 36 are novel classes. For ScanNet, we also adopt the same setting, using single-view small scenes corresponding to individual images for training. The top 10 classes are utilized for base classes and the other 50 ones for novel classes. Their category labels are removed during training for class laas-tgnootic classification. For outdoor 3D detection, we mainly conduct experiments on the KITTI [23] dataset. We treat the car class as the base class and the cyclist and pedestrian classes as novel classes. We mainly utilize its official metric, the AP7o metric with 40 recall positions for evaluation.
Since the target is to identify all objects within a scene for 3D objectness learning and 3D object discovery, and not all bounding boxes are necessarily annotated in the test set, we mainly employ Average Recall (AR) under loU thresholds of 0.25. Average precision (AP) is also utilized. However, under class-agnostic binary classification, AP for base and novel classes cannot be straightforwardly computed, so we only report AP across all categories. For more discussions and experimental results about the AP metric, please refer to the Appendix C.

---

Table I: The cross-category performance of OP3Det on the SUN RGB-D and ScanNet dataset.
Clored-world 3D detection methods are trained on 3D point clouds with only seen categories annotated. Open-vocabulary methods are trained on 3D point clouds with class-specific 3D bounding boxes for annotations, thus requiring more information compared our class-agnostic setting.
SUN RGB-D
ScanNet
Method
ARnovel
ARmovel wooled-world 3D object detection methods
AR
ARtase
APaul
ARai
ARbase
APall
VoteNet [16]
33.7
68.3
79.1
55.1
35.3
44,6
56.1
13.8
GroupFree [60]
41.8
69.9
78.7
49.2
32.1
40.9
51.8
9.4
FCAF3D [17]
65.3
86.5
92.7
62.0
61.7
71.3
83.2
24.7
Uni3DETR [15]
51.8
82.1
91.6
61.3
54.6
67.6
80.1
16.9
Tr3D [56]
62.1
84.8
91.9
53.4
47.1
58.1
71.6
17.2
open-vocabllary 3D object defection methods
Det-PoiniCLIPv2 [8]
22.4
31.1
64.5
10.2
33.1
38.7
55.9
3.1
3D-CLIP [26]
23.6
32.3
66.8
25.7
32.9
36.2
55.5
5.6
CoDA [7]
33.9
60.2
71.5
48.2
44.3
53.4
68.3
23.9
OV-Uni3DETR [9]
62.8
82.5
88.8
57.4
67.6
71.6
76.5
25.9
ImOV3D [61]
46.9
63.1
74.1
28.3
56.9
70.6
77.9
25.0
class-agnostic open-world 3D object detection method
OP3Det (ours)
78,8
89.7
93.1
65.4
79.9
83.2
87.3
28.6
Implementation Details. We implement with mmdetection3D [62], and train with the AdamW [63] optimizer. We use ResNet50 [64] and FPN [65] for the image feature extractor, and sparse 3D ResNet for the voxel feature extractor. We use the multi-scale of ó = (0.2, 0.5, 1. 2). .Npoine is set to the half number of the total points. We utilize the 0.6 threshold to filter low-quality discovered 3D objects.
Baselines. Since class-agnostic 3D detection in the open world has not yet been explored, we compare OP3Det with methods from the related fields of closed-world 3D detection (i.e. traditional fully- supervised 3D detection) [66, 60, 17, 15, 56] and open-vocabulary 3D detection [7, 9], and adapt baselines accordingly. For closed-world 3D detection methods, we convert all seen categories into single class label during training so that these SOTA supervised methods only classify the bounding boxes are objects not. For open-vocabulary methods, we similarly construct class-agnostic training setting by replacing all class-specific text prompts with "object". This ensures the model learns to detect general objects without relying on specific category semantics.
4.1 Cross-Category Generalization
As is shown in Tab. 1, OP3Det demonstrates significant improvements over existing methods. For novel class discovery, AR increases by 13.5% compared to the state-of-the-ar cloled-world 3D detector FCAF3D, and by 16% compared to the open-vocabulary 3D detector OV-Uni3DET Furthermore, our model also shows improvements on base classes, without any decline. This thus contributes to an overall increase in AR across all classes. It is worth mentioning that since novel class objects only represent a small proportion of the scenes, their impact on the overall AR and AP is relatively limited. Nevertheless, even under these conditions, our method still achieves an average increase of over 3% in AR and APall, strongly demonstrating the 3D object discovery capability of our method. Such substantial improvement highlights that 3D objectness learning addresses challenging issue about the ambiguous nature of class-agnostic 3D object detection tasks that remains unsolved by existing 3D models, underlining the importance and necessity of our new intuition.
On the larger-scale ScanNet dataset, where the number of categories is higher, our OP3Det continues to demonstrate strong performance, with a clear advantage over existing methods, achieving a 12.3% improvement in ARnovel. This further validates the capability of our model in the large-vocabulary setting. Notably, under these conditions, the performance gap between base and novel classes is even smaller, highlighting the strong cross-category generalization ability of Oet Compared to the traditional closed- world 3D detectors, our method benefits from leveraging 2D semantic knowledge for 3D object discovery, effectively mitigating the limitations of category information in 3D point clouds. In contrast to open-vocabulary 3D detectors, our use of -lass-agnostic classification aligns more closely with the objectives of 3D objectness learning. Additionally, the cross-modal MoE effectively integrates multi-modal information, allowing the most relevant features to be applied for class-agnostic detection. The significance of the class-agnostic open-won world 3D detection problem as valuable new direction can also be validated.
7

---

Table 2: The cross-dataset performance of OP3Det on the SUN RGB-D and ScanNet dataset for class-agnostic open-world 3D object defection. We directly test the trained model on another dataset to obtain the below cross-dataset results.
ScanNet + SUN RGB-D
SUN RGB-D - ScanNet
Method
AR25
AR50
AP2s
APso
AR25
AR50
AP25
AP50
closed world 3D object detection methods
VoteNet [16]
34.8
2.0
10.8
0.1
30.4
6.3
9.6
1.2
GreupFree3D [60]
41.4
0.4
1,9
0.1
39.4
5.2
8.7
0.1
FCAF3D [17]
59.3
8.1
17.9
0.6
47.7
14.6
12.9
1.9
Uni3DETR [15]
51.3
6.4
11.9
0.2
45.7
10.9
11.3
1.3
Tr3D [56]
54.6
4.5
11.4
0.2
45.2
10.7
9.4
1.6
ooca-voca 3D object detection methods
CoDA [7]
21.4
2.8
6.2
0.1
32.7
5.2
8.9
0.4
OV-Uni3DETR [9]
49.5
3.2
8.1
0.3
52.0
15.4
9.5
0.8
class-agnostic open-world 3D object detection method
OP3Det (ours)
73.1
10.7
22.3
1.1
77.9
37.3
21.2
5.1
Table 3: The performance of OP3Det KITTI dataset for class-agnostic open-world 3D object detection. The models are trained only on car and are evaluated on car, pedestrian and cyclist. We report 7o with 40 recall positions. *; AD3D the moderate difficulty is the most important metric.
AP3D
APBEV medium
Method
easy
medium*
easy 3D object detection methods
hard
hard
clold-worrd
SECOND [69]
61.05
62.36
61.36
63.15
69.00
68.46
PointPillar [70]
59.54
62.13
60.04
63.04
68.87
66.75
Part-A² [71]
61.28
63.43
63.57
62.93
69.04
69.88
3DSDD [72]
61.42
62.34
62.06
62.93
68.50
68.29
PV-RCNN [73]
59.88
65.18
65.67
63.01
69.36
70.42
Uni3DETR [15]
63.54
65.74
65.43
62.74
69.01
69.87
open-vocabudary 3D object detection method
OV-UniDDETR 19]
62.66
63.20
62.82
64.33
69.15
68.98
class-agnoslic open- open-world 3D object detection method
OP3Det (ours)
63.56
66.75
66.42
65.13
71.37
70.34
4.2 Cross-Dataset Generalization
Moreover, since class-agnostic open-world 3D detection aims for robust performance in unseen or unknown domains, we further validate the cross-domain detection capability through cross-dataset experiments. In this case, point clouds in different datasets are generally collected through varying methods or sensors. Specifically, SUN RGB-D provides point clouds directly captured by single- view RGB-D cameras, whereas ScanNet reconstructs point clouds from multi-view RGB-D image sequences. As a result, the two datasets exhibit substantial differences in the structure and content of their point clouds, making cross dataset evaluation in the 3D domain considerably more challenging.
To this end, we conduct cross cross-dataset experiments for both the SUN RGB-D + ScanNet and ScanNet SUN RGB-D settings. The results are presented in Tab. 2. Due to differences in category definitions among datasets [67, 68], we only measure AR and AP across all objects. As can be seen, for many existing 3D detectors, significant performance drop appears in cross-dataset scenarios. For instance, CoDA performance deteriorates noticeably. This is largely due to the reliance of some 3D detectors on boint-based backbones for feature extraction, making them highly dependent on dataset-specific geometric information, which limits their effectiveness in cross-dataset generalization. Closed-world 3D detectors suffer from limited supervision due restricted annotations, while open-vocabulary 3D detectors with class cass-spec classification are vulnerable to category definition conflicts. In contrast, our method demonstrates substantial performance gains, achieving 30% AR25 improvement in the SUN RGB-D + ScanNet setting. The AP25 improvement is also almost 10%. Besides, the cross-dataset performance also closely approaches in-dataset results, with only 2% lower AR25. This further confirms the strong cross-dataset generalization ability of our method and its effectiveness in learning open-world 3D objectness. Through both cross-category and cross-datase evaluation, the strong 3D object detection capability of OP3Det in indoor scenes can be demonstrated.
8

---

Table 4: Comparison with 3D open-vocabulary methods on the SUN RGB-D and ScanNet dataset for open-vocabulary 3D object detection (class-specific). The experimental setting
totally the same as CoDA, and the utilized data are downloaded from CoDA officially released code.
SUN RGB-D
ScanNet
Method
AP novel
APbuse
APast
AP novel
APbase
AP ail
CoDA [7]
6.71
38.72
13.66
6.54
21.57
9.04
INHA [74]
8.91
42.17
16.18
7.79
25.1
10.68
CoDAv2 [75]
9.17
42.04
16.31
9.12
23.35
11.49
OV-UmiDEETR [9]
12.96
49.25
20.85
15.21
31.86
17.99
GLRD [76]
12.96
49.40
20.88
17.29
26.78
18.87
OP3Det (ours)
14.31
49.63
21.99
17.77
32.12
20.16
4.3 Outdoor 3D Detection Generalization
We then evaluate OP3Det on the outdoor KITTI dataset, and list the comparative results in Tab. 3. Unlike indoor point clouds, outdoor point clouds are usually collected by the LiDAR sensor. In outdoor 3D scenes, foreground objects are usually small and sparse, with significantly fewer points. Background points dominate the scene thus disturbing the detection process significantly. The gap between outdoor LiDAR points and 2D images is thus larger than indoor ones, making leveraging 2D semantic knowledge more challenging in outdoor scenes. Additionally, since the counts of pedestrian and cyclist classes are considerably lower than that of cars, the detection AP for novel classes has only a limited effect on the overall AP. Despite this, OP3Det still achieves the best performance. Notably, on the medium difficulty level, the most important metrie, AD3D outperforms existing methods by more than I %. with consistent improvements also observed in AP BEV. This largely underscores the generalization of approach across diverse point cloud scenes and highlights the adaptability of our method. The universality of OP3Det for exploring 2D semantic knowledge is thus further validated.
4.4 Class-Specific 3D Detection Generalization
The ability of class- object detection to learn open-world objectness and thereby locate all objects in 3D scenes makes it highly valuable for a wide range of downstream tasks. To further demonstrate the strength of this capability, we extend OP3Det class-specific 3D detection in this section. Specifically, we replace the 2D class-agnostic model with a class-specific detector [30], enabling the assignment of category labels during the 3D object discovery process. We follow the experimental setup of CoDA [7] and compare our approach with open-vocabulary methods, presenting the results in Tab. As shown, under the class-specific setting, our method outperforms OV-UniDDETR by more than 2% in AP and also surpasses the current ffate-of-theas method, GLRD. This further validates the strong capability and practical value of our approach.
Although OP3Det designed in a class-agnostic setting, it still performs well on class-specific detection tasks. This can be attributed to its strong 3D objectness learning, which provides high object recall and precise localization. When coupled with a class-specific head, the model readily adapts to semantic recognition, demonstrating that robust objectness understanding serves transferable foundation for both class-agnostic and class-specific 3D detection.
4.5 Ablation Study
We conduct an ablation study to evaluate the effectiveness of the SAM, multi-scale point sampling (PS), and cross-modal MoE (CM-MoE), Such a study listed in Tab. and Tab. 6.
Table Ablation Study on the SUN RGB-D dataset. SAM: utilizing SAM for object discovery. PS: multi-scale point sampling in 3D novel object discovery. CM-MoE: cross-modal MoE.
SAM
PS
CM-MoE
ARnovel
AR
AR
54.2
84.0
92.3
50.0
74.1
81.6
69.2
87.9
92.5
V
78.8
89.7
93.1

---

Table 6: Ablation Study on the SUN RGB-D dataset about the cross-modal MoE. PC and Img indicate whether the point cloud or image modalities are used during training, and method denotes the multi-modal fusion approach. Addition and Concat represent feature summation and concatenation, respectively, while CM-MoE refers to the proposed cross-modee Mixture of Experts.
PC
Img
method
ARgove
ARail
ARiase
69.2
87.9
92.3
38.4
64.4
72.5
addition
65.4
85.6
91.4
concatenation
66.0
85.8
92.1
CM-MoE
78.8
89.7
93.1
Figure 4: The visualized results of OP3Det on the SUN RGB-D (the first row) and ScanNet (the second row) dataset. The red boxes are base classes and blue boxes are novel classes.
The core design of our method consists of two main components: 3D novel object discovery and cross-modal Mixture of Experts (MoE). For 3D object discovery, we first employ the robust SAM model to identify potential objects. However, using SAM alone leads to suboptimal performance on both novel and base categories, resulting in a 4.2% decrease in ARnover and a 10.7% decrease in ARtuse. This is primarily because SAM is not inherently object-centric and tends to produce numerous fragmented or partial masks, introducing substantial noise that degrades overall detection performance. After incorporating our multi-scale point sampling strategy and a class-agnostic 2D detector for post-procs,,ing, the 3D objectness learning is significantly enhanced - ARnovel improves by 19.2% and ARbase by 10.9%. These results demonstrate that our multi-scale point sampling effectively suppresses noisy masks and leverages 2D semantic cues to accurately discover 3D objects. Additional results and analyses are provided in the Appendix D.
As can be seen in Tab. 6, both single-mode modal point cloud and RGB image modalities achieve commend- able 3D objectness learning, validating the effectiveness of our strategy of using 3D spatial proximity for 3D object discovery. However, naive fusion approaches such as summation or concatenation-40 commonly used in prior works-lead to degraded performance compared to the point-cloud-only model. This stems from the open-world class-agnostic setting, where binary foreground -oruuund-hhkgouunnd prediction can cause RGB features to interfere with critical 3D geometric cues if fusion is not prop- erly guided. In contrast, our cross-modal MoE dynamically balances uni-modal and multi-modal representations, allowing each modality contribute adaptively. As result, OP3Det improves AR.ll by 1.8% and ARnovet by 9.6%, effectively leveraging complementary 2D semantic and 3D geometric information while preserving modality knowledge for robust 3D objectness learning.
Visualization. We provide visualization in Fig. to further validate the effectiveness of our OP3Det.
5 Conclusion
We introduce OP3Det, the first work to learn 3D objectness in a class-agnostes manner an open- world setting. We leverage multi-modal learning to bridge the gap between limited 3D annotations and extensive 2D semantic knowledge. First, we utilize 2D and 3D object priors for 3D object discovery. By integrating semantic knowledge and a cross cross-modal mixture of experts. OP3Det captures intra- and inter-modal dependencies cohesively and demonstrates impressive generalization capability across a diverse range of categories and scenes, especially unseen classes. Extensive experiments demonstrate its discover-all ability. We believe OP3Det represents a significant step forward in enabling scalable, real-world applications of 3D object detection in complex, open- settings.
10

---

6 Acknowledge
The authors would like to thank anonymous reviewers for their insightful comments and valuable suggestions. This work is partially supported by the National Science Foundation under Grant No. 2047822, 1952096, and 2411151.
References
[1] C. Huang, V. Abdelzad, C. G. Mannes, L. Rowe, B. Therien, R. Salay, K. Czarnecki, et al., "Out-of- distribution detection for lidar-based 3d object detection," in ITSC, 2022.
[2] M. Kösel, M. Schreiber, M. Ulrich, C. Gläser, and K. Dietmayer, "Revisiting out-of distribbution detection in lidar-based 3d object detection," in 2024 IEEE Intelligent Vehicles Symposium (IV), 2024.
[3] R. Ding, J. Yang, C. Xue, W. Zhang. S. Bai, and X. Qi, (Lowis3d:: Language-driven open-works instance- level 3d scene understanding." TPAMI, 2024.
W. Hu, W. Lin, H. Fang. Y. Wang, and D. Luo, "Ow3det: Toward open-world 3d object detection for autonomous driving." " IROS, 2024.
[S] R. Zhang, Guo, Zhang, K. Li, Miao, B. Cui, Y. Qiao, P. Gao, and Li, "Pointelip: Point cloud understanding by clip." " CVPR, 2022.
[6] Z. Huang. Wu, X. Chen, H. Zhao, L. Zhu, and Lasenby, "Openins3d: Snap and lookup for 3d open-vocabulary instance segmentation," in ECCV, 2023.
[7] Y. Cao, Y. Zeng. H. Xu, and D. Xu, "Coda: Collaborative novel box discovery and cross-modal alignment for open-vocabulary 3d object detection," in NewrIPS, 2023.
[8] X. Zhu, R. Zhang, B. He. Z. Guo, Z. Zeng. Z. Qin, S. Zhang, and P. Gao, "Pointclip v2: Prompting clip and gpt for powerful 3d open- learning." in CVPR, 2023.
[9] Z. Wang, Y. Li, T. Liu, H. Zhao, and S. Wang, "Ov-uni3detr: Towards unified open-vocabulary 3d object detection via eycle-modality propagation," in ECCV, 2024.
[10] P. Nguyen, T. Ngo, E. Kalogerakis, C. Gan, Tran, C. Pham, and Nguyen, "Open3dis: Open- vocabulary 3d instance segmentation with mask guidance," CVPR, 2024.
[11] Oquab. T. Darcet, Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, Fernandez, D. Haziza, F. Massa. A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P.-Y. Huang, S.-W. Li, Misra, M. Rabbat, V. Sharma, G. Synnacve, H. Xu, H. Jegou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski, "Dinov2: Learning robust visual features without supervision," 2024.
[12] R. Fang, G. Pang. W. Miao, X. Bai, J. Zheng, and X. Ning, "Unsupervised recognition of unknown objects for open-world object detection," IEEE Transactions on Neural Networks and Learning Systems, 2025.
[13] C. L. Zitmick and P. Dollár, "Edge boxes: Locating object proposals from edges," in ECCV, 2014.
[14] M. Maaz, H. Rasheed, S. Khan, F.S. Khan, R. M. Anwer, and M.-H. Yang. "Class- object detection with multi-modal transformer," in ECCV, 2022.
[15] Z. Wang, Y.-.. Li, X. Chen, H. Zhao, and S. Wang, "Uni3detr: Unified 3d detection transformer." in NeurIPS, 2023.
[16] C. Qi. Litany, He, and L. Guibas, "Deep hough voting for object detection in point clouds." in ICCV, 2019.
[17] D. Rukhovich, A. Vorontsova, and A. Konushin, "Feaf3d: fully convolutional anchor-free 3d object detection," in ECCV, 2022.
[18] T. Yin, X. Zhou, and P. Krahenbuhl, "Center-hased 3d object detection and tracking," in CVPR, 2021.
[19] G. Zhang, J. Chen, G. Gao, J. Li, S. Liu, and X. Hu, "Safdnet: A simple and effective network for fully sparse 3d object detection," in CVPR, 2024.
[20] G. Zhang. L. Fan, C. He, Z. Lei, Z.X ZHANG, and L. Zhang, "Voxel mamba: Group-free state space models for point cloud based 3d object detection," NewrIPS, 2024.
11

---

[21] S. Song. P. Lichtenberg, and J. Xiao, "Sun rgb-d: A rgb-d scene understanding benchmark suite," in CVPR, 2015.
[22] A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner, "Scannet: Richly- annottadd 3d reconstructions of indoor scenes," in CVPR, 2017.
[23] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, "Vision meets robotics: The kitti dataset," IJRR, 2013.
[24] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan. G. Baldan, and
O. Beijhom, "nuscencs: A multimodal datasct for autonomous driving," in CVPR, 2020.
[25] J. S. Lim, Z. Chen, M. Baktassmottlggh, Z. Chen, X. Yu, Z. Huang, and Y. Luo, "Dipex: Dispersing prompt expansion for class-agnostic object detection," arXiv preprint arXiv:2406. 149224444.4444,,, 2024.
[26] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., "Learning transferable visual models from natural language supervision." in ICML, 2021.
[27] A. Kirillov, Mintun, N. Ravi, H. Mao, C. Rolland, L., Gustafson, Xiao, S. Whitehead, A, C. Berg, W.-Y. Lo, et al., "Segment anything," in ICCV, 2023.
[28] X. Zhou, R. Girdhar, A. Joulin, P. Krähenbühl, and I. Misra, "Detecting twenty-thousand classes using image-level supervision," in ECCV, 2022.
[29] Z. Wang. Li, X. Chen, S.-N. Lim, A. Torralba, H. Zhao, and S. Wang. "Detecting everything in the open world: Towards universal object detection," in CVPR, 2023.
[30] S. Liu, Z. Zeng, Ren, F. Li, H. Zhang, J. Yang. C. Li, J. Yang. H. Su, Zhu, et al., "Grounding dino: Marrying dino with grounded pre-training for open-set object detection," in ECCV, 2024.
[31] T.-Y. Lin, M. Maire, S. Belongie, Hays, Perona, D. Ramanan, Dollár, and C. L. Zitnick, "Microsoft coco: Common objects in context," in ECCV, 2014.
[32] A. Kuznetsova, H. Rom, N. Alldrin, J. Uijlings, I. Krasin, J. Pont-Tuset, Kamali, Popov, M. Malloci,
A. Kolesnikov, et al., "The open images dataset v4," ICV,, 2020.
[33] S. Shao, 7. Li, T. Zhang, C. Peng, G. Yu, X. Zhang, J. Li, and Sun, "Objects365: A large-scale, high-quality dataset for object detection," " in ICCV, 2019.
[34] H. Chen, P. Wei, G. Guo, and S. Gao, "Sam-cod: Sam-guided unified framework for weakly-supervised camouflaged object detection," " ECCV, 2025.
[35] Z. Wei, P. Chen, X. Yu, G. Li. J. Jino, and Z. Han, "Semantic-aware sam for print-prompte instance segmentation." in CVPR, 2024.
[36] S. Vora, A. H. Lang, B. Helou, and O. Beijbom, "Pointpainting: Sequential fusion for 3d object detection." in CVPR, 2020.
[37] C. Wang. C. Ma, M. Zhu, and X. Yang, "Pointaugmenting: Cross-modal augmentation for 3d object detection," in CVPR, 2021.
[38] 7. Liu, H. Tang, A. Amini, X. Yang, H. Mao, D. I.. Rus, and S. Han, "Bevfusion: Multi-task multi-sensor fusion with unified hird's s-eye view representation," in ICRA, 2023.
[39] T. Liang, H. Xie, K. Yu, Z. Xia, Z. Lin, Y. Wang. T. Tang, B. Wang, and Z. Tang, "Bevfusion: A simple and robust lidar-camera fusion framework," NeurlPS, 2022.
[40] Y. Li, Y. Chen, X. Qi, Z. Li, J. Sun, and J. Jia, "Unifying voxel-based representation with transformer for 3d object detection," " in NeurlIPS, 2022.
[41] Y. Xie, C. Xu, M.-J. Rakotosaona, P. Rim, F. Tombari, K. Keutzer, M. Tomizuka, and W. Zhan, "Sparsefu- sion: Fusing multi-modal sparse representations for multi-sensor 3d object detection." " in ICCV, 2023.
[42] Q Cai, Y. Pan, T. Yao, C.-W. Ngo, and T. Mei, "Objectfusion: Multi-modal 3d object detection with object-centric fusion," " in ICCV, 2023.
[43] X. Bai, Z. Hu, X. Zhu, Q. Huang, Y. Chen, H. Fu, and C.-L. Tai, "Transfusion: Robust lidar-camera fusion for 3d object detection with transformers," in CVPR, 2022.
[44] J. Yin, J. Shen, R. Chen, W. Li, R. Yang, P. Frossard, and W. Wang, "Is-fusion: Instance-scent collaborative fusion for multimodal 3d object detection." in CVPR, 2024.
12

---

[45] J. Cen, P. Yun, S. Zhang, J. Cai, D. Luan, M. Tang, M. Liu, and M. Yu Wang, "Open-world semantic segmentation for lidar point clouds," in ECCV, 2022.
[46] Y. Yang. X. Wu. T. He, H. Zhao, and X. Liu, "Sam3d: Segment anything in 3d scenes," arXiv preprint wxXiv:2306.03900 2023.
[47] J. Li and Q. Dong, "Open-set semantic segmentation for point clouds via adversarial prototype framework," in CVPR, 2023.
[48] E. A. Boudjoghra, Al Khatib, J. Lahoud, H. Cholakkal, R. Anwer, S. H. Khan, and Shahbaz Khan, "3d indoor instance segmentation in an open-world," NewrIPS, 2023.
[49] Xu, S. Yang, X. Li, Tang, Y. Hao, L. Hu, and Chen, "A probability framework for open world 3d point cloud semantic segmentation." in CVPR, 2024.
[50] H. Zhang, Xu, T. Tang, H. Sun, X. Yu, Z. Huang, and K. Yu, "Opensight: A simple open- upen-vocabulary framework for lidar-hased object detection," in ECCV, 2024.
[51] R. Huang, S. Peng, A. Takmaz, F. Tombari, M. Pollefeys, S. Song, G. Huang, and F, Engelmann, "Seg- ment3d: Learning fine-grate fine-grained class-agnostic 3d segmentation without manual labels." in ECCV, 2024.
[52] H. Kweon, J. Kim, and K.-J. Yoon, "Weakly supervised point cloud semantic segmentation via artificial oracle," in CVPR, 2024.
[53] H. Guo, H. Zhu, S. Peng. Y. Wang, Y. Shen, R. Hu, and X. Zhou, "'aam-guided graph cut for 3d instance segmentation," in ECCV, 2024.
[54] A. Takmaz, E. Fedele, R. Sumner, M. Pollefeys, F. Tombari, and E. Engelmann, "Openmask3d: Open- vocabulary 3d instance segmentation," NeurlPS, 2023.
[55] K. Saito, P. Hu, T. Darrell, and K. Saenko, "Learning detect every thing in open world," in ECCV, 2022.
[56] D. Rukhovich, A. Vorontsova, and A. Konushin, "Tr3d: Towards real-time indoor 3d object detection," in ICIP, 2023.
[57] A. Vaswani, "Attention all you need," in NewrlPS, 2017.
[58] N. Carion, Massa, G. Synnacve, N. Usunier, A. Kirillov, and S. Zagoruyko, "End-to-end object detection with transformers," in ECCV, 2020.
[59] D. Rozenberszki, Litany, and Dai, "Language-groundee indoor semantic segmentation the wild," in ECCV, 2022.
[60] Z. Liu, Zhang, Cao, H. Hu, and X. Tong, "Group-free 3d object detection via transformers," in ICCV, 2021.
[61] T. Yang, Y. Ju, and I. Yi, "Imov3d: Learning open-vocabulary point clouds 3d object detection from only 2d images," in NewrlPS, 2024.
[62] M. Contributors, "Mmddtection3d: Openmmlab next-generation platform for general 3d object detection,"
2020.
[63] T. Loshchilov and F. Hutter, "Decoupled weight decay regularization," ICLR, 2019.
[64] K. He. X. Zhang, S. Ren, and J. Sun. "Deep residual learning for image recognition," in CVPR, 2016.
[65] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, "Feature pyramid networks for object detection," in CVPR, 2017.
[66] C. R. Qi, X. Chen, O. Litany, and L. Guibas, "Imvotenet: Boosting 3d object detection in point clouds with image votes," in CVPR, 2020.
[67] X. Zhou, V. Koltun, and P. Krähenbühl, "Simple multi-dataset detection," in CVPR, 2022.
[68] X. Zhao, S. Schulter, G. Sharma, Y.HH. Tsai, M. Chandraker, and Y. Wu, "Object detection with a unified label space from multiple datasets," in ECCV, 2020.
[69] Y. Yan, Y. Mao, and B. Li. "Second: Sparsely embedded convolutional detection," Sensors, 2018
13

---

[70] A. H. Lang, S. Vora, H. Caesar. L. Zhou, J. Yang, and O. Beijbom, "Poinipillars: Fast encoders for object detection from point clouds," " in CVPR, 2019.
[71] S. Shi, Z. Wang, J. Shi, X. Wang, and H. Li, "From points parts: 3d object detection from point cloud with part-aware and part-aggregation network," TPAMI, 2020.
[72] Z. Yang, Y. Sun, S. Liu, and J. Jia, "3dssd: Point-based 3d single stage object detector," in CVPR, 2020.
[73] S. Shi, C. Guo, I.. Jinng, Z. Wang, J. Shi, X. Wang, and H. Li, "Pv-renn: Point-voxel feature set abstraction for 3d object detection," in CVPR, 2020.
[74] P. Jiao, N. Zhao, J. Chen, and Y.GG. Jiang, "Unlocking textual and visual wisdom: Open-vocabulary 3d
object detection enhanced by comprehensive guidance from text and image." in ECCV, 2024.
[75] Y. Cao, Y. Zeng, H. Xu, and D. Xu, "Collaborative novel object discovery and box- box-guided cross-modal alignment for spen-vocabulary 3d object detection," arXiv preprint arX:6:2406. 0000000030, 2024.
[76] X. Peng, S. Liu, C. Gao, Y. Bai, B. Mu, X. Wang, and H. Xia, "Glrd: Global-local collahorative reason and debate with psl for 3d open-vocabulary detection," arXiv preprint arXiv:2503.2000 20682, 2025.
[77] H. Zhang, F. Li, S. Liu, L. Zhang, H. Su, J. Zhu, L. M. Ni, and H.-Y. Shum, "Dino: Detr with improved denoising anchor boxes for end-to-end object detection," in ICLR, 2023.
[78] X. Gu, T-Y. Lin, W. Kuo, and Y. Cui, "Open-vocabulary object detection via vision and language knowledge distillation," in ICLR, 2022.
[79] Y. Zang, W. Li, K. Zhou, C. Huang, and C. C. Loy, "Open-vocabllay detr with conditional matching," in ECCV, 2022.
[80] K. Joseph, S. Khan, F. S. Khan, and V. N. Balasubramanian, "Towards open world object detection," in CVPR, 2021.
[81] C. Xu, B. Wu, J. Hou, S. Tsai, R. Li, J. Wang. W. Zhan, Z. He, P. Vajda, K. Keutzer, et al. "Nerf-det: Learning geometry-awe aware volumetric representation for multi view 3d object detection," in ICCV, 2023.
[82] Z. Li, W. Wang, H. Li, E. Xie, C. Sima, T. Lu, Y. Qiao, and J. Dai, "Bevformer: Learning bird's e-ye---we representation from multi-camers images via spatiotemporal transformers," in ECCV, 2022.
[83] C. Ge. J. Chen, E. Xie, Z. Wang, L. Hong, H. Lu, Z. Li. and P. Luo, "Metabev: Solving sensor failures for 3d detection and map segmentation." in ICCV, 2023.
[84] D. Rukhovich, A. Vorontsova, and A. Konushin, "Imvoxelnet: Image to voxels projection for monocular and multi-view general-purpose 3d object detection," in WACV, 2022.
[85] G. Brazil, A. Kumar, J. Straub, N. Ravi, J. Johnson, and G. Gkioxari, "Omni3d: A large benchmark and model for 3d object detection in the wild," in CVPR, 2023.
[86] Y. Lu, C. Xu, X. Wei. X. Xie, M. Tomizuka, K. Keutzer, and S. Zhang, "Open-vocabulary point-cloud object detection without 3d annotation," in CVPR, 2023.
[87] K. He, G. Gkioxari, P. Dollár, and R. Girshick, "Mask r-can," in ICCV, 2017.
[88] D. Kim, T.-Y. Lin, A. Angelova, I. S. Kweon, and W. Kuo, "Learning open-world object proposals without learning to classify." RAL, 2022.
[89] W. Wang, M. Feiszli, H. Wang, J. Malik, and D. Tran, "Open-world instance segmentation: Exploiting pseudo ground truth from leamedd pairwise affinity," in CVPR, 2022.
[90] J. Wu, Y. Jiang, B. Yan, H. Lu, Z. Yuan, and P. Luo, "Exploring transformers for open-world instance segmentation." in ICCV, 2023.
[91] T. Kalluri, W. Wang, H. Wang, M. Chandraker, I. Torresani, and D. Tran, "Open-world instance segmenta- tion: Top-down learning with bottom-up supervision." in CVPR, 2024.
[92] C. Wilms, T. Rolff, M. Hillemann, R. Johanson, and S. Frintrop, "Sos: Segment object system for open-world instance segmentation with object priors," in ECCV, 2024.
[93] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, "The pascal visual object classes (voc) challenge," IJCV, 2010.
14

---

Appendix
A Overview of 3D Open-Word World Learning Research
Table 7: The overview of existing detectors on their capability, For category. we discuss whether the detector recognize novel classes during inference ("closed" v.s. "open"), and "open (no cfs)"
denotes whether the category confusion problem exists in the large-vocabulary scene.
Task
Scene
Modality (3D)
Caregory
Method
Venue
2D
3D
indoor outdoor
PC
Ing
closed
open
open (no cfs)
DETR [58]
ECCV'20 20
V
K
DINO [77]
ICLR'23
/
x
X
K
VILD [78]
ICLR'22
/
X
X
OV-DETR [79]
ECCV
/
X
Detic [28]
ECCV*22
/
ORE [80]
CVPR'21
LDET [SS]
ECCV'22
/
UniDetector [29]
CVPR'23
V
VoteNet [16]
19
X
X
FCAF3D [17]
ECCV"22
X
X
NeRF-Det [81]
ICCV'23
x
PointPillars [70]
CVPR 19
X
CenterPoint [18]
CVPR'21
x
x
/
x
BEVFormer [82]
BCCV'22
x
VotoNNtt [66]
CVPR'20
X
X
TR3D [56]
ICIP'23
x
MetaBEV [83]
ICCV'23
X
Im VoxelNNe [84]
WACV'22
X
7
Cude RCNN [85]
CVPR'23
x
/
x
7
X
Uni3DETR [15]
NeurIPS'23
x
X
5
X
OV-3DET [86]
CVPR'23
X
x
CoDA [7]
NeurlPS 23
x
<
x
x
OV-Uni3DEIR [9]
BCCV'24
* OP3Der (ours)
/
We list the overview of existing object detectors about their capability in Tab. In the 2D detection area, significant ad advancements have been achieved across various approaches, including traditional closed-world detectors, open-vocabulary detectors constrained by category confusion issues, and open-world detectors utilizing class-agnostic classification. However, in the 3D domain, progress remains substantially lagging behind the rapid developments observed in 2D detection. The majority of 3D detectors are designed to operate in either indoor or outdoor point clouds, lacking the ability to generalize across different environments. In terms of modality, most 3D detectors are limited to utilizing only one type of data, either point clouds or RGB images, and are constrained to the closed-world world setting. While recent advancements have introduced open-vocabulary 3D detection methods, class-agnostic open-world 3D detectors overcoming the category confusion problem have still yet to emerge.
In comparison, our OP3Det, as the first class-agnostic open-worid 3D detector, can not only recognize both base and novel classes during inference but also effectively mitigate the issue of category confusion. Furthermore, it leverages data from multiple modalities for multimodal training and is capable of functioning seamlessly both indoor and outdoor scenes. Additionally, our method can be easily extended to 2D detection tasks, demonstrating its versatility and robustness. Therefore, it greatly advances existing research towards the goal of universal 3D object detection and we believe OP3Det can become significant step towards the future of 3D foundation models.
B More Method Details
We summarize our method Algorithm and 2. Specifically, OP3Det utilizes both point clouds and RGB images for multi-modal training to detect in the 3D open world. To recognize novel classes the open world and achieve the cross-category ability, the core idea of our method is leverage abundant 2D semantic knowledge to enhance 3D open-world detection. Specifically, we utilize 3D spatial proximities to refine the uniformly distributed point prompts provided to SAM. Instead of uniformly sampling 64x64 grid, we assign point-wiss object prior to each point by combining its
15

---

Algorithm 3D object discovery algorithm
Input:
1. (Xp, Xj): point clouds and corresponding 3D detection images, with camera parameters K, R
2. 3D annotations {(c, ] = {0, } due to the class-agnostic setting.
2. The pre-trained foundation model SAM SSAM. pre-trained 2D class-agnostic detector PCA
3. Point number threshold point and multi-scale threshold {ō,, i 2, .m. No}
3D Object Discovery:
1. Multi-scale point sampling:
for L N2 do
Initialize selected set Sş +- 0, extract object prior map Oprior
while [ < poinn do
Select point 8* with the highest value in O prioe, add s* to S₁ for all p3 E Xp Se do
Compute 3D distance d(ss, .Pi)
d(ss Pi) < òş then
Set object prior Opiior at corresponding 2D pixel to end if end for end while end for The ultimate selected set S = NMS(U St) 2. Apply $SAM and &CA for selected points, then utilizing K, Rg to project into the 3D space: {bb.} (KR,) &ca (CA(4sAM(M))
Algorithm 2 Cross-Modal MoE algorithm
Cross-Modal MoE Training:
1. Obtain point cloud features Fp from Xp, image features the voxel space Fj from X,.
2. Obtain multi-modal features FM [Fr, F/]-
3. Utilize self-attention module on the features:
Fp = SelfAttn(F'p). and F7. FM is obtained in the same way.
4. Utiiize the multi-modal router R to obtain the routing probability: (pp, PP..P) R(FM).
5. Apply the cross-modal MoE for multi-modal fusion: F
Pi (F.).
(E(P.L.)
6. Utilize F for 3D bounding box prediction, supervised with {bb,} + {bb,} for the model training.
IoU scores with the maximum attention value across self-attention heads from the self-supervised model (DINO). These object prior points then progressively refined using seurse-to-fine multi- scale sampling strategy, which allows for increasingly precise localization across spatial resolutions. Specifically, we project all 3D points onto the 2D image plane and establish mapping between 2D pixels and 3D points by associating each pixel with its nearest 3D point. In each iteration, the point with the highest object prior value is selected the source point. For each selected source point, we compute its 3D distance to all other points. Points within a predefined 3D distance threshold are suppressed by setting their 2D object prior values zero. We then select the next source point with the highest remaining object prior to the score and repeat this process until N points are selected. The 2D bounding boxes generated from the refined points are then used in 2D class-agnostic models for 3D object discovery.
Ultimately, during training, the cross modal MoE is utilized for multi modal fusion and the discovered 3D objects serve as supervision.
C More Experimental Results
We further conduct more experiments in this section to demonstrate the effectiveness our designs. We first list the AP metric of OP3Det on the SUN RGB-D dataset in Tab. 8, then conduct ablation study
16

---

Table 8: AP metric of the cross-category performance of OP3Det on the SUN RGB-D dataset.
SUN RGB-D
Method
AP novel
allli
Aosase
wloled-world 3D object detection methods
VoteNet [16]
2.0
55.1
66.3
GroupFree [60]
2.3
49.2
58.4
FCAF3D [17]
3.4
61.0
74.1
Uni3DETR [15]
2.3
61.3
74.4
Tr3D [56]
3.7
53.4
62.7
open-vocabulary 3D object detection methods
CoDA [7]
48.2
57.8
OV-Uni3DETR [9]
10.2
57.4
67.8
ImOV3D [61]
8.1
28.3
35.4
class-agnostic open-warld 3D object detection method
OP3Det (ours)
12.6
65.4
75.7
Table 9: Ablation Study the SUN RGB-D dataset about the multi-scale point sampling. We conduct 3D object discovery using the corresponding methods, and directly evaluate the AR and AP metrics of discovered 3D objects, without training the detector. PS is short for point sampling.
method
AR nRveve
ARau
ARßase
AP
SAM
64.0
55.4
52.8
6.8
SAM + PS (r=0.2)
47.5
43.1
41.7
5.9
SAM PS (7=2)
49.9
12.6
40.3
5.7
SAM multi-scale PS
61.9
54.2
51.9
7.6
SAM + multi-scale PS + LDET [55]
66.1
59.2
57.1
10.0
mainly on the multi-scale point sampling strategy during 3D object discovery and the cross-modal MoE.
AP metrie. In the original paper, we mainly report the AR metric of our OP3Det, consisting of AR ARnovel and ARbase- We only report AP in the original paper. The main reason is that aim to discover "all" 3D objects in the scene, while not all bounding boxes are necessarily annotated the ground truth of the test set. As result, objects that are not in the test set annotation but found by the model will also be counted as false positives (FP) and introduce errors in AP metrics. Besides, calculating AP novel and APhose is also not suitable for class-agnostic detection, because the definition of FP can be ambiguous. For example, when calculating AP noweet it is unclear whether the detected base objects should be FP. Therefore, the AR metric is more suitable for our setting.
Despite this, we can still ignore base or novel objects when calculating APnovel or APbece: to provide comprehensive comparison. We list the results in Tab. 8. As can be seen, OP3Det also obtains better performance, achieving the 12.6% AP novel and 65.4% AP. Compared with existing methods, we achieve 2.4% higher AP novel and 4.1% higher AP. This further demonstrates the effectiveness of our method.
Multi-scale point sampling. In our original paper, for 3D object discovery, we first extract class- agnostic masks using SAM. During this process, the multi-scale point sampling strategy is employed to alleviate fragmented masks object parts. Finally, LDET is applied for post-processing. We analyze the impact of these design choices sequentially, and evaluate the AR and AP metrics of discovered 3D objects, as shown in Tab. 9.
As observed, the direct results from SAM achieve relatively strong AR metrics for 3D discovered objects, with an ARnove of 64% and an AR,ll of 55.1%. This demonstrates that SAM effectively uncovers a broader range of objects. With these objects participating in the training, the diversity of training can be boosted greatly. This validates the effectiveness of our idea to introduce broader 2D semantic knowledge into the 3D domain. However, the presence of numerous fragmented masks introduces a significant amount of noisy masks, leading to very low AP of only 6.8%. This low precision indicates that directly using these objects during training would result poor model performance, can be seen in our original paper.
17

---

Table 10: Comparison with 2D open-world methods for the COCO (VOC) to COCO (non-VOC,
setting. Here, we compare with various 2D open-world instance segmentation methods and report metrics based on masks.
Method
AP
AR100
F1
Mask R-CNN [87]
1.0
8.2
1.8
SAM [27]
3.6
48.1
6.7
OLN [88]
4.2
28.4
7.3
LDET [55]
4.3
24.8
7.3
GGN [89]
4.9
28.3
8,4
SWORD [90]
4.8
30.2
8.3
UDOS [91]
2.9
34.3
5.3
SOS [92]
8.9
39.3
14.5
OP3Det (ours)
13.9
42.9
21.0
SAM3D
OP3Det
(Ours)
Figure 5: Qualitative comparison with SAM3D [46]. Benefiting from our contributions, our method OP3Det yields more accurate localization and precise discovery of novel objects.
Using point sampling effectively filters out many fragmented masks, reducing noise in object masks. However, this also inadvertently filters some useful objects. Consequently, regardless of the choice of both AR and AP metrics show decline, negatively impacting the performance. By leveraging multi-scale point sampling that combines the strengths of different values, becomes possible to balance noise reduction while retaining diverse objects. This results in an improved AP of 7.6%, demonstrating a notable enhancement in the quality discovered 3D objects. However, the AR metric still remains lower than when directly using SAM, indicating that some important masks are still being filtered out. By further incorporating LDET, the holistic object understanding ability the 2D detector can be utilized to better filter out noise within object masks. As result, both AR and AP metrics show significant improvement. This enhancement demonstrates that the quality of discovered 3D objects is notably elevated, enabling their effective use in subsequent training and ensuring the model's 's cross-category generalization ability.
Comparison with 2D methods. Additionally, since a part of our method is conducted in the 2D domain, we also compare it with 2D open-world detectors. Specifically, we first perform 2D object discovery and then train Mask R-CNN [87] on the COCO [31] dataset, for instance, segmentation. The 20 classes overlapping with VOC [93] are treated as seen classes, while the remaining 60 classes are treated as novel ones. We compare the predicted masks against existing methods with the results presented in Tab. For fair comparison, we do not utilize the text prompts here, as utilizing the class-specific 2D detector may result in the category leakage problem. As can be seen, OP3Det also surpasses existing methods by 5% in AP and 3.6% in AR. This demonstrates the effectiveness of our designed multi-scale point prompts in 2D object discovery.
18

---

Figure 6: More visualized results of OP3Det on the SUN RGB-D (the first three rows) and ScanNet (the last row) dataset. The red boxes are base classes and blue boxes are novel classes.
Comparison with SAM-related methods. Recent approaches such as SAM3D [46] leverage SAM- generated masks from RGB images and project them into 3D to obtain class- instance segmentation. To improve mask consistency, SAM3D adopts a bidirectional merging strategy across adjacent frames. However, it does not perform object detection. The instance segmentation task requires more fine-grained mask annotations for training; thus utilizing geometric information can be easier in such setting. In comparison, the class-agnostic object detection task has still not been explored yet. Meanwhile, existing SAM-related methods usually rely heavily on temporal fusion for filtering low-quality objects and achieving better localization quality, which cannot be applied in our object detection setting, where only one frame is available for a 3D scene. To ensure a fair comparison under favorable conditions for SAM3D, bounding boxes around the resulting point cloud masks. As shown in Figure 5, our method OP3Det outperforms SAM3D significantly in both the number and localization quality of novel objects. This demonstrates that OP3Det is not only more effective in discovering novel instances, but also robust under minimal inputs achieving superior results without requiring multi-frame fusion.
re-implement its pipeline and extract 3D
D More Visualized Results
We provide more visualized results on SUN RGB-D ScanNet datasets Fig. From these examples, it is evident that our model effectively detects abundant base class objects, such chairs and tables, while also accurately identifying numerous rare novel classes, such as various small objects on tables or beds. These visualizations further validate the effectiveness of our approach.
19

---

Figure 7: Failure Case. Despite accurately detecting most objects using both 2D semantic and 3D geometry knowledge, OP3Det fails non-rigid and low-contrast regions such as the white curtains.
Limitation. All methods have the potential for errors, and here we discuss the potential failure cases in our results. As observed in the visualization results, OP3Det successfully detects most objects, regardless of their size or whether their category can clearly identified. However, some objects remain undetected, particularly in complex scenes, such paper items cluttered desk stickers on a wall. This is partially because the training data and annotations do not cover a sufficiently diverse range of scenarios. Additionally, in highly complex environments, missed detections may occur when objects have rigidity and color similar to the background, making them difficult to distinguish.
Failure Case Analysis. As shown in the figure7, our model successfully detects most objects in this complex scene. The two closely black monitors on the desk are correctly localized, benefiting from the use of 3D geometric information. However, our method still fails to detect some objects such as white curtain and the white sofa. attribute these failures to lack distinctive geometric or color features in these objects. Since both curtains and sofas have relatively flat geometry and low texture contrast, especially under overexposed lighting conditions, becomes difficult for the model to distinguish them from the background surrounding surfaces. This suggests the need for better handling of low-texture, non-rigid and oolo-hhoogenneus regions in open-world 3D detection.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly describe the proposed class-agnostic open-world 3D object detection setting, the OP3Det model with multi-scale point sampling and cross-modal MoE, and these claims are supported by experiments demonstrating strong open-world world generalization.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The supplementary material acknowledges that although OP3Det performs well on both common and uncommon objects, it may still miss certain objects in complex scenes due to limited point cloud coverage and the reliance on accurate 2D-3D projection.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
21

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not present formal theoretical results. Instead, it focuses on the design and empirical evaluation of the proposed framework.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The paper includes all necessary details to reproduce the main experimental results, including dataset configurations, model architecture, training settings, and evaluation metrics. The code will be made publicly available after publication.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
22

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: While the code and data will be released after publication, the paper and supplementary material provide detailed descriptions of the datasets, model architecture, results.
training procedure, and evaluation protocols, which are sufficient to reproduce the main
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. wtt.cc/cpccclccciiie.eeo gwwc//gg/ge//eeeeeeeoeoo httpbi//gggiiiee/deeeuuuiiiiiiia //to///ooiissiiiiieeae
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies all necessary training and evaluation details, including dataset splits and manually defined typer-parameters, The selection of these parameters is documented in the main text and supplementary material to ensure clarity and reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Although error bars standard deviations are not explicitly reported, all results are obtained by averaging multiple runs to ensure stability. The consistent improvements
over baselines across benchmarks indicate the robustness of the findings.
23

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
We provide details on the GPU type (e.g., NVIDIA A100), batch size, and number of training epochs. We will include estimated training time and compute usage per experiment in the supplementary material to support reproducibility.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: //tter/is....eeeeee cttpps..cccccoooooocoe cctpubiic//tthhccstteeelleleell
Answer: [Yes]
Justification: The research complies fully with the NeurIPS Code of Ethics. It does in- volve human subjects, personal data, or sensitive content, and all experiments are conducted with academic integrity and transparency.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts

---

Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our work advances class-agnostic 3D object detection, which could positively impact robotics and autonomous systems by improving their ability to detect unseen objects.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The paper does not involve the release of models or datasets that pose a high risk of misuse. No scraped data, generative models, sensitive content are used or released.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
25

---

Justification: The paper makes use of several existing assets, including open-source toolkits (e.g,, MMDettection3) and public datasets (e.g. SUN RGB-D, ScanNet, KITTI). All assets are properly cited in the main paper. We have ensured that all data and code dependencies are used in accordance with their terms of use.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the assct's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The paper introduces OP3Det, and the implementation code will be released soon. The release will include documentation covering installation, training, evaluation, and usage instructions to facilitate reproducibility. No personal or sensitive data is involved.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve any experiments with human subjects or crowd- sourcing. All data used are from publicly available datasets.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
26

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve any human subjects or crowdsourcing experiments, and thus no IRB or equívalent approval is required.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration not required.
Answer: [NA]
Justification: LLMs were not used as part of the core methods experiments in this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: http:///eeeisccceeeeeeee http:////cc//oooeeeeeee///////// for what should or should not be described.
27


Reviews:
[{"review_id": "VZfp1aq9Hl", "reviewer": "Reviewer_99tt", "summary": "This paper proposes a class-agnostic open-world prompt-free 3D Detector for 3D objectness learning. The is framework adopts RGB image and point cloud as input, then outputs class-agnostic 3D bounding boxes. In the training pipieline, a 3D object discovery mechanism is used to cover more object instances in the training data. Then a cross-modal mixture of experts fusion module is proposed to fuse features from image and point cloud. Finally, a 3d object detector Uni3DETR is used to predict final outputs. The authors conduct experiments on SUN RGB-D and ScanNet dataset and compare the proposed method with other closed-world or open-vocabulary 3D object detection methods. The experiments shows that the proposed class-agnostic detector can achieve higher performances comparing to other methods.", "questions": "1. Please add more evaluation details in the rebuttal.\n2. Why did the authors need to design a new detector instead of using the expanded 3D bounding boxes to train an existing closed-world or open-vocabulary detector?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. This paper focus on a interesting task, 3D objectness learning and proposed a pipeline which can obtain high performances class-agnostic 3d bounding boxes with multi-modal inputs.\n2. The experiments show the proposed method achieve a large margin performance improvements comparing to other closed-world or open-vocabulary 3D object detection methods.\n\nWeaknesses:\n1. Missing evaluation details. \n- How many objects are retained for evaluation when computing the Average Recall (AR) metric? \n- Is the lower AR in closed-world/open-vocabulary detectors due to false negatives (FN) being filtered out by a high confidence threshold?\n2. Necessity of a new detector\n- Could the proposed 3D object discovery pipeline be used to expand training data, followed by fine-tuning an existing closed-world/open-vocabulary detector (instead of designing a new one)?\n- Would integrating SAM3D [46] with the proposed point-sampling strategy yield comparable or better 3D bounding boxes?", "was_revised": true, "final_justification": "The rebuttal address my concerns. I keep my rating as positive.", "rebuttal": "### **(Weakness 1.1) Quantifying object candidates in AR computation**\n\nWe retain **1,000 predicted objects per scene** when computing AR, following common practice in 3D detection benchmarks. This ensures a fair comparison across methods. We will clarify such evaluation details in the final version.\n\n\n### **(Weakness 1.2) On the role of confidence thresholds in AR evaluation**\n\nFor evaluation, we follow prior open-vocabulary 3D detection works (e.g., OV-Uni3DETR) and adopt a standardized post-processing strategy where a **fixed number of top-K predictions (1000 per scene)** and the **same confidence threshold (0.01)** are retained for AR computation.\n\nTherefore, the lower AR observed in closed-world or open-vocabulary detectors is not due to high thresholds filtering out correct predictions, but rather reflects the actual limitations of these models in generating high-recall proposals for novel categories. We will clarify this in the final version.\n\n\n### **(Weakness 2.1 and Question 2) Adaptability of 3D object discovery & Necessity of a new detector**\n\nYes, our 3D object discovery pipeline can be leveraged to expand training data, especially for previously unseen object categories. With a broader range of annotated objects, the discovered 3D regions can be used to fine-tune existing closed-world or open-vocabulary detectors.\n\nTo demonstrate this, we conducted experiments where we used our discovered objects to augment training data and fine-tune existing detectors such as FCAF3D[1] and Tr3D[2]. As shown in the table below, incorporating discovered objects leads to substantial improvements in AR$_{\\text{novel}}$:\n\n| Method                        | AR$_{\\text{novel}}$ | AR$_{\\text{all}}$ |\n|------------------------------|--------------------|------------------|\n| FCAF3D                       | 65.3               | 86.5             |\n| FCAF3D + object discovery    | 74.9               | 88.4             |\n| Tr3D                         | 62.1               | 84.8             |\n| Tr3D + object discovery      | 72.9               | 85.1             |\n| **OP3Det (ours)**            | **78.8**           | **89.7**         |\n\nAlthough existing detectors benefit from our discovery pipeline\\, **OP3Det further outperforms them**. It is worth noting that both FCAF3D and TR3D are multi-modal models, showing that a newly designed detector is necessary for effective multi-modal fusion in an open world.\n\n\n### **(Weakness 2.2) Evaluating the compatibility of our point-sampling strategy**\n\n  We agree that combining SAM3D[3] with our proposed point-sampling strategy could be a promising direction. To validate this, we conducted an ablation study where we integrated the SAM3D strategy with our multi-scale point sampling method before 3D box generation.\n\n| Method                            | AR$_{novel}$ |\n|----------------------------------|-------------------------|\n| SAM3D                             | 58.6                    |\n| SAM3D + multi-scale sampling      | 61.7                    |\n| **Ours (SAM + multi-scale sampling)** | **62.8**                |\n\nAs shown in the table above: SAM3D alone performs worse due to unfiltered noisy masks containing clutter; Adding our point-sampling strategy improves SAM3D performance (from 58.6% to 61.7%); Our full pipeline still outperforms (62.8%) due to additional refinement from the 2D class-agnostic detector, which helps filter non-object regions before 3D lifting.\n\nThese findings support the generalizability and effectiveness of our point-sampling method for robust 3D box generation.\n\n**Reference**\n\n[1] FCAF3D: Fully Convolutional Anchor-Free 3D Object Detection, ECCV 2022\n\n[2] TR3D: Towards Real-Time Indoor 3D Object Detection, ICIP 2023\n\n[3] SAM3D: Segment Anything in 3D Scenes, ICCV 2023 Workshop"}, {"review_id": "RNtKGs7Cf4", "reviewer": "Reviewer_PU8d", "summary": "The paper's main contribution lies in its ability to perform 3D object detection in an open-world setting without relying on predefined object categories or text prompts. The core idea behind OP3Det is to leverage the strengths of both 2D semantic priors, derived from pre-trained 2D foundation models, and 3D geometric priors to generate class-agnostic proposals. Specifically, the method employs the Segment Anything Model (SAM) to produce 2D segmentation masks from RGB images, which are then refined using a multi-scale point sampling strategy and a pre-trained class-agnostic 2D detector. These 2D masks are then lifted into 3D proposals using the available 3D geometric information. To further enhance the detection capabilities, OP3Det integrates complementary information from both point cloud and RGB images.", "questions": "Even though there are significant weaknesses I fail to overlook, I still lean towards a positive rating for this submission, as the open-world setting in 3D can be valuable to the community. For the detailed questions, please refer to the weaknesses section. I may increase or decrease my rating depends on the response from the authors.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The proposed method achieves SOTA performance compared to existing open-vocabulary 3D detection methods. \n- The multi-scale point sampling strategy and the pre-trained class-agnostic 2D detector is an interesting approach to further refine these masks, leading to more accurate 3D proposals. \n\nWeaknesses:\n\nDespite the strengths of this paper, I have identified several significant weaknesses:\n \n- Inadequate Task Definition & Contextualization (Sec. 1):\nThe paper fails to rigorously define \"Open-World Prompt-free 3D Detection\" or distinguish it meaningfully from prior art. While claiming novelty, it conflates concepts like \"class-agnostic detection,\" \"open-world,\" and \"open-vocabulary\" without clarifying the unique requirements of 3D open-worldness (e.g., handling partial occlusions, viewpoint invariance in 3D space, or scale ambiguity in point clouds). Crucially, it lacks a formal comparison against well-established 2D open-world detection paradigms (e.g., Detic, OWL-ViT). The distinction boils down to \"it’s 3D,\" ignoring whether core challenges like unknown object discovery fundamentally differ beyond the modality shift. A rigorous analysis of why 3D necessitates a wholly new task definition is absent.\n\n- Superficial Treatment of 3D-Specific Challenges (Sec. 1 & 2):\nThe discussion of challenges unique to 3D open-world detection is cursory. Beyond mentioning data scarcity, it neglects critical issues like severe sparsity/occlusion in point clouds affecting object completeness, ambiguity in separating foreground objects from complex 3D backgrounds (e.g., clutter, structures), or the inherent noise and registration errors in multi-modal (LiDAR-RGB) sensor fusion. Compared to dense 2D pixels, these factors drastically impact defining \"objectness\" in 3D. The paper acknowledges 2D richness but doesn’t deeply analyze why transferring 2D objectness concepts to sparse 3D is fundamentally harder or how their method specifically overcomes these geometric hurdles beyond using SAM and proximity.\n\n- Insufficient Baseline Comparisons (Sec. 4):\nThe baselines are limited to adapted closed-world and open-vocabulary 3D detectors. Crucially, it fails to compare against relevant SOTA methods from adjacent modalities that could serve as strong baselines or ablations. This includes: (1) SOTA 2D open-world detectors (e.g., FOUND, DetPro) applied to the input images – essential to gauge the added value of 3D; (2) Multi-view or video-based 3D detectors leveraging temporal consistency for robustness; (3) Panoramic image-based 3D understanding methods; (4) 3D self-supervised/unsupervised object discovery approaches. Omitting these leaves the reader questioning if OP3Det truly advances beyond simply applying SAM to images and projecting results.\n\n- Failure to Explore MLLM-Based Alternatives (Sec. 2 & 4):\nThe paper entirely ignores the potential of Multimodal Large Language Models (MLLMs) for open-world 3D understanding, a significant oversight in 2025. It doesn't compare against or discuss: (1) Using MLLMs directly (e.g., LLaVA-3D, 3D-LLM) for prompt-free 3D object recognition and description; (2) MLLMs as generators of descriptive text or region proposals for existing 3D open-vocabulary models; (3) MLLMs for fusion instead of the proposed MoE. Given the strong zero-shot capabilities of MLLMs, their absence as a baseline or discussion point weakens the claim that OP3Det’s approach (relying heavily on SAM) is optimal or state-of-the-art for prompt-free understanding.\n\n- Inadequate Ablation Studies (Sec. 4.5):\nThe ablations are superficial and fail to validate core design choices critically. Crucially, they do not ablate the necessity of the 3D (point cloud) modality itself. Removing 3D point cloud input (testing RGB-only model's performance) is essential to demonstrate if 3D geometry is truly indispensable for their defined \"objectness\" or if strong 2D features (SAM + MoE) suffice. Furthermore, ablations lack: (1) Testing the impact of different 2D foundation models beyond SAM; (2) Removing the MoE router to show the value of dynamic fusion vs. static fusion; (3) Quantifying the contribution of the 2D detector post-processing independently. Without these, the claimed benefits of multi-modal 3D fusion and the MoE architecture remain weakly supported.", "was_revised": true, "final_justification": "The rebuttal and discussions from the authors partly address my concerns. Nevertheless, I maintain my rating for the effort and other addressed points. Please incorporate the content from your rebuttal (e.g., added ablation experiments) into the final version of the submission.", "rebuttal": "### **(Weakness 1) Task Definition & Contextualization for \"Open-World Prompt-free 3D Detection\"**\n\n  (1) \"Class-agnostic detection\" refers to the task of detecting object instances without predicting their specific category labels. It simply determines whether a region contains an object. \"Open-vocabulary detection\" assumes access to category names or prompts and aims to recognize objects from an open set, often including unseen classes. \"Open-world detection\" emphasizes generalization to unknown categories. Our proposed task, open-world prompt-free 3D detection, is a class-agnostic formulation that avoids any use of semantic labels or prompts. Instead, it focuses on learning 3D objectness with the goal of detecting all object instances, especially novel ones. Grounded in 3D detection, it contributes to the broader vision of open-world learning through a semantic-free, discovery-driven perspective.\n\n  (2) Learning 3D objectness poses fundamentally different challenges compared to 2D objectness. Unlike in 2D vision, where object boundaries are often well-defined in dense pixel grids, 3D point clouds are sparse, noisy, and incomplete due to occlusions and sensor limitations, making it significantly harder to infer objectness. Furthermore, 3D datasets are scarce, manually annotated, and limited in category diversity, in contrast to the abundance of large-scale image-text datasets available for 2D. These fundamental differences make the direct transfer of 2D open-world methods like Detic or OWL-ViT to 3D neither straightforward nor effective. We will clarify these in the revised version.\n\n### **(Weakness 2) Superficial Treatment of 3D-Specific Challenges**\n\n While our original submission emphasized data scarcity and modality fusion, our model was in fact designed with these 3D-specific challenges, aiming to better learn 3D objectness. For the sparsity problem, we utilize a voxel-based architecture that supports learning from sparse point clouds. For occlusion and foreground-background ambiguity, we incorporate RGB images that anchor dense 2D semantics into 3D space. The multi-modal paradigm allows the model to effectively resolve incomplete geometry, missing parts, and ambiguous object boundaries, which fundamentally motivates our choice of a multi-modal framework.\n\n We also emphasize that transferring objectness from 2D to 3D is non-trivial. Unlike dense RGB pixels, 3D data is incomplete, lacks uniform resolution, and contains structural noise, making direct application of 2D paradigms inadequate without careful cross-modal alignment and fusion.\n\n### **(Weakness 3) Insufficient Baseline Comparisons**\n\nTo further demonstrate the effectiveness of our OP3Det, we re-implement more recent methods as baselines and compare them, as shown in the table below. The 2D open-world method DetPro[1], when combined with 2D-to-3D projection, underperforms compared to our OP3Det (72.4% vs. 78.8%). This is because the projection of 2D masks into 3D introduces information loss and geometric errors, and such methods fail to exploit the complementarity of RGB and point cloud modalities. Multi-view 3D detectors such as ImVoxelNet[2] and NeRF-Det[3] also yield much lower performance (34.2% and 37.5%, respectively), as they rely solely on 2D images. These methods do not utilize 3D point clouds, where abundant 3D information exists, making them ill-suited for precise 3D object detection under open-world conditions. We also include PointContras[4], a representative 3D self-supervised approach, which achieves AR$_{novel}$ of 73.3% but is still outperformed by OP3Det. This is because PointContrast is designed for representation learning, not open-world object detection, and lacks both explicit 3D objectness modeling and localization knowledge. Together, these results demonstrate that OP3Det is not merely “SAM + projection,” but a carefully designed framework that addresses the core challenges of 3D open-world detection over multi-modal inputs.\n  \n  | method                 | AR$_{novel}$ | AR$_{all}$ | AR$_{base}$|\n  |------------------------|--------------|------------|------------|\n  | DetPro + Projection (SOTA 2D open-world)   | 72.4     |  85.3  |     89.4      |\n  |ImVoxelNet (multi-view) | 34.2        |\t 62.6     |\t  70.8.     |\n  |NeRF-Det (multi-view) | 37.5        |\t 64.2     |\t  71.4     |    \n  | PointContrast (3D self-supervised) |   73.3 |  82.8 | 85.7  |\n  | **OP3Det (ours)**      | **78.8**    | **89.7**   | **93.1**    |\n\n### **(Weakness 4) Failure to Explore MLLM-Based Alternatives**\n\n(1) Existing MLLMs such as LLaVA-3D[6] and 3D-LLM[7] are not directly applicable baselines for our task. Despite recent progress, most of the existing 3D MLLMs still cannot accept raw point clouds as input. Instead, they rely on intermediaries such as multi-view images or depth images to process 3D information. Therefore, in terms of input modality, these methods are difficult to apply directly to our scenario.\n\n (2) We argue that current MLLMs are typically not suitable tools for generating dense region proposals in our setting. These models are inherently prompt-driven and operate over 2D image inputs. Given that a user query is conditioned on language, they typically output only a small number of semantic-level bounding boxes or region captions and cannot generate dense region proposals with confidence scores required for 3D object discovery. In comparison, our method targets prompt-free, class-agnostic 3D object discovery by generating hundreds to thousands of candidate regions per scene to achieve high recall, a capability that current MLLMs are not equipped for.\n\n (3) While MLLMs excel at vision-language tasks, they lack the ability to perform fine-grained, token-level fusion of RGB and point cloud features. Our **MoE** module is specifically designed for this purpose, enabling spatially adaptive fusion based on scene context. It dynamically adjusts modality weights, which is crucial for accurate, prompt-free 3D detection.\n\n### **(Weakness 5) Inadequate Ablation Studies**\n\n**(1) Models beyond SAM.** While our pipeline is built on SAM due to its strong generalization ability and wide adoption, we agree that evaluating alternative or improved segmentation backbones can provide additional insight. To this end, we conducted an experiment replacing the default SAM with HQ-SAM[5], a recent enhanced variant that introduces special tokens and global-local feature fusion for improved mask quality. All other components of the pipeline were kept unchanged. These findings validate the flexibility of our pipeline and indicate that stronger 2D priors can translate into better 3D detection performance.\n\n| 2D Foundation Model | AR$_{novel}$ | AR$_{all}$ | AR$_{base}$ | AP   |\n|---------------------|:------:|:-----------:|:----------:|:----:|\n| SAM                 | 78.8                      | 89.7                     | 93.1                      | 65.4  |\n| HQ-SAM              | **79.1**                   | **90.2**                  | **93.6**                   | **65.8** |\n\n**(2) Necessity of the 3D point cloud modality.** To assess the indispensability of 3D input, we ablate the point cloud modality and only use RGB images as inputs. As shown in the Table, removing 3D leads to a significant performance drop (e.g., AR$_{novel}$ drops from 74.4% to 38.4%), especially on novel categories. This confirms that 2D alone lacks sufficient spatial and structural information to model 3D objectness in open-world 3D scenes, where distinguishing foreground from cluttered backgrounds is particularly difficult.\n\n\n| Modality | AR$_{novel}$ | AR$_{all}$ | AR$_{base}$ |\n|:---:|:-------------:|:-----------:|:------------:|\n| PC | 68.8         | 86.0        | 91.4         |\n| Img| 38.4         | 64.4        | 72.5         |\n| PC + Img | **74.4**     | **87.9**    | **92.2**     |\n\n\n**(3) Dynamic fusion (MoE) vs. static fusion.** To validate our fusion design, we compare our Cross-Modal Mixture-of-Experts (**CM-MoE**) with static strategies like addition and concatenation. As shown in the Table, CM-MoE consistently outperforms the baselines (e.g., AR$_{novel}$: 74.4% vs. 66.0%/65.4%), demonstrating its ability to dynamically adjust to modality-specific reliability.\n\n\n  | Method         | AR$_{novel}$ | AR$_{all}$ | AR$_{base}$ |\n  |----------------|:-------------:|:-----------:|:------------:|\n  | addition        | 65.4         | 85.6        | 91.4         |\n  | concatenation   | 66.0         | 85.8        | 92.1         |\n  | CM-MoE          | **74.4**     | **87.9**    | **92.2**     |\n\n**(4) 2D detector post-processing.** As shown in the Table, adding the 2D class-agnostic detector (LDET)[8] as a post-processing step to SAM + multi-scale point sampling yields clear performance gains:  \n  AR$_{novel}$ improves from 61.9% to 66.1%, AR$_{all}$ from 54.2% to 59.2%, and AP from 7.6% to 10.0%.\n\n\n  | Method                          | AR$_{novel}$ | AR$_{all}$ | AR$_{base}$ | AP   |\n  |---------------------------------|:-------------:|:-----------:|:------------:|:----:|\n  | w/o 2D detector post-processing  | 61.9         | 54.2        | 51.9         | 7.6  |\n  | w/ 2D detector post-processing  | **66.1**     | **59.2**    | **57.1**     | **10.0** |\n\n\n**Reference**\n\n[1] Learning To Prompt for Open-Vocabulary Object Detection With Vision-Language Model, CVPR 2022\n\n[2] ImVoxelNet: Image to Voxels Projection for Monocular and Multi-View General-Purpose 3D Object Detection, WACV 2022\n\n[3] NeRF-Det: Learning Geometry-Aware Volumetric Representation for Multi-View 3D Object Detection, ICCV 2023\n\n[4] PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding, ECCV 2020\n\n[5] Segment Anything in High Quality, NeurIPS 2023\n\n[6] LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness, 2024\n\n[7] 3D-LLM: Injecting the 3D World into Large Language Models, NeurIPS 2023\n\n[8] Learning to Detect Every Thing in an Open World, ECCV 2022"}, {"review_id": "YMnxbnJXY4", "reviewer": "Reviewer_kkf7", "summary": "This paper addresses the critical and underexplored problem of generalized 3D objectness learning in open-world scenarios. The authors introduce and formally define the novel problem setting of class-agnostic open-world 3D object detection, propose a multi-modal 3D detector specifically designed for learning open-world 3D objectness, and present extraordinary performance through extensive experiments across cross-category, cross-dataset, and outdoor generalization settings.", "questions": "1. I am curious about the phrase \"prompt-free\" used in the paper. Authors claims that \"discover 3D objects without the requirement of any category labels and text prompts\" in the caption of Figure 1. But authors still use SAM's internal prompting mechanism (e.g., a 64x64 grid of points). Could the authors clarify this distinction more explicitly, perhaps by emphasizing \"semantic prompt-free\" or \"label-free prompting\" to avoid potential misinterpretations?\n2. Is the \"3D Object Discovery\" phase (SAM inference, multi-scale point sampling, 2D detector post-processing) performed as an offline pre-processing step for the training data, or is it integrated into the online inference pipeline? If it's offline, how does its computational cost (time and memory) scale with the size of the dataset (e.g., number of scenes)? If it's online, what is its latency contribution to the overall inference time?\n3. Can the authors provide a more detailed qualitative or quantitative analysis of when the multi-modal router chooses to emphasize a specific expert (semantic, geometric, or fused)? For example, are there certain object characteristics (e.g., highly textured vs. purely geometric shapes) or scene conditions (e.g., occluded objects, sparse point clouds) where one modality's expert is predominantly weighted by the router?\n\n\n\n\n\n\n\n\nCould the authors provide a more granular analysis of the failure cases specifically from the 3D Object Discovery phase? For instance, what types of objects or scene configurations (e.g., highly occluded, very small, low-contrast, or non-rigid objects) are consistently missed or poorly localized by the SAM + multi-scale point sampling + 2D detector pipeline?", "limitations": "The authors acknowledge several limitations in the supplementary material (L124~138), such as missed detections in complex scenes due to limited point cloud coverage, reliance on accurate 2D-3D projection, and difficulties with low-texture, non-rigid, and color-homogeneous regions, they could further discuss how these limitations affect performance with more quantitative analysis, explore uncertainty estimation in 2D-3D projection, incorporating more robust representations for low-texture objects, or integrating temporal information.\n\nThe authors should add a paragraph to the \"Broader Impacts\" section discussing these potential negative impacts. They could also propose general mitigation strategies, such as emphasizing the need for diverse and unbiased training data for foundation models, advocating for ethical deployment guidelines, or highlighting the importance of human oversight in critical applications.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n1. The paper tackles a crucial and timely problem in 3D perception: detecting all objects in an open-world setting, regardless of their semantic category. This is a significant step towards truly generalized 3D scene understanding and has direct implications for real-world applications like autonomous driving and robotics, where unseen objects are common.\n2. The idea of leveraging powerful, pre-trained 2D foundation models (like SAM) to enrich 3D data with semantic priors is a strong and creative approach to overcome the inherent data scarcity and limited annotation in 3D datasets. This cross-modal knowledge transfer is a key enabler for open-world capabilities.\n3. The proposed multi-scale point sampling strategy, coupled with the 2D class-agnostic detector for post-processing, effectively addresses the practical challenges of using SAM. The ablation studies clearly demonstrate the necessity and effectiveness of these components in generating high-quality 3D object proposals, which are vital for subsequent training.\n4. OP3Det consistently achieves state-of-the-art results, showing significant improvements over existing methods across diverse benchmarks and challenging generalization scenarios. The substantial gains in average recall for novel classes are particularly impressive and directly validate the paper's core claim of improved object discovery.\n\n**Weaknesses:**\n1. The paper provides details on training compute resources but lacks a thorough quantification of the computational overhead (time, memory) associated with the initial \"3D Object Discovery\" phase (SAM inference, multi-scale point sampling, 2D detector post-processing). If this is an online process, its efficiency is critical for real-world deployment. If it's an offline pre-processing step, its scalability for extremely large datasets needs to be discussed.\n2. The method heavily relies on accurate 2D-3D projection using camera parameters. In practical, real-world scenarios with potentially noisy sensor data, imperfect calibration, or significant occlusions, errors in this projection could propagate and negatively impact the quality of the discovered 3D objects, which then serve as crucial supervision. While acknowledged as a limitation, a more detailed analysis of its impact would be beneficial.", "was_revised": true, "final_justification": "The rebuttal addressed my concerns. I keep my rating as positive.", "rebuttal": "### **(Question 1) Clarification on the use of \"prompt-free\"**  \n  We thank the reviewer for clarifying this. We will revise the paper to explicitly state that our method is **semantic prompt-free**, i.e., it requires no category labels, text prompts, or semantic priors at inference time.  \n  \n While we do utilize point-grid prompts (e.g., 64x64 uniform sampling) to interact with SAM, these prompts are purely spatial and entirely semantic-agnostic, meaning they carry no category or textual information. This design intentionally avoids biasing the model toward any specific object class and aligns with our class-agnostic manner. We will revise the manuscript to explicitly clarify that our approach is semantic prompt-free.\n\n\n### **(Weakness 1 and Question 2) Is the 3D Object Discovery phase offline or online?**  \n  The entire 3D object discovery phase, including SAM inference, multi-scale point sampling, and LDET refinement, is performed offline as a one-time pre-processing step before training. This design avoids any runtime overhead during inference, and does not impact inference latency or memory usage. In practice, we process 5,285 training samples using 4 RTX 3090 GPUs in parallel, each applying one 3D proximity threshold. On average, each GPU consumes 7,877 MB of memory, and each sample takes approximately 3.98 seconds to process. This step is decoupled from the detection pipeline. We will clarify this in the revised version. It can be seen that the inference time is not long, and measures that support parallelization can further accelerate the process, making this part of the offline algorithm relatively easy to apply to large-scale scenarios.\n\n\n### **(Question 3) Detailed qualitative or quantitative analysis of the multi-modal router**  \n  We thank the reviewer for the insightful suggestion. To better understand the behavior of our cross-modal Mixture-of-Experts (MoE) router, we conducted a quantitative analysis of expert weights across different object and scene types. The router assigns dynamic soft weights to the semantic (RGB), geometric (point cloud), and fused experts based on local reliability. We observe that in high-texture regions (e.g., books, monitors), the semantic expert is predominantly selected (averaging 62\\% of the weight), while in sparse or textureless scenes (e.g., boxes, outdoor structures), the geometric expert dominates (58\\%). In cluttered or occluded indoor environments, where neither modality is fully reliable, the fused expert contributes the highest share (47\\%), enabling the model to integrate complementary cues. These findings validate the router's ability to adaptively leverage different modalities under varying 3D conditions. We will include visualizations and detailed statistics in the supplementary material.\n\n### **(Weakness 2) Impact of 2D-to-3D Projection Error**  \n  We acknowledge the reliance on camera parameters for 2D-to-3D projection. Inaccuracies caused by sensor noise or imperfect calibration can introduce projection errors, potentially leading to mislabeling background points as foreground and generating fragmented or incomplete 3D supervision signals. To address this, we apply frustum-based filtering followed by clustering to remove outliers and suppress background points. Moreover, our cross-modal MoE module adaptively balances semantic (2D) and geometric (3D) cues, enabling the model to down-weight unreliable 3D inputs and mitigate the impact of projection noise during training.\n\nIn general, our method successfully localizes a wide range of objects, including those with challenging spatial arrangements such as closely placed or partially occluded instances. This is enabled by the joint use of 2D semantic priors and 3D geometric cues. Nonetheless, we observe occasional misses in cases involving objects with deformable surfaces, low color and texture contrast. This is likely caused by occasional failures of SAM and the 2D detector. We will include analyses of such cases in the revised version."}, {"review_id": "ePFy6DoIOT", "reviewer": "Reviewer_HxVB", "summary": "The paper introduces a novel 3D object detection approach aimed at detecting all objects in open-world scenarios, including unseen novel categories. OP3Det leverages the Segment Anything Model (SAM) and a multi-scale point sampling strategy to discover objects from 2D images and generate 3D bounding boxes. It employs a cross-modal Mixture of Experts (MoE) module to dynamically fuse point cloud and RGB image features, enhancing generalization. Experiments on SUN RGB-D, ScanNet, and KITTI datasets demonstrate the method’s effectiveness, with significant improvements in novel category detection (e.g., 13.5%-16% ARnovel increase on SUN RGB-D).\n\nContributions:1.Proposes the class-agnostic open-world 3D object detection problem. 2.Develops OP3Det, using SAM and multi-scale point sampling for object discovery. 3.Introduces a cross-modal MoE module for robust multi-modal feature fusion.", "questions": "Q1: The definition of \"3D objectness\" is vague, unclear whether it refers solely to bounding box localization or includes geometric and semantic features. In autonomous driving or robotics scenarios, how does the method distinguish foreground objects from background clutter (e.g., roadside debris, walls)? The paper does not address misdetection of non-target objects in complex scenes (e.g., cluttered ScanNet rooms or sparse KITTI point clouds).\nSuggestion: Provide a precise definition of \"3D objectness\" (e.g., mathematical or conceptual formulation) in the introduction or a dedicated section. Discuss strategies for handling background interference in scenarios like autonomous driving obstacle avoidance, supported by experiments or case studies.\n\nQ2: The multi-scale point sampling thresholds ((\\tau = {0.2, 0.5, 1, 2})) lack justification for their selection, with no explanation of the tuning process or adaptability to dense indoor versus sparse outdoor point clouds. Why is performance limited on KITTI?\nSuggestion: Provide the rationale for selecting (\\tau) values (e.g., based on point cloud density), include cross-dataset experiments (e.g., nuScenes) to validate robustness, and attach pseudocode or a detailed algorithm description for the point sampling process.\n\nQ3: Baseline methods (e.g., VoteNet, OV-Uni3DETR) are adapted by simply replacing labels with \"object,\" disregarding their design intent, leading to underestimated performance (e.g., VoteNet ARnovel at 33.7%). Why was this simplistic adaptation chosen? How does OP3Det compare to baselines optimized for class-agnostic tasks (e.g., OpenScene)?\nSuggestion: Redesign baseline adaptations to preserve their design integrity (e.g., retrain VoteNet with a class-agnostic head), compare with recent class-agnostic 3D methods (e.g., 3D-OVS), and include ablation studies to analyze relative advantages.", "limitations": "The paper insufficiently addresses its limitations and potential negative societal impacts, only briefly mentioning the low proportion of novel classes and limited performance gains on KITTI (1% AP3D improvement). It lacks systematic analysis of challenges in sparse point clouds or occlusion-heavy scenarios and does not discuss risks such as misdetections in autonomous driving or privacy concerns in surveillance applications.\n\nSuggestions for Improvement:\nAnalyze limitations in sparse point clouds and complex backgrounds, including failure cases.\nEvaluate robustness under extreme conditions (e.g., low-light environments).", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1.The paper addresses a challenging and novel problem of class-agnostic open-world 3D object detection, which is an emerging and valuable direction.\n2.The method integrates 2D semantic priors with 3D geometric cues, and proposes a SAM-based multi-scale point sampling strategy to address the issue of fragmented masks.\n3.The experimental design is relatively comprehensive, covering cross-category, cross-dataset, and both indoor and outdoor generalization evaluations, which adds credibility to the results.\n\nWeaknesses：\n1.The method combines existing components like SAM, point sampling, and multi-modal MoE, which is not inherently problematic, but it lacks systematic reasoning. The pipeline feels like a patchwork of engineering integrations, failing to demonstrate a \"core technical contribution\" or theoretical depth.\n2. The multi-scale point sampling thresholds (e.g., (\\tau = {0.2, 0.5, 1, 2})) and NMS merging rely entirely on empirical choices without theoretical justification or systematic ablation. \n3.Heavy Dependence on SAM Without Addressing Its Core Issues: The authors acknowledge SAM’s fragmented outputs and poor mask quality but address this with a crude approach (brute-force sampling + NMS + re-detection). There is no innovative improvement to SAM’s mask quality, making the pipeline heavily reliant on 2D segmentation quality and poorly adaptive to complex scenes.\n4.The so-called MoE is merely an attention mechanism followed by a routing FC layer, using softmax for linear expert combination. It lacks analysis of gating rationality or evidence of dynamic behavior across scenarios.\n5.The class-agnostic baselines are author-modified versions of existing models, with unclear adaptations for CoDA/OV-Uni3DETR and inconsistent evaluation protocols, compromising fairness and making it difficult to trust the reported performance gains.\n6.The paper frequently uses terms like “significant improvement,” and “strong generalization,”", "was_revised": true, "final_justification": "The author's response addressed my concerns and I hope the suggestions will be updated in the final version.", "rebuttal": "### **(Question 1) Defining 3D Objectness and distinguishing true objects from background clutter**  \n\n In our work, 3D objectness denotes the likelihood that a spatial region corresponds to a physically discrete object, distinguishable from background structures or noise, regardless of semantic category. Specifically, denote the input features as $F_{input}$, a learnable model that maps input features to an objectness confidence score as $\\phi(\\cdot)$, this process can be denoted as: $I \\left[ \\phi(F_{input}) > \\tau \\right]$, where $\\tau$ is the threshold for foreground-background separation; I$[\\cdot]$ is the indicator function (1 for foreground, 0 for background). Our model $\\phi$ is trained to approximate this decision function, assigning high scores to foreground objects and low scores to background or clutter.\n\n In autonomous driving or robotic applications, it is true that certain complex scenarios make distinguishing foreground objects more challenging. However, our OP3Det method is designed to mitigate this issue as much as possible in multiple ways. On the one hand, for obstacle avoidance in autonomous driving, rare and novel categories are inevitable in real-world scenarios, making it difficult to collect and label all object classes during training. In this context, the ability of OP3Det to detect new classes in an open-world setting can greatly assist the detector in improving obstacle avoidance, leading to a more robust and reliable system. To this end, we conducted experiments on nuScenes, a dataset focused on autonomous driving, treating \"car, trailer, construction vehicle, motorcycle, bicycle\" as seen classes and the remaining five as unseen classes. We measured AP for both novel classes and all classes, with the results presented in the table below. As can be seen, our method outperforms previous state-of-the-art closed-vocabulary and open-vocabulary detectors, effectively demonstrating its potential in autonomous driving applications.\n  \n  | method |  AP$_{novel}$ | AP$_{all}$ |\n  |-|-|-|\n  | Uni3DETR    | 0.63    | 32.4       |\n  | OV-Uni3DETR   |  25.4     | 51.8      |\n  | **OP3Det**   | **30.2**    | **56.3**     |\n\n  On the other hand, in the case of sparse KITTI point clouds, the multimodal fusion and the MoE architecture enable the $F_{input}$ to adaptively select information from different modalities (point cloud, RGB images, and multimodal data), allowing downstream tasks such as obstacle avoidance to be minimally affected by issues in any single modality. With these designs, the learned model $\\phi(\\cdot)$ can be more robust, making $I \\left[ \\phi(F_{input}) > \\tau \\right]$ more accurate.\n\n### **(Weaknesses 1) On the Coherent Integration and Novelty of Our Pipeline Design**  \n  **(1) Task motivation:**  \n  Our goal is to detect all objects in a 3D scene, including unseen categories. This shifts the challenge from category recognition to learning a generalized notion of “3D objectness.” Achieving this is non-trivial, especially under data scarcity and incomplete 3D information. While 2D data benefits from abundant labels and well-established foundation models, 3D data lacks both. This motivates us to exploit 2D semantic richness to support 3D generalization.  \n\n  **(2) Methodological design:**  \n  Rather than a patchwork, each module is purposefully chosen to solve a core sub-problem toward the above objective. SAM provides class-agnostic 2D priors, which we project and refine using a multi-scale point sampling strategy to robustly localize 3D candidates, resulting in a 3.6% improvement in AR. To address the inherent sparsity and occlusion in point clouds, we introduce multi-modal fusion. However, naive fusion methods (e.g., addition or concatenation) yield suboptimal results. Our Cross-Modal MoE module dynamically routes information between modalities based on local reliability, outperforming static fusion by +13.3% $AR_{novel}$.  \n\n  Each design decision is systematically motivated by the goal of learning robust, category-agnostic 3D objectness—not merely combining components, but integrating them to overcome the unique challenges of 3D open-world detection.\n\n### **(Weakness 2 and Question 2) Threshold $\\tau$ Selection and Generalization Across 3D Scenes**  \n\n  **(1) Choice of sampling thresholds.**  \n Our multi-scale point sampling is designed to improve robustness across diverse 3D scenes. By using a set of 3D proximity thresholds ($\\tau$ = {0.2, 0.5, 1, 2}), we avoid overfitting to a particular point cloud density. This design ensures that both small, dense objects (common in indoor environments) and large, sparse structures (typical in outdoor scenes) can be effectively sampled. We conducted ablation studies varying $\\tau$ and found that this particular set yields the best, regardless of whether for indoor or outdoor scenes, as shown in the table below. We also found that the performance is relatively robust to the choice of $\\tau$. This demonstrates that our multi-scale design is not arbitrarily chosen, but tuned for generality for both indoor and outdoor scenes.\n\n| Sampling scale $\\tau$ | SunRGBD (Indoor) | KITTI (Outdoor) |\n|-|-|-|\n| (0.1, 0.2, 0.5, 1)| 78.4| 66.6|\n| (0.2, 0.5, 1, 2)| 78.8| 66.7|\n| (0.5, 1, 2, 5)| 77.4| 66.0|\n| (0.2, 0.5, 1, 2, 5) | 78.3| 66.4|\n\n\n  **(2) KITTI performance and cross-dataset generalization.**  \n  We agree that KITTI poses unique challenges—its limited category diversity and the significant modality gap between RGB and LiDAR reduce the utility of 2D priors. Despite this, benefiting from the 3D object discovery process, together with the generalizable sampling thresholds, our method still generalizes well when transferred to other outdoor datasets. Specifically, when trained on KITTI and evaluated on nuScenes, OP3Det continues to achieve higher AP$_{all}$ compared to standard baselines, validating the robustness of our sampling and fusion strategy across domains.\n\n\n|**KITTI → nuScenes** ||\n|-|-|\n| Method | AP$_{all}$ |\n| Uni3DETR | 19.4 |\n| OV-Uni3DETR | 21.2  |\n| **OP3Det (ours)** | **23.5** |\n\n\n  **(3) Multi-scale point sampling algorithm pseudocode.**  \n  We include the pseudocode for our multi-scale point sampling process in the supplementary material.\n\n### **(Weaknesses 3) On the Role of SAM**  \n\n  Due to SAM's strong robustness, we chose to use it. Issues such as fragmented masks are common problems for segmentation models like SAM because they do not consider the concept of “objects” or objectness during training. Therefore, the problem we address is a fundamental and general one, rather than being closely related to scene complexity. To tackle this, we employ sampling and a 2D class-agnostic detector trained on large-scale data. Such 2D detectors typically exhibit good robustness as well, enabling them to fundamentally filter out these fragmented masks. As can be seen, since both the problem and the solution we consider are fundamental rather than specific to failures in particular scenarios, our method demonstrates generality and strong generalization ability in complex environments.\n\n### **(Weaknesses 4) The Adaptive Behavior of the MoE Module**  \n\n  While our MoE adopts a lightweight architecture—attention followed by a softmax-based routing layer—this simplicity does not undermine its effectiveness. Crucially, its strength lies in context-aware expert selection across modalities. To assess the rationality of the gating mechanism, we conducted a quantitative analysis of the expert weight distribution across diverse scenes and object types. Results demonstrate clear and meaningful patterns: the semantic expert is favored in high-texture scenes (averaging 62% of the weight), the geometric expert is prioritized in sparse or textureless regions (58%), and the fused expert dominates in cluttered indoor environments (47%). These trends are consistent with intuition and confirm that the router does not apply a static weighting scheme, but rather dynamically adjusts based on the local reliability of each modality. This empirical evidence substantiates the MoE's adaptive behavior. We sincerely thank the reviewer for raising this insightful concern and we will include this analysis in the revised version of the paper.\n\n### **(Weakness 5 and Question 3) Fairness and Integrity of Baseline Adaptations in Class-Agnostic 3D Detection**  \n\n We have already carefully modified existing methods like VoteNet, CoDA, and OV-Uni3DETR to align with our proposed open-world, class-agnostic detection setting. Specifically, for closed-vocabulary baselines like VoteNet, we retrained by replacing the original classification heads with a class-agnostic head. For open-vocabulary baselines like OV-Uni3DETR, we tested various generic textual prompts (e.g., “object”, “thing”, “entity”) and also utilized a class-agnostic head.  \n\n  We also compare our method with other class-agnostic 3D detection approaches. The mentioned OpenScene[2] and 3D-OVS[3] are both semantic segmentation methods, which are difficult to adapt to object detection. Therefore, we compare with CuTR[1], a recent method for category-agnostic 3D object detection, which aligns closely with our setting. We re-implement this method and list the comparative results in the table below. As can be seen, our method significantly outperforms CuTR across all metrics. This highlights the superiority of our approach in discovering diverse, novel 3D objects without category labels, validating the effectiveness of our objectness-based design.\n\n  | Method| AR$_{novel}$ | AR$_{all}$ | AR$_{base}$  |\n  |-|-|-|-|\n  |CuTR|60.2|83.1|91.0|\n  | **OP3Det (ours)** | **78.8** | **89.7** | **93.1** |\n\n  **Reference**\n\n[1] Cubify Anything: Scaling Indoor 3D Object Detection, CVPR 2025\n\n[2] OpenScene: 3D Scene Understanding with Open Vocabularies, CVPR 2023\n\n[3] Weakly Supervised 3D Open-vocabulary Segmentation, NeurIPS 2023"}]

Author rebuttal:

Meta-review: The paper receives four acceptance rating. Initially, the reviewers have concerns about some technical clarity (e.g., task definition) and experimental results (e.g., baseline comparisons, ablation study, computational analysis). After the rebuttal, despite the concerns about the 3D objectness definition, all the reviewers are satisfied with the addressed issues in the rebuttal. The AC closely checks the paper, reviews, rebuttal, and the discussed points, and agrees with the assessment from reviewers. Therefore, the AC recommends the acceptance rating and encourages the authors to incorporate the suggestions raised by the reviewers in the final version, as well as releasing the model and code for reproducibility.
Decision: Accept (poster)