Title: Towards 3D Objectness Learning in an Open World
Venue: NeurIPS 2025
Accepted: yes
Keywords: Open world

3D Object Detection
Abstract: Recent advancements in 3D object detection and novel category detection have made significant progress, yet research on learning generalized 3D objectness remains insufficient. In this paper, we delve into learning open-world 3D objectness, which focuses on detecting all objects in a 3D scene, including novel objects unseen during training. Traditional closed-set 3D detectors struggle to generalize to open-world scenarios, while directly incorporating 3D open-vocabulary models for open-world ability struggles with vocabulary expansion and semantic overlap. To achieve generalized 3D object discovery, We propose OP3Det, a class-agnostic Open-World Prompt-free 3D Detector to detect any objects within 3D scenes without relying on hand-crafted text prompts. We introduce the strong generalization and zero-shot capabilities of 2D foundation models, utilizing both 2D semantic priors and 3D geometric priors for class-agnostic proposals to broaden 3D object discovery. Then, by integrating complementary information from point cloud and RGB image in the cross-modal mixture of experts, OP3Det dynamically routes uni-modal and multi-modal features to learn generalized 3D objectness. Extensive experiments demonstrate the extraordinary performance of OP3Det, which significantly surpasses existing open-world 3D detectors by up to 16.0% in AR and achieves a 13.5% improvement compared to closed-world 3D detectors.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel 3D object detection approach aimed at detecting all objects in open-world scenarios, including unseen novel categories. OP3Det leverages the Segment Anything Model (SAM) and a multi-scale point sampling strategy to discover objects from 2D images and generate 3D bounding boxes. It employs a cross-modal Mixture of Experts (MoE) module to dynamically fuse point cloud and RGB image features, enhancing generalization. Experiments on SUN RGB-D, ScanNet, and KITTI datasets demonstrate the method’s effectiveness, with significant improvements in novel category detection (e.g., 13.5%-16% ARnovel increase on SUN RGB-D).\n\nContributions:1.Proposes the class-agnostic open-world 3D object detection problem. 2.Develops OP3Det, using SAM and multi-scale point sampling for object discovery. 3.Introduces a cross-modal MoE module for robust multi-modal feature fusion.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1.The paper addresses a challenging and novel problem of class-agnostic open-world 3D object detection, which is an emerging and valuable direction.\n2.The method integrates 2D semantic priors with 3D geometric cues, and proposes a SAM-based multi-scale point sampling strategy to address the issue of fragmented masks.\n3.The experimental design is relatively comprehensive, covering cross-category, cross-dataset, and both indoor and outdoor generalization evaluations, which adds credibility to the results.\n\nWeaknesses：\n1.The method combines existing components like SAM, point sampling, and multi-modal MoE, which is not inherently problematic, but it lacks systematic reasoning. The pipeline feels like a patchwork of engineering integrations, failing to demonstrate a \"core technical contribution\" or theoretical depth.\n2. The multi-scale point sampling thresholds (e.g., (\\tau = {0.2, 0.5, 1, 2})) and NMS merging rely entirely on empirical choices without theoretical justification or systematic ablation. \n3.Heavy Dependence on SAM Without Addressing Its Core Issues: The authors acknowledge SAM’s fragmented outputs and poor mask quality but address this with a crude approach (brute-force sampling + NMS + re-detection). There is no innovative improvement to SAM’s mask quality, making the pipeline heavily reliant on 2D segmentation quality and poorly adaptive to complex scenes.\n4.The so-called MoE is merely an attention mechanism followed by a routing FC layer, using softmax for linear expert combination. It lacks analysis of gating rationality or evidence of dynamic behavior across scenarios.\n5.The class-agnostic baselines are author-modified versions of existing models, with unclear adaptations for CoDA/OV-Uni3DETR and inconsistent evaluation protocols, compromising fairness and making it difficult to trust the reported performance gains.\n6.The paper frequently uses terms like “significant improvement,” and “strong generalization,”\n\nQUESTIONS:\nQ1: The definition of \"3D objectness\" is vague, unclear whether it refers solely to bounding box localization or includes geometric and semantic features. In autonomous driving or robotics scenarios, how does the method distinguish foreground objects from background clutter (e.g., roadside debris, walls)? The paper does not address misdetection of non-target objects in complex scenes (e.g., cluttered ScanNet rooms or sparse KITTI point clouds).\nSuggestion: Provide a precise definition of \"3D objectness\" (e.g., mathematical or conceptual formulation) in the introduction or a dedicated section. Discuss strategies for handling background interference in scenarios like autonomous driving obstacle avoidance, supported by experiments or case studies.\n\nQ2: The multi-scale point sampling thresholds ((\\tau = {0.2, 0.5, 1, 2})) lack justification for their selection, with no explanation of the tuning process or adaptability to dense indoor versus sparse outdoor point clouds. Why is performance limited on KITTI?\nSuggestion: Provide the rationale for selecting (\\tau) values (e.g., based on point cloud density), include cross-dataset experiments (e.g., nuScenes) to validate robustness, and attach pseudocode or a detailed algorithm description for the point sampling process.\n\nQ3: Baseline methods (e.g., VoteNet, OV-Uni3DETR) are adapted by simply replacing labels with \"object,\" disregarding their design intent, leading to underestimated performance (e.g., VoteNet ARnovel at 33.7%). Why was this simplistic adaptation chosen? How does OP3Det compare to baselines optimized for class-agnostic tasks (e.g., OpenScene)?\nSuggestion: Redesign baseline adaptations to preserve their design integrity (e.g., retrain VoteNet with a class-agnostic head), compare with recent class-agnostic 3D methods (e.g., 3D-OVS), and include ablation studies to analyze relative advantages.\n\nLIMITATIONS:\nThe paper insufficiently addresses its limitations and potential negative societal impacts, only briefly mentioning the low proportion of novel classes and limited performance gains on KITTI (1% AP3D improvement). It lacks systematic analysis of challenges in sparse point clouds or occlusion-heavy scenarios and does not discuss risks such as misdetections in autonomous driving or privacy concerns in surveillance applications.\n\nSuggestions for Improvement:\nAnalyze limitations in sparse point clouds and complex backgrounds, including failure cases.\nEvaluate robustness under extreme conditions (e.g., low-light environments).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the critical and underexplored problem of generalized 3D objectness learning in open-world scenarios. The authors introduce and formally define the novel problem setting of class-agnostic open-world 3D object detection, propose a multi-modal 3D detector specifically designed for learning open-world 3D objectness, and present extraordinary performance through extensive experiments across cross-category, cross-dataset, and outdoor generalization settings.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. The paper tackles a crucial and timely problem in 3D perception: detecting all objects in an open-world setting, regardless of their semantic category. This is a significant step towards truly generalized 3D scene understanding and has direct implications for real-world applications like autonomous driving and robotics, where unseen objects are common.\n2. The idea of leveraging powerful, pre-trained 2D foundation models (like SAM) to enrich 3D data with semantic priors is a strong and creative approach to overcome the inherent data scarcity and limited annotation in 3D datasets. This cross-modal knowledge transfer is a key enabler for open-world capabilities.\n3. The proposed multi-scale point sampling strategy, coupled with the 2D class-agnostic detector for post-processing, effectively addresses the practical challenges of using SAM. The ablation studies clearly demonstrate the necessity and effectiveness of these components in generating high-quality 3D object proposals, which are vital for subsequent training.\n4. OP3Det consistently achieves state-of-the-art results, showing significant improvements over existing methods across diverse benchmarks and challenging generalization scenarios. The substantial gains in average recall for novel classes are particularly impressive and directly validate the paper's core claim of improved object discovery.\n\n**Weaknesses:**\n1. The paper provides details on training compute resources but lacks a thorough quantification of the computational overhead (time, memory) associated with the initial \"3D Object Discovery\" phase (SAM inference, multi-scale point sampling, 2D detector post-processing). If this is an online process, its efficiency is critical for real-world deployment. If it's an offline pre-processing step, its scalability for extremely large datasets needs to be discussed.\n2. The method heavily relies on accurate 2D-3D projection using camera parameters. In practical, real-world scenarios with potentially noisy sensor data, imperfect calibration, or significant occlusions, errors in this projection could propagate and negatively impact the quality of the discovered 3D objects, which then serve as crucial supervision. While acknowledged as a limitation, a more detailed analysis of its impact would be beneficial.\n\nQUESTIONS:\n1. I am curious about the phrase \"prompt-free\" used in the paper. Authors claims that \"discover 3D objects without the requirement of any category labels and text prompts\" in the caption of Figure 1. But authors still use SAM's internal prompting mechanism (e.g., a 64x64 grid of points). Could the authors clarify this distinction more explicitly, perhaps by emphasizing \"semantic prompt-free\" or \"label-free prompting\" to avoid potential misinterpretations?\n2. Is the \"3D Object Discovery\" phase (SAM inference, multi-scale point sampling, 2D detector post-processing) performed as an offline pre-processing step for the training data, or is it integrated into the online inference pipeline? If it's offline, how does its computational cost (time and memory) scale with the size of the dataset (e.g., number of scenes)? If it's online, what is its latency contribution to the overall inference time?\n3. Can the authors provide a more detailed qualitative or quantitative analysis of when the multi-modal router chooses to emphasize a specific expert (semantic, geometric, or fused)? For example, are there certain object characteristics (e.g., highly textured vs. purely geometric shapes) or scene conditions (e.g., occluded objects, sparse point clouds) where one modality's expert is predominantly weighted by the router?\n\n\n\n\n\n\n\n\nCould the authors provide a more granular analysis of the failure cases specifically from the 3D Object Discovery phase? For instance, what types of objects or scene configurations (e.g., highly occluded, very small, low-contrast, or non-rigid objects) are consistently missed or poorly localized by the SAM + multi-scale point sampling + 2D detector pipeline?\n\nLIMITATIONS:\nThe authors acknowledge several limitations in the supplementary material (L124~138), such as missed detections in complex scenes due to limited point cloud coverage, reliance on accurate 2D-3D projection, and difficulties with low-texture, non-rigid, and color-homogeneous regions, they could further discuss how these limitations affect performance with more quantitative analysis, explore uncertainty estimation in 2D-3D projection, incorporating more robust representations for low-texture objects, or integrating temporal information.\n\nThe authors should add a paragraph to the \"Broader Impacts\" section discussing these potential negative impacts. They could also propose general mitigation strategies, such as emphasizing the need for diverse and unbiased training data for foundation models, advocating for ethical deployment guidelines, or highlighting the importance of human oversight in critical applications.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper's main contribution lies in its ability to perform 3D object detection in an open-world setting without relying on predefined object categories or text prompts. The core idea behind OP3Det is to leverage the strengths of both 2D semantic priors, derived from pre-trained 2D foundation models, and 3D geometric priors to generate class-agnostic proposals. Specifically, the method employs the Segment Anything Model (SAM) to produce 2D segmentation masks from RGB images, which are then refined using a multi-scale point sampling strategy and a pre-trained class-agnostic 2D detector. These 2D masks are then lifted into 3D proposals using the available 3D geometric information. To further enhance the detection capabilities, OP3Det integrates complementary information from both point cloud and RGB images.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The proposed method achieves SOTA performance compared to existing open-vocabulary 3D detection methods. \n- The multi-scale point sampling strategy and the pre-trained class-agnostic 2D detector is an interesting approach to further refine these masks, leading to more accurate 3D proposals. \n\nWeaknesses:\n\nDespite the strengths of this paper, I have identified several significant weaknesses:\n \n- Inadequate Task Definition & Contextualization (Sec. 1):\nThe paper fails to rigorously define \"Open-World Prompt-free 3D Detection\" or distinguish it meaningfully from prior art. While claiming novelty, it conflates concepts like \"class-agnostic detection,\" \"open-world,\" and \"open-vocabulary\" without clarifying the unique requirements of 3D open-worldness (e.g., handling partial occlusions, viewpoint invariance in 3D space, or scale ambiguity in point clouds). Crucially, it lacks a formal comparison against well-established 2D open-world detection paradigms (e.g., Detic, OWL-ViT). The distinction boils down to \"it’s 3D,\" ignoring whether core challenges like unknown object discovery fundamentally differ beyond the modality shift. A rigorous analysis of why 3D necessitates a wholly new task definition is absent.\n\n- Superficial Treatment of 3D-Specific Challenges (Sec. 1 & 2):\nThe discussion of challenges unique to 3D open-world detection is cursory. Beyond mentioning data scarcity, it neglects critical issues like severe sparsity/occlusion in point clouds affecting object completeness, ambiguity in separating foreground objects from complex 3D backgrounds (e.g., clutter, structures), or the inherent noise and registration errors in multi-modal (LiDAR-RGB) sensor fusion. Compared to dense 2D pixels, these factors drastically impact defining \"objectness\" in 3D. The paper acknowledges 2D richness but doesn’t deeply analyze why transferring 2D objectness concepts to sparse 3D is fundamentally harder or how their method specifically overcomes these geometric hurdles beyond using SAM and proximity.\n\n- Insufficient Baseline Comparisons (Sec. 4):\nThe baselines are limited to adapted closed-world and open-vocabulary 3D detectors. Crucially, it fails to compare against relevant SOTA methods from adjacent modalities that could serve as strong baselines or ablations. This includes: (1) SOTA 2D open-world detectors (e.g., FOUND, DetPro) applied to the input images – essential to gauge the added value of 3D; (2) Multi-view or video-based 3D detectors leveraging temporal consistency for robustness; (3) Panoramic image-based 3D understanding methods; (4) 3D self-supervised/unsupervised object discovery approaches. Omitting these leaves the reader questioning if OP3Det truly advances beyond simply applying SAM to images and projecting results.\n\n- Failure to Explore MLLM-Based Alternatives (Sec. 2 & 4):\nThe paper entirely ignores the potential of Multimodal Large Language Models (MLLMs) for open-world 3D understanding, a significant oversight in 2025. It doesn't compare against or discuss: (1) Using MLLMs directly (e.g., LLaVA-3D, 3D-LLM) for prompt-free 3D object recognition and description; (2) MLLMs as generators of descriptive text or region proposals for existing 3D open-vocabulary models; (3) MLLMs for fusion instead of the proposed MoE. Given the strong zero-shot capabilities of MLLMs, their absence as a baseline or discussion point weakens the claim that OP3Det’s approach (relying heavily on SAM) is optimal or state-of-the-art for prompt-free understanding.\n\n- Inadequate Ablation Studies (Sec. 4.5):\nThe ablations are superficial and fail to validate core design choices critically. Crucially, they do not ablate the necessity of the 3D (point cloud) modality itself. Removing 3D point cloud input (testing RGB-only model's performance) is essential to demonstrate if 3D geometry is truly indispensable for their defined \"objectness\" or if strong 2D features (SAM + MoE) suffice. Furthermore, ablations lack: (1) Testing the impact of different 2D foundation models beyond SAM; (2) Removing the MoE router to show the value of dynamic fusion vs. static fusion; (3) Quantifying the contribution of the 2D detector post-processing independently. Without these, the claimed benefits of multi-modal 3D fusion and the MoE architecture remain weakly supported.\n\nQUESTIONS:\nEven though there are significant weaknesses I fail to overlook, I still lean towards a positive rating for this submission, as the open-world setting in 3D can be valuable to the community. For the detailed questions, please refer to the weaknesses section. I may increase or decrease my rating depends on the response from the authors.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a class-agnostic open-world prompt-free 3D Detector for 3D objectness learning. The is framework adopts RGB image and point cloud as input, then outputs class-agnostic 3D bounding boxes. In the training pipieline, a 3D object discovery mechanism is used to cover more object instances in the training data. Then a cross-modal mixture of experts fusion module is proposed to fuse features from image and point cloud. Finally, a 3d object detector Uni3DETR is used to predict final outputs. The authors conduct experiments on SUN RGB-D and ScanNet dataset and compare the proposed method with other closed-world or open-vocabulary 3D object detection methods. The experiments shows that the proposed class-agnostic detector can achieve higher performances comparing to other methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. This paper focus on a interesting task, 3D objectness learning and proposed a pipeline which can obtain high performances class-agnostic 3d bounding boxes with multi-modal inputs.\n2. The experiments show the proposed method achieve a large margin performance improvements comparing to other closed-world or open-vocabulary 3D object detection methods.\n\nWeaknesses:\n1. Missing evaluation details. \n- How many objects are retained for evaluation when computing the Average Recall (AR) metric? \n- Is the lower AR in closed-world/open-vocabulary detectors due to false negatives (FN) being filtered out by a high confidence threshold?\n2. Necessity of a new detector\n- Could the proposed 3D object discovery pipeline be used to expand training data, followed by fine-tuning an existing closed-world/open-vocabulary detector (instead of designing a new one)?\n- Would integrating SAM3D [46] with the proposed point-sampling strategy yield comparable or better 3D bounding boxes?\n\nQUESTIONS:\n1. Please add more evaluation details in the rebuttal.\n2. Why did the authors need to design a new detector instead of using the expanded 3D bounding boxes to train an existing closed-world or open-vocabulary detector?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-06"}