ImOV3D: Learning Open Vocabulary Point  Clouds 3D Object Detection from Only 2D Images

Authors: timing  yang
yuanliang  ju
li  yi
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 16:27:42

Abstract: Open-vocabulary 3D object detection (OV-3Det) aims to generalize beyond the limited number of base categories labeled during the training phase. The biggest bottleneck is the scarcity of annotated 3D data, whereas 2D image datasets are abundant and richly annotated. Consequently, it is intuitive to leverage the wealth of annotations in 2D images to alleviate the inherent data scarcity in OV-3Det. In this paper, we push the task setup to its limits by exploring the potential of using solely 2D images to learn OV-3Det. The major challenges for this setup is the modality gap between training images and testing point clouds, which prevents effective integration of 2D knowledge into OV-3Det. To address this challenge, we propose a novel framework ImOV3D to leverage pseudo multimodal representation containing both images and point clouds (PC) to close the modality gap. The key of ImOV3D lies in flexible modality conversion where 2D images can be lifted into 3D using monocular depth estimation and can also be derived from 3D scenes through rendering. This allows unifying both training images and testing point clouds into a common image-PC representation, encompassing a wealth of 2D semantic information and also incorporating the depth and structural characteristics of 3D spatial data. We carefully conduct such conversion to minimize the domain gap between training and test cases. Extensive experiments on two benchmark datasets, SUNRGBD and ScanNet, show that ImOV3D significantly outperforms existing methods, even in the absence of ground truth 3D training data. With the inclusion of a minimal amount of real 3D data for fine-tuning, the performance also significantly surpasses previous state-of-the-art. Codes and pre-trained models are released on the  https://github.com/yangtiming/ImOV3D.

Main paper: 
 introduction: 
 In the 3D vision community, there is a notable surge in interest surrounding open-vocabulary 3D object detection (OV-3Det). This task focuses on the detection of objects from unbounded categories that were not present during the training phase, using 3D point clouds as input. Such capability holds immense significance in dynamic 3D environments where a wide range of object categories constantly emerge and evolve, which is critical in downstream applications including robotics [8, 18, 28, 23] , autonomous driving [31, 46] , and augmented reality [35, 44] .With the advancements in OV-3Det, which is not only scarce in terms of labels but also in the data itself. However, the collection and annotation of 3D point clouds scenes pose significant challenges. The availability of accessible and scannable scenes (e.g. indoor scenes) may be limited. Additionally, obtaining 3D annotations often requires substantial human effort and time-consuming. These limitations impact the model's performance in handling novel objects. Existing methods [30, 29, 28, 10, 49] seek help from powerful open-vocabulary 2D detectors. A common method In the inference phase, with only point clouds as input, we still construct a pseudo-multimodal representation to enhance detection performance. leverages paired RGB-D data together with 2D detectors to generate 3D pseudo labels to address the label scarcity issue, as shown in Figure 1 left. But they are still restricted by the small scale of existing paired RGB-D data. Moreover, the from scratch trained 3D detector can hardly inherit from powerful open-vocabulary 2D detector models directly due to the modality difference. We then ask the question, what is the best way to transfer 2D knowledge to 3D for OV-3Det?Observing that the modality gap prevents a direct knowledge transfer, we propose to leverage a pseudo multi-modal representation to close the gap. On one hand, we can lift a 2D image into a pseudo-3D representation through estimating the depth and camera matrix. On the other hand, we can convert a 3D point cloud into a pseudo-2D representation through rendering. The pseudo RGB image-PC multimodal representation could serve as a common ground for better transferring knowledge from 2D to 3D.In this paper, we present ImOV3D, which addresses these challenges by employing pseudomultimodal representation as a unified framework. As shown in Figure 1 right side, In both the training and the inference phase, we construct pseudo-multimodal representation to achieve our goal of training solely with 2D images and better integrating multimodal features to enhance the performance of OV-3Det. Our key idea lies in proper modality conversion. Specifically, the entire pipeline consists of two flows: (1) Image → Pseudo PC, by leveraging a large-scale 2D images training set, our method begins by converting images to pseudo point clouds through monocular depth estimation and approximate camera parameter. We automatically generate pseudo 3D labels based on 2D annotations, providing the necessary training data. We also designed a set of revision modules, which significantly improve the quality of the pseudo 3D data through the use of GPT-4 [1] 's size prior and the orientation of the estimated normal map. (2) Pseudo PC → Pseudo Image, we learn a point clouds renderer capable of producing natural-looking textured 2D images from pseudo 3D point clouds. This enables ImOV3D to leverage pseudo-multimodal 3D detection even for point cloud-only inputs during inference, transferring 2D rich semantic information and proposals into the 3D space, further enhancing the detector's performance.Despite being trained solely with the 2D image set, ImOV3D exhibits impressive detection results when directly processing real 3D scans. This is attributed to the high fidelity of the lifted point clouds and the point clouds rendering. Additionally, when a small amount of real 3D data becomes available, even without any 3D annotations, ImOV3D can further narrow the gap between pseudo and real data by fine-tuning on such 3D data, leading to improved detection performance. To validate the effectiveness of ImOV3D, we perform extensive experiments on two benchmark datasets: SUNRGBD [43] and ScanNet [12] . Notably, in scenarios where real 3D training data is unavailable, ImOV3D surpasses previous state-of-the-art open-vocabulary 3D detectors by an mAP@0.25 improvement of at least 7.14% on SUNRGBD and 6.78% on ScanNet. Furthermore, when real 3D training data is accessible, ImOV3D continues to outperform various challenging baselines by a large margin. Thorough ablations are also conducted to validate the efficacy of our designs. In summary, our contributions are three-fold:• We propose ImOV3D, the first OV-3Det method that can be trained solely with 2D images without requiring any 3D point clouds or 3D annotations.• We introduce a novel pseudo-multimodal representation pipeline which converts 2D internet images and corresponding detections into pseudo point clouds, pseudo 3D annotations, and point clouds renderings to support point clouds-based multimodal OV-3Det.• ImOV3D achieves state-of-the-art performance on two general OV-3Det benchmark datasets across various settings, showcasing its ability to enhance open world 3D understanding despite the lack of 3D data and annotations. 
related work: 
 Open-Vocabulary 2D Object Detection encompasses two primary series of works: The first [50, 19, 32, 42, 3, 15, 37, 45, 11, 9, 50] , which draws upon knowledge from pre-trained Vision-Language models (e.g., CLIP [40] ), comprehends the relationships between images and their corresponding textual descriptions, thereby enhancing object recognition and classification. The second series [26, 47, 48, 51, 17, 57, 33, 25, 58] involves the use of extensive training data, specifically text/image pairs, enabling the model to learn a more diverse set of object representations. Detic [58] leverages vocabularies from image classification datasets to train the classification head of object detectors, addressing the issue of insufficient training data and enabling inference on a larger vocabulary set.In the 2D component of our pseudo-multimodal detector, we utilize Detic [58] to predict 2D labels and bounding boxes. These 2D visual information features are then converted and augmented for the 3D point cloud detector, significantly enhancing our model's ability to recognize a broader range of objects.Open-Vocabulary Scene Understanding has recently gained increased attention [36, 41, 54, 21, 22, 24, 14, 10] and plays a critical role in robotics, autonomous driving, etc. OpenScene [36] achieves open-world scene understanding without the need for labeled data by densely embedding 3D scene points together with text and image pixels into the CLIP [40] feature space. PLA [14] develops a hierarchical approach to pairing 3D data with text for open-world 3D learning. We focus on OV-3Det, where merely extracting CLIP [40] features is insufficient. We also require the intricate spatial structure of point clouds to enhance detection accuracy and robustness. By integrating both CLIP [40] 's visual knowledge and the detailed geometric information from point clouds, our approach aims to enable the recognition of a broader range of objects beyond the predefined categories.Open-Vocabulary 3D Object Detection in 3D vision is still in its early stages, especially when compared to traditional 3D object detection [38, 39, 34, 27, 6] . OV-3DETIC [29] leverages Ima-geNet1K [13] to expand the detector's vocabulary set and conducts contrastive learning between images and point clouds modalities for more effective knowledge transfer. OV-3DET [30] generates pseudo 3D annotations for localization using a pre-trained 2D open-vocabulary detector [58] . CoDA [5] leverages 2D and 3D prior information and a cross-modal alignment module to simultaneously learn the localization and classification capabilities. CoDAv2 [7] improves CoDA [5] further by proposing the novel object enrichment strategy and 2D box guidance. FM-OV3D [52] combines multiple foundation models without the need for 3D annotations. However, they are still subject to the influence of the volume of 3D data and still require strict correspondence between RGB-D data. Our method can generate training data for OV-3Det task using only 2D images, without any 3D ground truth data. It can directly achieve state-of-the-art performance when tested on the evaluation set. The designed pseudo-multimodal representation pipeline provides a novel solution for the utilization of both 2D and 3D information.3 Method pre-training using pseudo 3D point clouds and corresponding annotations. Subsequently, we initiate an adaptation stage aimed at minimizing the domain discrepancy between 2D and 3D datasets. 
point cloud lifting module: 
 The success of open-vocabulary object detection relies heavily on the availability of large-scale labeled datasets. However, the scarcity of comparable 3D datasets poses a challenge for open world 3D Object Detection. To address this, we bridge 2D images I 2D ∈ R M ×H×W ×3 (where M is the number of images, and H and W are the height and width, respectively) to 3D detection by generating pseudo 3D point clouds P pseudo ∈ R M ×N ×3 (where N is the number of points, each with coordinates (x, y, z)).Utilizing 2D datasets for 3D detection presents difficulties due to the absence of metric depth images and camera parameters. To overcome these obstacles, we use a metric depth estimation model to obtain single-view depth images D metric ∈ R M ×H×W . Additionally, we employ fixed camera intrinsics K ∈ R 3×3 , with the focal length f calculated based on a 55-degree field of view (FOV) and the image dimensions.However, the absence of camera extrinsics E = {R | t} (where R is the rotation matrix and t is the translation vector set to [0, 0, 0] ⊤ ) results in the arbitrary orientation of point clouds. To correct this, we use a rotation correction module to ensure the ground plane is horizontal, as shown in Figure 3 (a) . First, we estimate the surface normal vector at each pixel using a normal estimation model [2] , creating a normal map. From this, we selectively extract the horizontal normal vector N i at each pixel, defined as (N x , N y , N z ). We then compute the normal vector of the horizon surface as N pred = Cluster(N i ). To align N pred with the Z axis , we calculate the rotation matrix R using the following equation:EQUATIONwhere I is the identity matrix, v is the cross product of N pred and Z axis , expressed as N pred × Z axis , K is the skew symmetric matrix constructed from the vector v, represented as: In the 3D box filtering module, prompts related to object dimensions are first provided to GPT-4 to determine the mean size for each category. This mean size is then used to filter out boxes that do not meet the threshold criteria.K = 0 -v z v y v z 0 -v x -v y v x 0 (2) 𝒁 !"#$ 𝐍 !"#$ 𝐍 !"#$ 𝒁 !"#$After obtaining the camera intrinsics matrix K and the camera extrinsics matrix E through the previous steps, depth images D metric are converted into point clouds P pseudo .Building upon the vast collection of pseudo 3D point clouds P pseudo acquired from 2D datasets, our next step is to generate pseudo 3D bounding boxes B 3Dpseudo ∈ R M ×K×7 (where K is the number of bounding boxes and each box has 7 parameters: center coordinates, dimensions, and orientation).2D datasets contain rich segmentation information that can be used to generate 3D boxes by lifting.Using the camera intrinsics matrix K and camera extrinsics matrix E obtained through the Point Cloud Lifting Module, we lift the 2D bounding boxes B 2Dgt ∈ R M ×K×4 from the 2D datasets into 3D space by extracting 3D points that fall within the predicted 2D boxes, generating frustum 3D boxes B 3Dpseudo . The extracted point clouds may contain background points and outliers. To remove these, we employ a clustering [16] algorithm to analyze point clouds. Through the clustering results, we can identify and remove background points and outliers that do not belong to the target objects.However, these lifted 3D boxes may still contain noise from the depth images D metric obtained by monocular depth estimation. To address this issue, we use a 3D box filtering module to filter out inaccurate 3D boxes, as shown in Figure 3 (b). First, we construct a database of median object sizes using GPT-4 [1] . By prompting GPT-4 with "Please tell me the average length, width, and height of this [category], using meters as the unit", we obtain the median dimensions L GP T , W GP T , H GP T . Each object in a scene, defined by dimensions L, W, H, is compared to these median dimensions using a threshold T . An object is preserved if each element of:EQUATIONThe 3D box filtering module consists of two components: Train Phase Prior Size Filtering and Inference Phase Semantic Size Filtering. The first component filters out boxes that do not match the size criteria before training. The second component removes semantically similar but size-different categories during inference, preventing errors such as misidentifying a book as a bookcase.Point cloud data has inherent limitations, such as the inability of sparse point clouds to capture detailed textures. 2D images can enrich 3D data by providing additional texture information that point clouds lack. To utilize 2D images, we transform point clouds P into rendered imagesI rendered ∈ R M ×H×W .Integrating rendered images into a 3D detection pipeline is challenging. A naive approach, as mentioned in PointClip [59] , is to append raw depth values across the RGB channels, but this fails to apply a mature open-world 2D detector effectively. To leverage multimodal information without additional inputs beyond 3D point clouds, we develop a point cloud renderer to convert point clouds into detailed pseudo images. This process can also be learned solely from 2D image datasets.The point cloud renderer has two key modules: The point cloud rendering module converts point clouds P into rendered images I rendered , and the color rendering module then processes these images to produce colorized outputs using ControlNet [55] . ControlNet [55] is a method designed to control diffusion models, transforming rendered images I rendered into pseudo images I pseudo ∈ R M ×H×W ×3 .In the pretraining stage, we use the camera intrinsics K and extrinsics E from the Point Cloud Lifting Module to render P pseudo into rendered images I rendered . During adaptation and inference, we render ground truth point clouds P gt into images using the intrinsics K obtained in the same way. Due to the lack of extrinsics E, the final rendered images I rendered are obtained by finding the optimal angle from different horizontal and vertical perspectives to make the images most compact.In reality, we cannot project a point cloud while guaranteeing that every pixel corresponds to some points. There will be holes and missing areas due to point cloud imperfections or incompatible viewpoint selection. We adjust the camera's position horizontally and vertically to observe point clouds from various angles, removing obscured portions. Finally, we render the point clouds back into images from their original perspective, resulting in partial view rendered imagesI partial ∈ R M ×N ×3 .The angle range for adjustments is set from -75 to 75 degrees, with a 15-degree interval:θ h , θ v ∈ {-75 + 15k|k = 0, 1, 2, . . . , 10} • (4)where k is an integer indicating the stepwise adjustment of the camera's angle. After generating partial view rendered images I partial , the next step is to fine-tune ControlNet [55] using these images to obtain pseudo images I pseudo . Three types of data are prepared for fine-tuning: prompts, targets, and sources. RGB images I 2D from a 2D dataset serve as the targets, while the partial view rendered images I partial are the training sources. Prompts are not used during training.Finally, we use the pseudo images I pseudo and annotations B 2Dgt from 2D datasets to fine-tune an open-vocabulary 2D detector. Thus, we can use I pseudo to obtain corresponding pseudo 2D bounding boxes B 2DTpseudo ∈ R M ×K×4 .With an extensive dataset comprising abundant 3D data (P + B 3Dpseudo ) and pseudo images data I pseudo , our next step is to train a pseudo multimodal 3D detector using a two-stage approach.Training Strategy Our training process includes pretraining and adaptation stages. In the pretraining stage, we train on pseudo 3D point clouds P pseudo and annotations B 3Dpseudo , combined with pseudo images I pseudo . While the pre-trained model performs well for zero-shot detection, a significant domain gap exists between 2D and 3D datasets.In the adaptation stage, to minimize the domain gap, we follow the same approach as OV-3DET. First, a pre-trained open-vocabulary 2D detector is used to detect objects in the image. Then, these 2D boxes B 2Dpseudo ∈ R M ×K×4 , along with RGBD data, are lifted into 3D space. Through clustering to remove background and outlier points, we obtain precise and compact 3D boxes B 3Dpseudo . Finally, this processed data is used for adaptation. To further explore the benefits of pretrain, we use 3D datasets of varying sizes to test the model's performance under different data availability conditions.Loss Function In this section, we describe loss function used in the pretrain stage. By leveraging P pseudo and B 3Dpseudo , a 3D backbone is trained to obtain seed points K ∈ R K×3 , where K represents the number of seeds, along with 3D feature representations F pc ∈ R K×(3+F ) , with F denoting the feature dimension. Then, seed points are projected back into 2D space via the camera matrix. These seeds that fall within the 2D bounding boxes B 2DTpseudo retrieve the corresponding 2D cues associated with these boxes and bring them back into 3D space. These lifted 2D cues features are represented asF img ∈ R K×(3+F ′ ), where F ′ represents the feature dimension. Finally, the point cloud features F pc and image features F img are concatenated, forming the joint representation F joint ∈ R K×(3+F +F ′ ) . In the adaptation stage, P pseudo is replaced with P gt , keeping the workflow consistent with the pretrain stage.EQUATIONwhere i represents different features, such as pc, img, joint. W i is the weight corresponding to feature i. L loc represents the original localization loss function used in ImVoteNet [39] . F text denotes the feature extracted by the text encoder in CLIP.Implementation Details Our model is a point cloud-only ImVoteNet [39] +Clip architecture. The monocular depth estimation model used is ZoeDepth [4] , jointly trained on both indoor and outdoor scenes. In the pre-training phase, similar to ImVoteNet, we train for 180 epochs with an initial learning rate of 0.001. In the adaptation phase, we train for 100 epochs, reducing the learning rate to 0.0005.For 2D voting, there are three types of cues: Geometric cues, Texture cues, and Semantic Cues. Unlike ImVoteNet, we retain geometric cues but remove texture cues. For Semantic cues, instead of using a one-hot class vector, we use pre-trained CLIP text encoder features, which is more suitable for an open-vocabulary setting. 
experiments: 
 In this section, we compare our proposed ImOV3D with other baseline models. Our experimental setup is divided into two main stages: Pretraining and Adaptation. During the pretraining stage, the training data is pseudo 3D data, referring to pseudo 3D point clouds and their corresponding annotations (3D boxes). During the adaptation stage, we use ground truth point clouds and pseudo labels to minimize the domain gap. All experiments are conducted on two commonly used Object Detection datasets: SUNRGBD [43] and ScanNet [12] . Additionally, we carry out comprehensive ablation studies to validate the effectiveness of our model's components and the data generation pipeline.2D Images Dataset: We select the LVIS [20] dataset as our 2D image source for generating pseudo 3D data, utilizing 42,000 images provided in its training set, which spans 1,203 categories with rich and detailed annotations.3D Point Clouds Dataset: We select SUNRGBD [43] and ScanNet [12] as our 3D point clouds datasets for adaptation and testing, SUNRGBD [43] and ScanNet [12] encompass a diverse range of indoor environments and offer comprehensive annotations, including 2D and 3D bounding boxes for objects. We test on 20 common categories in both datasets.Evaluation Metrics: We employ mean Average Precision (mAP) at an IoU threshold of 0.25 as our primary evaluation metric. This metric effectively balances precision and recall in assessing how well our models perform on selected datasets. Pretraining: Due to the absence of existing baseline methods except OV-3DET [30] , we utilize CLIP [40] to make previous high-performance 3D detectors such as 3DETR [34] and VoteNet [38] compatible with OV3Det. Specifically, to adapt traditional point cloud detectors for Open Vocabulary detection, we first extract geometric features from point clouds. Then, we integrate CLIP [40] for classification by converting these features for compatibility with CLIP [40] visual encoder and creating textual prompts for zero-shot classification. Finally, we compare the encoded prompts with the visual features to classify objects beyond the predefined categories. Therefore, these baselines are denoted as OV-VoteNet [38] , OV-3DETR [34] .Adaptation: To ensure a fair comparison with the current SOTA OV3Det methods, during the adaptation stage, all baselines use OV-3DET [30] 's approach to generating pseudo labels for ground truth point cloud data, which serve as training data for adaptation. In this stage, comparisons are made with CoDA [5] and OV-3DET [30] .As shown in Table 1 , training solely with pseudo 3D data generated by our method, ImOV3D improves mAP@0.25 by 7.14% on SUNRGBD and 6.78% on ScanNet over the best baseline. This achievement, made without using any 3D ground truth annotated data, demonstrates the high quality of our generated data and the effectiveness of using extensive 2D datasets to enhance Open World perception. Unlike OV-VoteNet, which lacks 2D image integration, our method's mAP@0.25 outperforms OV-VoteNet by 7.43% and 6.78% on the two datasets, proving the effectiveness of our multimodal approach even with only point cloud inputs. OV-3DET and ImOV3D visualization results are shown in Figure 6 (b). We also discuss the efficiency of GPT-4 [1] in the 3D box filtering module using the SUNRGBD dataset [43] . For comparison, we select the top 10 classes with the most instances in the validation set. The volume ratio for these 10 classes is defined as Ratio V =LGT/GPT×WGT/GPT×HGT/GPT . This ratio is an insightful metric for comparing the performance of the GPT-4 powered 3D box filter module to the ground truth (GT). A ratio close to 1 indicates high precision. We calculate Ratio V for each instance and use Kernel Density Estimation (KDE) to analyze and plot the distributions of the volume ratios. Results are presented in Figure 6 (a). To validate the effectiveness of pseudo images generated by ControlNet [55] , we compare 2D depth maps from pseudo point clouds with pseudo images, shown in Figure 4 . On the SUNRGBD dataset, mAP@0.25 increased from 4.38% to 12.61%, and on the ScanNet dataset, it rose from 4.47% to 12.64% (see Table 4 ). This shows that rich texture information in 2D images significantly enhances 3D detection performance. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We have uploaded the code and fully disclosed all the necessary information required to reproduce the main experimental results, ensuring the reproducibility of the main claims and/or conclusions of the paper. We recognize the importance of enabling other researchers to replicate and verify our experimental findings and have taken appropriate measures, including providing detailed descriptions of our experimental setup and the necessary code, to facilitate transparency and reproducibility of our research. 
ablation study of data volume: 
 Our method fine-tunes with limited real ground truth 3D point cloud data and pseudo 3D annotations.Using OV-3DET's code, we train with varying data volumes. With 10% adaptation data, OV-3DET's mAP@0.25 on SUNRGBD drops from 20.46% to 15.24%, while ours drops from 22.53% to 19.24%. On ScanNet, OV-3DET's mAP@0.25 falls from 18.02% to 14.35%, and ours falls from 21.45% to 18.45% (Figure 5 (a)(b)). We observed a decrease in performance compared to using the full data set; however, our method was still able to maintain relatively high detection accuracy. This confirmed the robustness of our method and its adaptability to small datasets, enabling effective 3D Object Detection even under constrained data conditions. It also underscores the importance of developing models for OV3Det that are capable of learning from limited data and generalizing to a broader range of scenarios. Traditional 3D detectors struggle with transferability due to training and testing class differences. We test ImOV3D on ScanNet and SUN RGB-D on the opposite datasets. The results, shown in Figure 5 (c), demonstrate that our model outperforms OV-3DET by 7.1% on SUN RGB-D and 7.82% on ScanNet. ImOV3D demonstrates superior transferability across domains despite the domain gap. To validate the benefits of fine-tuning Detic with pseudo images, we compare the off-the-shelf Detic to the fine-tuned version. The fine-tuned Detic shows clear advantages in handling pseudo images.On the SUNRGBD dataset, the mAP@0.25 increases from 19.67% to 22.53%, and on the ScanNet dataset, it rises from 19.25% to 21.45% (see Table 5 ). These experiments were conducted under the adaptation setting, illustrating the model's ability to learn from and improve detection capabilities with not entirely real data. This not only confirms the efficacy of textured image but also highlights the importance of fine-tuning models to enhance their adaptability and accuracy.Ground Truth Point Clouds OV-3DET ImOV3D Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification:We have provided open access to the data and code, along with clear instructions in the supplementary material to replicate the main experimental results. This includes details on data access, preparation, and the exact commands needed to execute the experiments. Anonymous versions of data and code have been released to maintain submission integrity.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion and limitation: 
 In conclusion, this paper introduce ImOV3D, a novel framework that tackles the scarcity of annotated 3D data in OV-3Det by harnessing the extensive availability of 2D images. The framework's key innovation lies in its flexible modality conversion, which integrates 2D annotations into 3D space, thereby minimizing the domain gap between training and testing data. Empirical results on two common datasets confirm ImOV3D's superiority over existing methods, even without ground truth 3D training data, and its significant performance boost with the addition of minimal real 3D data for fine-tuning. Our method's success showcases the potential of leveraging 2D images for enhancing 3D object detection, opening new avenues for future research in pseudo-multimodal data generation and its application in 3D detection methodologies.Limitation:Although our method has demonstrated the potential of 2D images in OV-3Det tasks, especially with the proposed pseudo multimodal representation, we need dense point clouds here to ensure that the rendered images can help improve performance. In the future, we will explore more generalized strategies.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Our paper specifies all training and testing details, such as data partitioning, hyperparameter selection, and the type of optimizer used, which are essential for understanding our results. The experimental setup is presented within the core of the paper with a level of detail sufficient for reviewers to comprehend and make sense of the outcomes. Complete details are provided alongside the code, either in the appendix or as supplementary material.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
acknowledge: 
 We would like to express our gratitude to Yuanchen Ju, Wenhao Chai, Macheng Shen, and Yang Cao for their insightful discussions and contributions.Partial-View Removal: To obtain a point cloud from a 2D image, we first use a monocular depth estimation model ZoeDepth [4] to generate a depth map for the image, as shown in Fig 4 (A) . Inspired by [56] , utilizing partial-view depth images for data augmentation serves to bolster the robustness and generalization capabilities of machine learning models in computer vision tasks. By simulating realistic occlusions and offering a diverse array of viewpoints, this approach prepares models to handle real-world variability and complexity more effectively, improving their performance across a wide range of scenarios.In this process, we start by defining the point cloud and the camera viewpoints. The point cloud is a collection of points, which is formally represented as:EQUATIONwhere each point p i is a part of the 3D object or scene we are interested in.We consider two distinct viewpoints, A and B, characterized by their camera parameters θ A and θ B , respectively. These parameters include the position and orientation of the camera that define each viewpoint.Initially, we observe the point cloud from viewpoint A, resulting in a subset of points observed from this specific viewpoint, taking into account potential occlusions and the camera's field of view. This subset is denoted as:EQUATIONNext, we change our perspective to viewpoint B and observe the point cloud again, which gives us another subset of points, denoted as:EQUATIONThe key step in this process involves identifying the overlapping points that are visible from both viewpoints A and B. This overlapping set is represented as:EQUATIONAfter identifying the overlapping points, we proceed to remove these points from the subset observed from viewpoint A. The final set of points, after removing the overlapping ones, is represented as:EQUATIONFinally, this processed point cloud is rendered back into a depth image from viewpoint A. The resulting depth image, which includes holes where the overlapping points were removed, as shown in Fig 7 (B) , illustrating the effect of occlusions, is obtained through the rendering process, expressed as:EQUATIONNormal Estimator: In this paper, we employ Normal Estimator [2] for surface normal estimation.The process of normal estimation begins with the calculation of the surface normal vector at each pixel, a crucial step for understanding the orientation of surfaces in a scene. This calculation is conducted using a normal estimation model that utilizes either depth information or the 3D coordinates of each point in the scene. Mathematically, the surface normal vector at a point is defined as a 3D vector ⃗ N = (N x , N y , N z ), where ⃗ N represents the direction perpendicular to the surface at that pixel, as shown in the equation Following the estimation of normal vectors across the scene, the next phase involves the extraction of horizontal normal vectors ⃗ N i at each pixel. This selection process focuses on vectors that align with the horizontal plane of the scene, essential for applications such as ground plane detection or horizon line estimation. Once horizontal normal vectors are identified, they undergo a clustering process to find a consensus on the predominant horizontal surface orientation within the scene. This process yields ⃗ N pred , a vector representing the normal to the horizon surface, encapsulated by the equationEQUATION) (A) (B) (C)EQUATIONThe final step in the normal estimation process is the alignment of ⃗ N pred with the Z-axis of the coordinate system to ensure a standardized representation of orientation. Achieving this alignment involves calculating a rotation matrix R, which, when applied to ⃗ N pred , orients the vector parallel to the Z-axis. This is crucial for maintaining consistent orientation representation across different scenes and supports applications requiring a fixed reference frame. The alignment can be mathematically represented asEQUATIONwhere ⃗ N aligned is the resulting vector aligned with the Z-axis, ensuring a uniform approach to interpreting surface orientations in various applications.3D Box Filtering Module: Similar to [53] , this module's goal is to improve annotation quality by converting 2D bounding boxes into their 3D counterparts for objects within indoor scenes. This conversion process is pivotal for achieving a more accurate spatial representation of objects. However, the transition from 2D to 3D can introduce discrepancies between the generated 3D bounding boxes and the objects' actual dimensions. To mitigate this, we employ Large Language Models (LLMs), such as GPT-4 [1] , to obtain size priors for various object categories encountered in indoor environments. GPT-4 [1] 's capability to provide reliable size estimates ensures that the adjusted 3D bounding boxes more closely align with the real-world sizes of objects, thereby enhancing the precision of annotations in indoor scene analysis.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: Our paper provides appropriate error bars and statistical significance information for the experiments, clearly explaining the variability and the method of calculation. We have also specified the assumptions made. An error bar or the variance value is included in all the results, either in a table or a figure.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: Our paper provides sufficient information regarding the computational resources used for each experiment, including the type of computational work (CPU or GPU), memory requirements, and execution time. This information is essential for replicating the experiments and is detailed to ensure clarity and reproducibility.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Our paper discusses the broader applications of the work, we have conducted a comprehensive assessment of the societal implications that the technology might bring.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper does not present any data or models that carry a high risk of misuse, such as pre-trained language models, image generators, or scraped datasets. The research conducted does not involve risks that would necessitate the implementation of security measures to prevent abuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: This section provides a detailed discussion on the 20 categories within the SUNRGBD [43] and ScanNet [12] datasets that were not fully explored in the main paper. In this section, we will discuss the efficiency of GPT-4 [1] in the 3D box filtering module. In order to have a good comparison, we choose the SUNRGBD [43] as a visualizing dataset. To clearly demonstrate the results, we select the top 10 classes with the most instances in the validation set as our visualization targets.The volume ratio for these 10 classes is defined as Ratio V = L×W ×H (LGT/GPT×WGT/GPT×HGT/GPT) . The volume ratio provides an insightful metric to compare the performance of the 3D box filter module powered by GPT-4 [1] with the ground truth (GT). A ratio close to 1 indicates that the volume of the predicted box is very similar to the true value, indicating high precision. By calculating the volume ratio Ratio V for each instance, we gather statistical data and use Kernel Density Estimation (KDE) to analyze and plot the distributions of the volume ratios. Results are presented in Figure 8 . bed -L: 2.0, W: 1.6, H: 0.5 cabinet -L: 0.5, W: 1.0, H: 1.0 chair -L: 0.6, W: 0.6, H: 0.9 table -L: 1.0, W: 1.5, H: 0.8 desk -L: 0.7, W: 0.6, H: 0.8 lamp -L: 0.3, W: 0.3, H: 1.5 sofa -L: 2.0, W: 0.9, H: 0.9 pillow -L: 0.4, W: 0.6, H: 0.2 sink -L: 0.5, W: 0.6, H: 0.3 bookshelf -L: 0.3, W: 1.0, H: 2.0 Figure 8 fully demonstrates the effectiveness of the meansize database constructed by GPT-4 [1] . Subsequently, we used the constructed meansize in the 3D box filtering module to filter B 3Dpseudo , with the results shown in Figure 9 . 0IV.A Fine-tuning Model Details (Detic [58] and ControlNet [55] )This subsection delves into the model adjustment and parameter selection process during the fine-tuning phase.Detic [58] : To finetune Detic [58] for our specific application, we first need to clarify the necessity for such an adjustment. The off-the-shelf Detic model is designed and optimized for ground truth images. However, our requirement for Detic [58] significantly diverges because it is to be utilized within the 2D OV Detector component of our ImOV3D project. In this component, the images we use are not ground truth images but pseudo images generated through ControlNet [55] . Such images differ visually from ground truth images, which could potentially affect the detection performance of the Detic [58] model.Given this, we decided to finetune Detic [58] to better suit our specific scenario. The finetuning process involves replacing the annotations based on ground truth images, which the Detic [58] model originally uses, with annotations from a 2D dataset that are accurate for our ControlNet-generated pseudo images. This replacement is aimed at enabling Detic [58] to more accurately recognize and process objects within these generated images, considering they differ in quality and characteristics from the real images used in the model's initial training.Several key steps are involved in this finetuning process. First, we need to collect and prepare annotated data from the 2D dataset that applies to our pseudo images. Then, we use these annotations in the training process of the Detic [58] model to adapt it to our unique image characteristics. Through this process, we ensure that the Detic [58] model maintains efficiency and accuracy when dealing with images generated through ControlNet [55] , thereby maximizing its efficacy in our ImOV3D project.Finetuning the Detic [58] model not only improves its performance on specific images but also further validates the feasibility and flexibility of our approach. This process illustrates that by meticulously adjusting and optimizing model parameters in the face of varying application scenarios, we can effectively enhance the model's performance and adaptability.ControlNet [55] : The objective of finetuning ControlNet [55] is to add color to the rendered point cloud images. Originally, point cloud data lacks color information, meaning the depth images projected from these point clouds are also devoid of color. However, in many applications including but not limited to 3D reconstruction, augmented reality (AR), and virtual reality (VR), rich color and texture information are crucial for enhancing the visual quality of the final images. This necessitates the finetuning of ControlNet [55] .The finetuning process requires two types of data: original images and target images. In our scenario, the original images are depth images obtained by rendering point cloud after partial-view removal, as shown in During the finetuning of ControlNet [55] , we do not use any text prompts but instead adhere to the official parameter settings recommended by ControlNet [55] . We train the model for 180 epochs, through which the model learns how to effectively map the color information from the target images onto the corresponding depth images. This process not only enhances the visual appearance of the rendered images, making them more realistic and rich, but also improves the model's capability to handle color and texture variations across different scenes.Through such finetuning, ControlNet [55] can more accurately colorize depth images rendered from point clouds, preserving the geometric details from the original point cloud data while also integrating real-world color and texture characteristics. Such improvements are extremely beneficial for subsequent 3D vision applications, such as scene understanding and interaction in 3D. In summary, the process of finetuning ControlNet [55] is a meticulous attempt to adjust and optimize model performance, aiming to achieve better results in handling specific tasks.It outlines the setup and execution process of the main experiment, including comparisons with baseline experiments.We detail and present four sets of key pretraining parameters. Table 10 showcases the training configuration for the ImOV3D method, an improved model we propose. Table 11 provides a detailed description of the training parameters for OV-VoteNet [38] , serving as another baseline model. Table 12 focuses on OV-3DETR [34] , presenting its specific training settings. Finally, Table 13 outlines the training parameter details for the OV-3DET [30] model. These four tables aim to clearly display the specific parameter settings of each method during the pretraining stage, offering a direct perspective for comparison to the readers.Table 14 presents the parameter configurations of our experiments during the adaptation stage. Figure 11 : During the pretraining stage, the visual comparison tested on the ScanNet [12] dataset includes four columns: the first column shows the ground truth images, the second column displays the ground truth point clouds, the third column presents the detection results by OV-3DET [30] , and the fourth column reveals the detection outcomes by ImOV3D .Justification: For each theoretical result in our paper, we have provided a complete set of assumptions and a complete (and correct) proof. All theorems, formulas, and proofs are numbered and cross-referenced for clarity and consistency. All assumptions are explicitly stated or referenced in the statement of any theorem. The proofs are presented in the main paper, and we have included brief proof sketches to provide intuition. Additionally, any theorems and lemmas upon which the proofs depend are properly cited, ensuring the rigor and reliability of our theoretical results.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.Justification: Our research fully complies with the NeurIPS ethical guidelines.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: The paper aims to develop an open-vocabulary 3D detector with the help of 2D data. The key idea is to lift 2D images to 3D point clouds by using metric monocular depth estimation models, combined with estimating the extrinsics and intrinsics. Finally, the paper trains a detector on the lifted 2D image and point-cloud data to get a base detector in "pre-training" stage. Then, the pseudo labels are generated on real 3D point clouds and projected to 2D RGB images, and trained further for a few epochs. The paper compared against existing baselines like OV-3DET and outperform them. The initial key concerns are the clarity in many components, more real-world 3D dataset experiments are needed, overall pipeline complexity. In the rebuttal, the authors give more details of the key components and show additional experiments on KITTI dataset. The final ratings are two borderline accept, one weak accept, and one accept. Hence, the AC recommend accept.

Reviews: review: paper_summary: paper_summary: This work addresses the problem of the lack of 3D data and attempts to train open-vocabulary point-based 3D detection models solely with 2D images to avoid using 3D data and annotations. It proposes a pseudo data generation pipeline for this purpose, which generally follows a paradigm of estimating depth, projecting 2D to 3D, and rendering back from 3D to 2D to obtain 2D-3D paired pseudo data. Experiments show that models trained with the generated data achieve better results than previous methods.
strength_weakness: strength_weakness: My major concerns lie with the unclear motivation and implementation of some components.

1. Intuitively, using a fixed camera intrinsic causes inaccurate projection. This work does not propose specific designs for addressing this problem nor discuss the limitations of using fixed intrinsics.
2. The reasons for using Formula (1) are not explained.
3. The method for lifting 2D bounding boxes to 3D, as described from L161-L163, is unclear. Did the authors use the minimum and maximum depth of the four corners of the 2D bounding boxes to generate the frustum? How is the orientation of the 3D bounding boxes determined? How is the gap addressed between the pseudo 3D bounding boxes, which are oriented, and the 3D ground truth bounding boxes in ScanNet, which are axis-aligned?
4. For the pre-training stage, why not use the original 2D images directly for training, considering that there is still another adaptation training stage to handle the domain gap between images from real point clouds and pseudo point clouds? In other words, why not use the adaptation training stage to address the gap between real images and images from real point clouds, thereby skipping the process of generating pseudo images from pseudo point clouds?
5.  As the key contribution of this work is the data generation pipeline, there should be more qualitative results for each step to demonstrate how this pipeline truly works. In the rebuttal phase, I suggest the authors show the results (e.g., intermediate results in Figure 2, point clouds with pseudo 3D bounding boxes can be shown using Open3D) using the first image of scene0000_00 in ScanNet and the first image of scene0012_01 in ScanNet as examples.

The data generation pipeline itself is typical and lacks novelty (e.g., similar work like SpatialGPT exists). It is difficult to fully understand the implementations and evaluate the performance of the pipeline solely through the manuscript. Nevertheless, this work involves significant engineering effort and tricks and contributes to the 3D perception community. If the code is open-sourced, I suggest accepting the paper.
questions: questions: See the weakness section.
limitations: limitations: See the weakness section.
paper_summary: paper_summary: The paper introduces ImOV3D, a novel framework for open-vocabulary 3D object detection (OV-3Det) that learns exclusively from 2D images. The method addresses the scarcity of annotated 3D data by leveraging the wealth of annotations in 2D images. ImOV3D employs a pseudo-multimodal representation that bridges the gap between 2D images and 3D point clouds, enabling effective knowledge transfer. The framework converts 2D images to pseudo-3D point clouds using monocular depth estimation and vice versa, integrating 2D semantic information with 3D spatial characteristics. Extensive experiments on SUNRGBD and ScanNet datasets demonstrate significant performance improvements over existing methods.
strength_weakness: strength_weakness: Real-World Performance: How does ImOV3D perform when applied to real-world 3D data beyond the benchmark datasets used in the paper?

Adaptability: Can the model be easily adapted or fine-tuned for different types of 3D environments, such as outdoor scenes or highly dynamic environments?

Resource Requirements: What are the computational resources required for training and inference? Can the method be efficiently scaled for use in large-scale applications? What is the latency?
questions: questions: Please see the weakness.
limitations: limitations: Please see the weakness.
paper_summary: paper_summary: This paper presents a novel LiDAR-based open-vocabulary 3D detection model that relies solely on 2D images for training, without using any 3D annotations. During the training phase, a pre-trained monocular depth estimation model generates depth maps from 2D images, which are then projected into pseudo point clouds. ControlNet is used to render 2D pseudo images from these pseudo point clouds. The 3D detection model takes both pseudo 3D point clouds and pseudo images as input to predict 3D bounding boxes. During inference, the model uses ground truth point clouds and rendered 2D images to predict the 3D boxes. Additionally, the authors employ GPT to filter the pseudo annotations. The results demonstrate that this model achieves state-of-the-art performance on the SUNRGBD and ScanNet datasets. The authors have also released their code in the supplementary materials.
strength_weakness: strength_weakness: The model achieves great performance in open-vocabulary 3D detection, but there are several concerns that need to be addressed:

1. Given that the inference input is only the point cloud, why is the training input limited to images? The training data also includes point clouds; why not utilize these as well?
2. 3D Pseudo Label Generation: The pseudo point cloud is noisy, making it challenging to generate accurate 3D pseudo ground truth from it. The paper lacks details on how the 3D boxes are fitted to the point cloud and the accuracy of the pseudo ground truth. Providing these details is crucial.
3. Complex Pipeline: The overall pipeline is complex. Could the image generation step be removed and the original image used instead? During inference, could both image and LiDAR data be used, as in OV-3DET?
4. Training Data Overlap: The paper assumes that the training data does not include LiDAR. Is there any overlap between the depth model training data and the detection dataset?
5. Minor issue: Does not cite ControlNet in the main paper.
questions: questions: Please the authors address my questions presented in the weaknesses.
limitations: limitations: Not obvious limitation associated with broader societal impacts.
paper_summary: paper_summary: The paper aims to develop an open-vocabulary 3D detector with the help of 2D data. The key idea is to lift 2D images to 3D point clouds by using metric monocular depth estimation models, combined with estimating the extrinsics and intrinsics. The 2D bounding boxes are lifted to 3D as well, and filtered based on some constraints like their usual object sizes that are provided by GPT-4. Additionally, the 3D pointclouds are rendered to 2D images and re-texturerized using control-net. Finally, the paper trains a detector on the lifted 2D image and pointcloud data to get a base detector in "pre-training" stage. Then, the pseudo labels are generated on real 3D point clouds and projected to 2D RGB images, and trained further for a few epochs. The paper compared against existing baselines like OV-3DET and outperform them.
strength_weakness: strength_weakness: - the paper claims (even in the title) to make a 3D open vocabulary object detector ONLY from 2D images. However, the results indicate that such a detector is significantly worse than detectors that uses real point cloud data. I think the "only 2D images" can be misleading and the authors should consider removing/modifying it.
- While the writing for the most part is very clear, I am very unclear on the model architecture. The description seems to be missing, and from the figure it seems like the 2D images are processed by an open-vocabulary detector and 3D point clouds by a 3D detector and somehow their predictions are merged to generate the final output. The description of losses could also really benefit a lot from a more detailed description. 
- As already acknowledged in the limitations, the proposed pipelines adds significantly more complexities over OV-3DET while improving the performance of their point cloud version by about 2% (although the increase of the 2D only version is significant).
questions: questions: - Some more explanation on the model architecture and losses would help a lot
limitations: limitations: yes
paper_summary: paper_summary: This paper addresses the challenge in Open-vocabulary 3D object detection (OV-3Det), specifically the modality gap between training images and testing point clouds, which hinders effective integration of 2D knowledge into OV-3Det. The main contribution of the paper is a novel method to generate pseudo multimodal data, enabling the training of a 3D encoder to extract 3D features that are coupled with a frozen 2D detector for 3D detection.
strength_weakness: strength_weakness: The framework in Fig. 2 is confusing. The authors claim that their model's input is only point clouds, but the framework suggests that both images and point clouds are needed for inference. Additionally, the generalization ability should be discussed, particularly regarding performance on outdoor datasets.
questions: questions: What about the generalization ability? For example, how does the model perform on outdoor datasets?

The experiment needs more details. During the inference stage on the ScanNet dataset, do you use only the point cloud information without using the image information?
limitations: limitations: The authors have adequately addressed the limitations.

Comments: title: Post rebuttal
comment: Dear Reviewer TsJm, we are grateful for your valuable feedback and are committed to incorporating the suggested revisions in the next version of our manuscript！Thank you for your support！
Thank the authors for their response. It has successfully addressed my concerns. I would like to upgrade my voting of the paper. Please incorporate the suggested revisions in the next version of the manuscript.
Dear Reviewer m5vv，we are very pleased to have resolved your concern. Thank you for your affirmation, we hope to receive your support！
I appreciate the authors' clarifications.  Authors have addressed my concerns.
Thank you for your support! We are delighted that our paper has received your approval.
Cool. Now I raise my score to 7.
Dear Reviewer,

Thank you for your valuable suggestions. We agree to include a detailed discussion on the limitations of our method in the revised manuscript, and in addition, we will open source all the code to facilitate further research and community contributions!
Thanks.

I suggest the authors add more detailed limitations with potential solutions of the method in the revised paper for future improvement from the community.
Thank you for your reply! Below are our detailed answers to your questions:

**Q1:**  
When relying solely on the four corners of a 2D bounding box to estimate the 3D position of an object, we indeed encounter the issue you mentioned: it results in bounding boxes that are overly large and fail to accurately enclose the object's point cloud. To address this, we first extract 3D points within a frustum-shaped 3D bounding box. We then apply clustering techniques to remove background noise and outliers, thereby improving the precision of the bounding box relative to the point cloud. It's important to note that outlier points can inevitably occur, whether using bounding boxes or segmentation. However, while clustering might seem concerning due to the potential impact of these outliers, in practice, it does not pose a significant issue. This is because we follow up with a filtering process designed to further ensure the quality of the labels. Utilizing the clustered points, we accurately determine the object's center, dimensions, and orientation, which are crucial for refining the training of the 3D detector.

**Q2:**  
To address the challenges of over-segmentation and under-segmentation inherent in the DBSCAN algorithm, we began by systematically experimenting with a range of hyperparameters to identify the most effective combination for our dataset. Throughout this process, we closely monitored the sizes of the bounding boxes (bboxes) generated after each DBSCAN iteration, comparing them meticulously with object dimensions informed by GPT-4's common sense estimates. Our goal was to select hyperparameters that would maximize the retention of bounding box counts while ensuring that the dimensions closely aligned with the actual sizes of the objects.\
 After extensive trials and iterative refinements, we determined that setting the hyperparameters to eps=0.3 and min_samples=100 yielded the best results. These settings consistently produced optimal outcomes in our experiments, satisfying our criteria for 3D bounding boxes and ensuring that their dimensions closely matched the true object sizes. Following this, we implemented a 3D bounding box size filtering process based on these selected parameters, further enhancing the quality of both the point clouds and the bounding boxes generated post-clustering by the DBSCAN algorithm. We haven't identified significant impacts at this time, and we will release all the relevant code to benefit the community.

**Q3:**  
For horizontal plane segmentation, we set a relatively high threshold (threshold=230) to extract the horizontal plane regions from the normal map, corresponding to the darker green areas. However, the normal estimator may occasionally misclassify a few pixels as part of the horizontal plane. To mitigate the impact of these “false” planes, we implemented the following approach (as described in the paper from L146 to 149):  
After obtaining the horizontal mask, we extracted the normal vector for each pixel, \(N_i = (N_x, N_y, N_z)\). We then calculated the predominant normal vector for the horizontal plane, \(N_pred} = Cluster(N_i)\, using a clustering method to select the dominant cluster. This approach identifies the normal vector that best represents the overall horizontal plane by grouping similar vectors. Thus, even if individual pixel normals are slightly off, as long as the majority are accurate, the clustering result will generally yield a reliable normal vector for the horizontal plane.
Thanks for the reply!

I have some additional questions regarding the implementations:

1. When generating the visual frustum, if only the four corners are used, is it possible that the 3D locations of the corners of the 2D bounding box, having larger depth values than the object, could result in inaccurate cropping of the object point clouds? This concern arises because the 2D corners typically do not rest directly on the object.
2. When using DBSCAN, over-segmentation and under-segmentation often occur because the object point clouds are incomplete, especially when projected from a single view. Did the authors encounter these issues, and how did they address them?
3. For horizontal plane segmentation, the authors use a normal map, but this method appears to result in false segmentation, as evidenced by Figures 2 and 3 in the rebuttal PDF. Could the authors provide more details on this issue?
Dear Reviewer m5vv, thank you for your valuable comments. As the deadline is approaching, we kindly inquire whether our discussions have addressed your concerns. If you have any further questions, we would be happy to continue our conversation. If our response has resolved your concerns, we would greatly appreciate it if you could consider updating your score and providing feedback. Thank you again.
Dear Reviewer HQsS，thank you for your continued support and positive feedback. We're glad the detailed description and architecture diagram were helpful. Thank you again！
Dear Reviewer TsJm, thank you for your valuable comments. As the deadline is approaching, we kindly inquire whether our discussions have addressed your concerns. If you have any further questions, we would be happy to continue our conversation. If our response has resolved your concerns, we would greatly appreciate it if you could consider updating your score and providing feedback. Thank you again.
Dear Reviewer teog, thank you for your valuable comments. As the deadline is approaching, we kindly inquire whether our discussions have addressed your concerns. If you have any further questions, we would be happy to continue our conversation. If our response has resolved your concerns, we would greatly appreciate it if you could consider updating your score and providing feedback. Thank you again.
Dear Reviewer,

Thank you for your correction. We realized that we misunderstood the NeurIPS-related policy in our previous expression when comparing with SpatialVLM, which led to a lack of clear communication about the main differences between the two. In the revision, we will discuss the specific differences with SpatialVLM in more detail and highlight the key points.

Below are the important differences between ImOV3D (ours) and SpatialVLM:

1. **Innovation Focus**
   - Compared to SpatialVLM, although both involve the generation of pseudo point clouds and labels, the innovation of ImOV3D goes beyond this. The goal of ImOV3D is to create pseudo multimodal data for open-vocabulary 3D object detection. This process also includes a rendering stage, where we leverage a large number of 2D images to learn a general and color-free renderer that converts geometric point clouds into images. This allows us to generate high-quality pseudo images, which not only enhances the richness and reliability of the data but also improves our performance in 3D object detection tasks.

2. **Horizontal Plane Mask Extraction**
    - For the horizontal plane mask, ImOV3D extracts it through the normal map of the RGB image. In this normal map, the red, green, and blue channels correspond to the three different components of the normal vector. A darker green channel indicates a significant vertical component, allowing us to effectively extract the horizontal plane mask by setting a threshold for the green channel. Compared to the segmentation model used by Spatial VLM, which relies on identifying specific semantic labels such as "floor" or "table" to extract the horizontal plane mask, this method also has issues with stability and reliability because it depends on manually defined semantic categories that are not guaranteed to appear in all images. Our method directly extracts accurate geometric information from the image, providing the normal direction for each pixel point without relying on specific semantic labels. This direct extraction of normal information is widely applicable and not limited to specific scenes or objects. 

    - Additionally, Spatial VLM does not provide implementation details regarding the extrinsics estimation nor open source the code, making it difficult for us to perform a detailed methodological comparison. Our method mathematically ensures that we identify the minimal angular rotation among all possible transformations capable of aligning the normals, thereby avoiding unnecessary rotations and preserving the geometric integrity of the depth point cloud, ensuring its broad applicability and reliability.

3. **3D Bounding Box Quality Control**
   - SpatialVLM does not have quality control for 3D Bbox when generating 3D data. Due to the accumulation of errors from point cloud lifting, camera intrinsic and extrinsic parameters, 2D bounding boxes, and 3D clustering, quality control is crucial for high-quality box labeling. ImOV3D, focusing on 3D detection, introduces more quality control strategies, such as a 3D bounding box filtering module. By using the median size of objects generated by GPT-4, pseudo 3D bounding boxes are filtered to ensure the accuracy and reliability of the data. This strategy can effectively reduce the impact of noise data and improve the quality of the final detection results.

4. **Bounding Box Generation Method**
   - During the process of generating 3D bounding boxes, we have chosen a more universal approach, employing bounding boxes instead of segmentation. Particularly near the edges, segmentation can still be affected by outlier points, which does not resolve the issue but rather imposes a higher demand for 2D annotations. We utilize the 2D bounding boxes provided in the 2D dataset to construct the frustum, ensuring that the generation of 3D bounding boxes is both accurate and efficient.

Thank you for your professional and serious reply. If you have any other questions, we are happy to communicate with you!
Thank you for the response, especially the detailed description of the architecture (and the architecture diagram). I am still supportive of accepting this paper.
I thank the authors for addressing some of my concerns and suggest that these details be included in the final paper.

As a reminder, according to the NeurIPS 2024 policy on “Contemporaneous Work”: “For the purpose of the reviewing process, papers that appeared online within two months of a submission will generally be considered ‘contemporaneous’ in the sense that the submission will not be rejected on the basis of the comparison to contemporaneous work.” SpatialVLM (CVPR 2024), which appeared in January 2024, is not considered concurrent work. Therefore, I would like the authors to further clarify how SpatialVLM differs “significantly.”