ImOV3D: Learning Open Vocabulary Point  Clouds 3D Object Detection from Only 2D Images
Authors: Timing Yang
Yuanliang Ju
Li Yi
Venue: NeurIPS 2024 poster (2024)
Keywords: Open Vocabulary 3D Detection · Pseudo Multimodal Learning
Primary area: machine_vision

Abstract: Open-vocabulary 3D object detection (OV-3Det) aims to generalize beyond the limited number of base categories labeled during the training phase. The biggest bottleneck is the scarcity of annotated 3D data, whereas 2D image datasets are abundant and richly annotated. Consequently, it is intuitive to leverage the wealth of annotations in 2D images to alleviate the inherent data scarcity in OV-3Det. In this paper, we push the task setup to its limits by exploring the potential of using solely 2D images to learn OV-3Det. The major challenges for this setup is the modality gap between training images and testing point clouds, which prevents effective integration of 2D knowledge into OV-3Det. To address this challenge, we propose a novel framework ImOV3D to leverage pseudo multimodal representation containing both images and point clouds (PC) to close the modality gap. The key of ImOV3D lies in flexible modality conversion where 2D images can be lifted into 3D using monocular depth estimation and can also be derived from 3D scenes through rendering. This allows unifying both training images and testing point clouds into a common image-PC representation, encompassing a wealth of 2D semantic information and also incorporating the depth and structural characteristics of 3D spatial data. We carefully conduct such conversion to minimize the domain gap between training and test cases. Extensive experiments on two benchmark datasets, SUNRGBD and ScanNet, show that ImOV3D significantly outperforms existing methods, even in the absence of ground truth 3D training data. With the inclusion of a minimal amount of real 3D data for fine-tuning, the performance also significantly surpasses previous state-of-the-art. Codes and pre-trained models are released on the  https://github.com/yangtiming/ImOV3D.

ImOV3D: Learning Open-Vocabulary Point Clouds 3D Object Detection from Only 2D Images
Timing Yangl.2 Yuanliang Ju Li Y12.3.1f
Shanghai Qi Zhi Institute, 2 IIIS, Tsinghua University, Shanghai Al Lab
Abstract
Open-vocabulary 3D object detection (OV-3Det) aims to generalize beyond the limited number of base categories labeled during the training phase. The biggest bottleneck is the scarcity of annotated 3D data, whereas 2D image datasets are abundant and richly annotated. Consequently, is intuitive to leverage the wealth of annotations in 2D images to alleviate the inherent data scarcity in OV-3Det. In this paper, we push the task setup its limits by exploring the potential of using solely 2D images to learn Oe- The major challenges for this setup is the modality gap between training images and testing point clouds, which prevents effective integration of 2D knowledge into OV-3Det. To address this challenge, we propose novel framework ImOV3D to leverage pseudo multimodal representation containing both images and point clouds (PC) to close the modality gap. The key of ImOV3D lies flexible modality conversion where 2D) images can be lifted into 3D using monocular depth estimation and can also be derived from 3D scenes through rendering. This allows unifying both training images and testing point clouds into common image-PC representation, encompassing wealth of 2D semantic information and also incorporating the depth and structural characteristics of 3D spatial data. We carefully conduct such conversion minimize the domain gap between training and test cases. Extensive experiments on two benchmark datasets, SUNRGBD and ScanNet, show that ImOV3D significantly outperforms existing methods, even in the absence of ground truth 3D training data. With the inclusion of minimal amount of real 3D data for fine-tuning, the performance also significantly surpasses previous state-of-the-art, Codes and pre-trained models are released the attts://githuu com/
Im0V3D
Introduction
In the 3D vision community, there is notable surge in interest surrounding open-vocabulary 3D object detection (OV-3Det). This task focuses on the detection of objects from unbounded categories that were not present during the training phase, using 3D point clouds as input. Such capability holds immense significance dynamic 3D environments where wide range of object categories constantly emerge and evolve, which is critical in downstream applications including robotics 8
[8 28 23
autonomout autonomous driving 31 46 and augmented reality 355 44
With the advancements in OV-3Det, which is not only scarce in terms of labels but also in the data itself. However, the collection and annotation of 3D point clouds scenes pose significant challenges. The availability of accessible and scannable scenes (e.g. indoor scenes) may be limited. Additionally, obtaining 3D annotations often requires substantial human effort and time-consuming. These limitations impact the model's performance in handling novel objects. Existing methods
30
29 28
seck help from powerful open-vocabulary 2D detectors. A common method
'Equal contribution.
1 "orresponniig author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

BCB Jengo
Training Label
Please help me Find bed, lamp and cresser
RGB Images
Plasse help tamp
Training
Multimodal Detector
Paint Cloud Detector
Shared Weights
Shared Weights
Inference
Paler Claik
Multimodal
Point Cloud
Detector
Detector
Previous Methods
ImOV3D
Figure 1: Left: Traditional methods require paired RGB-D data for training and use single-mddality point clouds as input during inference. Right: ImOV3D involves using vast amount of 2D images to generate pseudo point clouds during the training phase, which are then rendered back into images. In the inference phase, with only point clouds as input, we still construct a representation to enhance detection performance.
leverages paired RGB-D data together with 2D detectors to generate 3D pseudo labels to address the label scarcity issue, as shown in Figure left. But they are still restricted by the small scale of existing paired RGB-D data. Moreover, the from scratch trained 3D detector can hardly inherit from powerful vocabblccyy 2D detector models directly due to the modality difference. We then ask the question, what is the best way to transfer 2D knowledge to 3D for OV-3Det?
Observing that the modality gap prevents a direct knowledge transfer, we propose to leverage a pseudo multi-modal representation close the gap. On one hand, we can lift 2D image into a pseudo-3D representation through estimating the depth and camera matrix. On the other hand, we can convert 3D point cloud into a pscudo-2D representation through rendering. The pseudo RGB image-PC multimodal representation could serve as a common ground for better transferring knowledge from 2D 3D.
In this paper, we present ImOV3D, which addresses these challenges by employing pseudo- multimodal representation training and the inference phase, we construct pseudo-multimodal representation to achieve our goal of training solely with 21) images and better integrating multimodal features enhance the performance of OV-3Det. Our key idea lies in proper modality conversion. Specifically, the entire pipeline consists of two flows: (1) Image - Pseudo PC, by leveraging large-scale 2D images training set, our method begins by converting images to pseudo point clouds through monocular depth estimation and approximate camera parameter. We automatically generate pseudo 3D labels based on 2D annotations, providing the necessary training data. We also designed a set of revision modules, which significantly improve the quality of the pseudo 3D data through the use of GPT-4 M's size prior and the orientation of the estimated normal map. (2) Pseudo PC - Pseudo Image, we learn a point clouds renderer capable of producing natural-looking textured 2D images from pseudo 3D point clouds. This enables ImOV3D to leverage inputs during inference, transferring 2D rich semantic information and proposals into the 3D space, further enhancing the detector's performance.
unified framework. As shown in Figure T right side, In both the
3D detection even for point cloud-only
Despite being trained solely with the 2D image set, ImOV3D exhibits impressive detection results when directly processing real 3D scans. This is attributed to the high fidelity of the lifted point clouds and the point clouds rendering. Additionally, when a small amount of real 3D data becomes available, even without any 3D annotations, ImOV3D can further narrow the gap between pseudo and real data by fine-tuning on such 3D data, leading improved detection performance. To validate the effectiveness of ImV33,, we perform extensive experiments on two benchmark datasets: SUNRGBD 43 and ScanNet [2]. Notably, in scenarios where real 3D) training data unavailable, ImOV3D open-vocabulay, 3D detectors by an mAP@0.25 improvement
surpasses previous state-of-the-arn of at least 7.14% on SUNRGBD and 6.78% on SeanNet. Furthermore, when real 3D training data is accessible, ImOV3D continues to outperform various challenging baselines by large margin. Thorough ablations are also conducted to validate the efficacy of our designs. In summary, our contributions are three-fold:
We propose ImOV3D, the first OV-3Det method that can be trained solely with 2D images without requiring any 3D point clouds or 3D annotations.

---

We introduce novel mseudd-multinodal representation pipeline which converts 2D inter- net images and corresponding detections into pseudo point clouds, pseudo 3D annotations, and point clouds renderings to support point clouds-based multimodal OV-3Det.
ImOV3D achieves state-of-the-art performance on two general OV-3Det benchmark datasets across various settings, showcasing its ability to enhance open world 3D understanding despite the lack of 3D data and annotations.
2 Related Work
Open- Vocabulary 2D Object Detection encompasses two primary series of works: The first [50
19
32
42
3
15
37
45
11
9 50), which draws upon knowledge from pre-trained Vision-Language comprehends the relationships between images and their corresponding
models (e.g., CLIP
40
textual descriptions, thereby enhancing object recognition and classification. The second series
25 58 involves the use of extensive training data, specifically text/image pairs, enabling the model to learn a more diverse set of object representations. Detic (58) leverages vocabularies from image classification datasets train the classification head of object detectors, addressing the issue of insufficient training data and enabling inference on a larger vocabulary set. In the 2D component of our pseudo- multimodal detector, we utilize Detic [58] to predict 2D labels and bounding boxes. These 2D visual information features are then converted and augmented for the 3D point cloud detector, significantly enhancing our model's ability recognize broader range of objects.
26
47 48
57 33
Open- Voca-Vocayylary Scene Understanding has recently gained increased attention 36
22
22 [0] and plays a critical role in robotics, autonomous driving, etc. OpenScene [36] achieves open-world scene understanding without the need for labeled data by densely embedding 3D scene points together with text and image pixels into the CLIP 4О feature space. PLA [ develops a hierarchical approach pairing 3D data with text for open-world 3D learning. We focus on OV,3Det, where merely extracting CLIP [40 features is insufficient. We also require the intricate spatial structure of point clouds enhance detection accuracy and robustness. By integrating both CLIP 40 's visual knowledge and the detailed geometric information from point clouds, our approach aims to enable the recognition of broader range of objects beyond the predefined categories.
Open- Ogen-Vocabulary 3D Object Detection in 3D) vision is still in its early stages, especially when compared to traditional 3D) object detection (3
39
34
27 6). OV-3DETIC (29 leverages Ima-
geNetlK 13 to expand the detector's vocabulary set and conducts contrastive learning between
images and point clouds modalities for more effective knowledge transfer. OV-3DET (30) generates
pseudo 3D annotations for localization using a pre-trained 2D open-vocabulary detector (58) CoDA
(5] leverages 2D and 3D prior information and a cross-modal alignment module to simultaneously
learn the localization and classification capabilities. CoDAv2 ( improves CoDA [5] further by
proposing the novel object enrichment strategy and 2D box guidance. FM-OV3D [52) combines
multiple foundation models without the need for 3D annotations. However, they are still subject to the
influence of the volume of 3D data and still require strict correspondence between RGB-D data. Our method can generate training data for OV-3Det task using only 2D images, without any 3D) ground truth data. It can directly achieve state-of-the-ar performance when tested on the evaluation set. The designed pseudo multioodl representation pipeline provides novel solution for the utilization of both 2D and 3D information.
3 Method
3.1
Overview
An overview of the proposed open world 3D Object Detection model, ImOV3D, is shown in Figure 2. ImOV3D is point cloud-only model that addresses the challenges of the scarcity of annotated
3D datasets in open- vocabularyyr 3D Object Detection. To overcome this challenge, ImOV3D uses large-scale 2D datasets to generate pseudo 3D point clouds and annotations. We use a monocular depth estimation model to create metric depth images, which are then converted into pseudo 3D point clouds for both indoor and outdoor scenes. To generate pseudo 3D annotations, we lift 2D bounding boxes into 3D space. To leverage multimodal data, we transform the point clouds into pseudo images using point cloud renderer. Our training strategy involves two-stage approach. Firstly, we conduct
3

---

Image
Pseudo Point Clouds
Pseudo Point Clouds
Pseudo Image
Point Cloud Lifting Module
Point Cloud
Renderer
Pseudo 3D Annotation Generator
RGB Image
Pseudo Point Glouds
Pseudo Image
Pseudo Point Gouds
3D Backbona
Text Prompts:
Inputt
CLIP
Photo
(Pillow, Lamp, Bed ...)
2D OveR Detecton
Fusion
Pseudo-Multimode Detector
Inference Phase
Our model takes 2D images as input and puts them into the Pseudo 3D Annotation Generator produce pseudo annotations. These 2D images are also fed into the Point Cloud Lifting Module to generate pseudo point clouds. Subsequently, using the Point Cloud Renderer, these pseudo point clouds are rendered into pseudo images, which then get processed by 2D open vocabulary detector to detect 2D proposals and transfer the 2D semantic information to 3D space. Armed with pseudo point clouds, annotations, and pseudo images data, proceed to train multimodal 3D detector.
Figure 2: Overview of ImOV3D:
pre-training using pseudo 3D point clouds and corresponding annotations. Subsequently, we initiate an adaptation stage aimed at minimizing the domain discrepancy between 2D and 3D datasets.
3.2 Point Cloud Lifting Module
The success of open-vocabulary object detection relies heavily on the availability of large-scale labeled datasets. However, the scarcity of comparable 3D datasets poses challenge for open world 3D Object Detection. To address this, we bridge 2D images Izd E R xxxxxxxx x3 (where M is the number of images, and H and W are the height and width, respectively) 3D detection by generating pseudo 3D point clouds P Peeado E R. M x X33 (where N is the number of points, each with coordinates (x, y, 2)).
Utilizing 2D datasets for 3D detection presents difficulties due to the absence of metric depth images and camera parameters. To overcome these obstacles, to obtain single-view depth images Deetric E xBMxHi Additionally, we employ fixed camera intrinsies K Rax3 with the focal length f calculated based on 55-degree field of view (FOV) and the image dimensions.
metric depth estimation model
However, the absence of camera extrinsics E {R t} (where R is the rotation matrix and t is the translation vector set to [0, O, 0, results in the arbitrary orientation of point clouds. To correct this, we use a rotation correction module to ensure the ground plane is horizontal, as shown Figure (a). First, estimate the surface normal vector at each pixel using normal estimation model [2). creating normal map. From this, we selectively extract the horizontal normal vector N; at each pixel, defined as (Nay N N.). We then compute the normal vector of the horizon surface as Nored Cluster (N,). To align Nped with the Zauis- we calculate the rotation matrix using the following equation:
Npred Zazis
R 12
(I)
||
where is the identity matrix, v is the cross product of Npred and Zazis, expressed as Npred X Zawier K is the skew symmetric matrix constructed from the vector represented as:
U2
Uy
K
02
0
Ux
0y
Cg

---

Normal map
Point cloud
average lanpth, widhh and height of tha festegery).
Nyssi
Normal Estimator
The standord the for thir Juttegery
Z
GPM4
Nyuad
RGB Image
Horionaal surface mask
Extracted point cloud
(b)
Figure 3: Illustration of 3D Data Revision Module: (a) The rotation correction module involves processing an RGB image through a Normal Estimator to generate a normal map. This map then helps extract horizontal surface mask for identifying horizontal point clouds, from which normal vectors Npred are obtained. These vectors are aligned with the Z-axis to compute the rotation matrix R. (b) In the 3D box filtering module, prompts related to object dimensions GPT-4 to determine the mean size for each category. This mean size is then used to filter out boxes that do not meet the threshold criteria.
first provided to
After obtaining the camera intrinsics matrix K and the camera extrinsics matrix E through the previous steps, depth images Dimetic converted into point clouds P pscudo-
3.3 Pseudo 3D Annotation Generator
Building upon the vast collection pseudo 3D point clouds P Pssuddo acquired from 2D datasets, our step is to generate pseudo 3D bounding boxes BaDpscudo X K is the number of bounding boxes and each box has 7 parameters: center coordinates, dimensions, and orientation).
2D datasets contain rich segmentation information that be used to generate 3D boxes by lifting. Using the camera intrinsics matrix and camera extrinsics matrix E obtained through the Point Cloud Lifting Module, lift the 2D bounding boxes B2Dgt R M xxxx from the 2D datasets into 3D space by extracting 3D points that fall within the predicted 2D boxes, generating frustum 3D boxes BaDpeendo- The extracted point clouds may contain background points and outliers. To remove these, we employ a clustering (16) algorithm to analyze point clouds. Through the clustering results, can identify and remove background points and outliers that do not belong the target objects.
However, these lifted 3D boxes may still contain noise from the depth images Dmetric obtained by monocular depth estimation. To address this issuc, First, we construct database of median object sizes
3D box filtering module to filter out
inaccurate 3D boxes, as shown in F using GPT-4 [1] By prompting GPT-4 with "Please tell me the average length, width, and height of this [category], using meters as the unit", , we obtain the median dimensions LGPT, WGrr, HGPT- Each object in a scene, defined by dimensions L, W. H, is compared to these median dimensions using a threshold T. An object is preserved if each element of:
R RIPT
T
VR E (L,W, H)
T
The 3D box filtering module consists of two components: Train Phase Prior Size Filtering and Inference Phase Semantic Size Filtering. The first component filters out boxes that do not match the size criteria before training. The second component removes semantically similar but size-different categories during inference, preventing errors such as misidentifying book as a bookease.
3.4 Point Cloud Renderer
Point cloud data has inherent limitations, such as the inability of sparse point clouds to capture detailed textures. 2D images can enrich 3D data by providing additional texture information that point clouds lack. To utilize 2D images, we transform point clouds P into rendered images Iyendered E R M H
Integrating rendered images into 3D detection pipeline is challenging. A naive approach, as mentioned in PointClip [59], is to append raw depth values across the RGB channels, but this fails to apply mature open- world 2D detector effectively. To leverage multimodal information without
5

---

additional inputs beyond 3D point clouds, we develop a point cloud renderer to convert point clouds into detailed pseudo images. This process can also be learned solely from 2D image datasets.
The point cloud renderer has two key modules:
The point cloud rendering module converts point clouds P into rendered images Trendered: and the color rendering module then processes these images to produce colorized outputs using ControlNet (55). ControlNet [55] is a method designed to control diffusion models, transforming rendered images Trendered into pseudo images Ipacudo E R RMx W
In the pretraining stage, we use the camera intrinsics K and extrinsics E from the Point Cloud Lifting Module to render Procudo into rendered images Iredeerd. During adaptation and inference, ground truth point clouds P&t into images using the intrinsics K obtained in the same way. Due to the lack of extrinsics E, the final rendered images Iendered are obtained by finding the optimal angle from different horizontal and vertical perspectives to make the images most compact.
render
In reality, we cannot project point cloud while guaranteeing that every pixel corresponds to some points. There will be holes and missing areas due to point cloud imperfections or incompatible viewpoint selection. We adjust the camera's 's position horizontally and vertically to observe point clouds from various angles, removing obscured portions, Finally, we render the point clouds back into images from their original perspective, resulting in partial view rendered images Ipartial E R RMxx x3 The angle range for adjustments is set from -75 to 75 degrees, with a de-eree interval:
On 0, E {-75 + 15k A = 0, 1, 2., 103
where A is an integer indicating the stepwise adjustment of the camera's angle.
After generating partial view rendered images Ipanial, the next step is to fine-tune ControlNet [55] using these images to obtain pseudo images Ipseudo- Three types of data are prepared for fine-tuning: prompts, targets, and sources. RGB images Izu from a 2D dataset serve as the targets, while the partial view rendered images Ipaiial are the training sources. Prompts are not used during training.
Finally, we use the pseudo images Ipscudo and annotations Bangt from 2D datasets to fine-tune an open-vocabulary 2D detector. Thus, we can use Ipscddo to obtain corresponding pseudo 2D) bounding boxes ByDppseddo BM
3.5 Pseudo Multimodal 3D Object Detector
With an extensive dataset comprising abundant 3D data (P + BiDpseudo) and pseudo images data Ipscndo next step is to train pseudo multimodal 3D detector using a two-stage approach.
Training Strategy Our training process includes pretraining and adaptation stages. In the pretraining stage, we train on pscudo 3D point clouds P paeudo and annotations BaDpeulos combined with pseudo images Ipweudk-- While the pre-trained model performs well for zero-shot detection, a significant domain gap exists between 2D and 3D datasets.
In the adaptation stage, to minimize the domain gap, we follow the same approach as OV-3DET. First, a pre-trained open-vocabulary 2D detector is used to detect objects in the image. Then, these 2D boxes BaDyseudo E M x xx1 along with RGBD data, are lifted into 3D space. Through clustering to remove background and outlier points, we obtain precise and compact 3D boxes BiDpscudo- Finally, this processed data used for adaptation. To further explore the benefits of pretrain, we use 3D datasets of varying sizes to test the model's 's performance under different data availability conditions.
Loss Function In this section, we describe loss function used in the pretrain stage. By leveraging Prsccdo and Ba.Dpeudo a 3D backbone is trained to obtain seed points IC E IR Kx where K represents the number of seeds, along with 3D feature representations Fpc E (x+3++ with F denoting the feature dimension. Then, seed points are projected back into 2D space via the camera matrix. These seeds that fall within the 2D bounding boxes B2D retrieve the corresponding 2D cues associated with these boxes and bring them back into 3D space. These lifted 2D cues features are represented as Fing E R (3++F', where F' represents the feature dimension. Finally, the point cloud features Fpc and image features Fimg are concatenated, forming the joint representation Fjoint E Kx (3+ (3+F+F"). In the adaptation stage, P Peeuuo is replaced with Pat. keeping the workflow consistent with the pretrain stage.
Gooall = Lioc + Σ W x CrossEntropy(Cls--uan (Cls-header(F:)) Fiext)
(5)
where represents different features, such as pc, img. joint. W; is the weight corresponding to feature Lixe represents the original localization loss function used Im VoteNet(39) Fiest denotes the feature extracted by the text encoder in CLIP.

---

Implementation Details Our model is point cloud-only VoVeNeee 39 +Clip architecture. The monocular depth estimation model used is [.. jointly trained on both indoor and outdoor scenes. In the pre-training phase, similar to ImVoteNet, we train for 180 epochs with an initial learning rate of 0.001. In the adaptation phase, we train for 100 epochs, reducing the learning rate to 0.0005.
For 2D voting, there are three types of cues: Geometric cues, Texture cues, and Semantic Cues. Unlike Im VoteNet, we retain geometric cues but remove texture cues. For Semantic cues, instead of using a one-hot class vector, we use pre-trained CLIP text encoder features, which is more suitable for open-vocabulary setting.
4 Experiments
In this section, compare proposed ImOV3D with other baseline models. Our experimental setup is divided into two main stages: Pretraining and Adaptation. During the pretraining stage, the training data is pseudo 3D data, referring to pseudo 3D point clouds and their corresponding annotations (3D boxes). During the adaptation stage, we use ground truth point clouds and pseudo conducted on two commonly used Object Detection datasets: SUNRGBD [43] and ScanNet [2]. Additionally, we carry out comprehensive ablation studies to validate the effectiveness of our model's components and the data generation pipeline.
labels to minimize the domain gap. All experiments
4.1 Experimental Setup
2D Images Dataset: We select the LVIS (0 dataset our 2D image souree for generating pscudo
3D data, utilizing 42,000 images provided its training set, which spans 1,203 categories with rich and detailed annotations.
3D Point Clouds Dataset: We select SUNRGED
our 3D point clouds datasets for adaptation and testing, SUNRGBD 43 and ScanNet [I2] encompass diverse range of indoor environments and offer comprehensive annotations, including 2D and 3D bounding boxes for objects. We test on 20 common categories in both datasets.
143 and ScanNet
Evaluation Metries: We employ mean Average Precision (mAP) an IoU threshold of 0.25 as our primary evaluation metric. This metric effectively balances precision and recall in assessing how well our models perform on selected datasets.
Table I: Results from the Pretraining stage comparison experiments on SUNRGBD and ScanNet, ImOV3D only require point clouds input.
SUNRGED mAP@0.
ScanNet mAP@0.25
Stage
Data Type
Method
Input
Training Strategy
NV-
Point Cloud
One-Stuge
5.18
5.86
Pre- training
Pseudo Data
OVE3DETR
4
Point Cloud
One-Stage
5.24
5.30
OV-3DET
30
Point Cloud + Image
Two-Stage
5.47
5.69
Ours
Point Cloud
One-Stage
12.61 7.14
12.64 6.78
Table 2: Results from the Adaptation stage comparison experiments on SUNRGBD and ScanNet
SUNRGED mAP@0.25
ScanNet mAP@0.25
Stage
Method
Input
Training Strategy
Adap- tation
OV-3DET 30
Point Cloud Image
Two-Stage
20.46
18.02
CoDA 5
Point Cloud
One-Stage
19.32
Ours
Point Cloud
One-Stage
22.531 2.07
21.451 2.13
4.2 Main Results
Pretraining: Due to the absence of existing baseline methods except OV-3DET (30], we utilize
CLIP [40 to make previous high-performance 3D detectors such as 3DETR 34 and VoteNet compatible with OV3Det. Specifically, to adapt traditional point cloud detectors for Open Vocabulary detection, we first extract geometric features from point clouds. Then, we integrate
38
7

---

for classification by converting these features for compatibility with CLIP 40 visual encoder and creating textual prompts for zero-shot classification. Finally, we compare the encoded prompts with the visual features to classify objects beyond the predefined categories. Therefore, these
CLIP 440
baselines are denoted as Ooteoteee [8 OV-3DETR
34
To ensure a fair comparison with the current SOTA OV3Det methods, during the adaptation stage, all baselines use OV-3DET [30]'s approach to generating pseudo labels for ground truth point cloud data, which serve as training data for adaptation. In this stage, comparisons are made with CoDA 5 and OV-3DET [30]
Adaptation:
4.2.1 Pretraining 3D Training Data Free OV-3Det
As shown in Table [ training solely with pseudo 3D data generated by our method, ImOV3D improves mAp @0.2 by 7.14% on SUNRGBD and 6.78% ScanNet over the best baseline. This achievement, made without using any 3D ground truth annotated data, demonstrates the high quality of our generated data and the effectiveness of using extensive 2D datasets to enhance Open World perception. Unlike OV- VoteNet, which lacks 2D image integration, our method's mAP@0.25 outperforms OV-VoteNe by 7.43% and 6.78% on the two datasets, proving the effectiveness of our multimodal approach even with only point cloud inputs. OV-3DET and ImOV3D visualization results are shown in Figure bb).
4.2.2 Adaptation - 3D Training Data Guided OV-3Det
Table 2 shows original OV-3DET results in the first row. CoDA only compares with OV-3DET on SeanNet. Our experiments indicate that after pretraining with pseudo 3D data, ImOV3D outperforms the best baseline by 2.07% SUNRGBD and 2. 133% on ScanNet in mAP@0.25 This highlights the crucial role of pseudo 3D data in training and its effectiveness as data augmentation.
5
Ablation Study
5.1
Ablation Study of 3D Data Revision
To validate the effectiveness of enhancing pseudo 3D data quality, we conducted ablation experiments with the Rotation Correction Module and 3D Box Filtering Module, The 3D Box Filtering Module includes Train Phase Prior Size Filtering and Inference Phase Semantic Size Filtering. Table 3 the results: the baseline without any modules, adding Train Phase Prior Size Filtering improves mAP@0.25 by 1.65% on SUNRGBD and 1. 1.27% on ScanNet. Adding the Rotation Correction ScanNet. Combining both modules results in 2.98% improvement on SUNRGBD and 3.3 % on ScanNet. Adding Semantic Size Filtering during inference further increases mAP@0.2. 0.25 by 4.26% on SUNRGBD and 4.3 % on ScanNet. These results highlight the effectivvness of each module in improving data quality and OV3Det accuracy.
shows
Module improves by 1.3% on SUNRGBD and 1.96%
Table 3: Results from the ablation study the Rotation Correction Module and the 3D Box Filtering Module, conducted on SUNRGBD and ScanNet, are presented. The 3D Box Filtering Module is divided into two components: Train Phase Prior Size Filtering and Inference Phase Semantic Size Filtering.
Train Phase
Rotation Correction
Inference Phase Semantic Size
SUNRGBD mAP@0..5
ScanNet mAP@0... 0.25
Stage
Prior Size
8.35
8.33
Pre-
XX
10.00
9.60
training
9.65
10.29
11.33
11.64
12.61
12.64
We also discuss the efficiency of GPT-4 ① in the 3D box filtering module using the SUNRGBD dataset [43] For comparison, we select the top 10 classes with the most instances the validation
LxW x set. The volume ratio for these 10 classes is defined as Ratioy Leaar W Wanger' Hanarr This ratio
is an insightful metric for comparing the performance of the GPT-4 powered 3D box filter module

---

to the ground truth (GT), A ratio close to indicates high precision. We calculate Ratioy for each instance and use Kernel Density Estimation (KDE) analyze and plot the distributions of the volume ratios. Results are presented in Figure 6 a).
5.2 Ablation Study of Depth and Pseudo Images
To validate the effectiveness of pseudo images generated by ControlNet [55]. compare 2D depth maps from pseudo point clouds with pscudo images, shown in Figure SUNRGBD dataset, mAP@0. 0.25 increased from 4.38% 12.61% %, and on the ScanNet dataset, rose from 4.47% to 12.64% (see Table 4). This shows that rich texture information in 2D im- ages significantly enhances 3D detection perfor- mance.
Table 4: The results from different types of 2D rendering images include depth maps and pseudo images.
On the
Rendered Images
SUNRGBD
ScanNet
Stage
Data Types
mAP@0.25
mAP@ 0.25
Pre- training
Depth Map
Pseudo Images
12.61
12.64
Figure 4: Qualitative results include (a) 2D RGB images, (b) 2D depth maps with 2D OVDetector annotations, and (c) pseudo images with annotations from fine-tuned 2D detector.
5.3 Ablation Study of Data Volume
Our method fine-tunes with limited real ground truth 3D point cloud data and pscudo 3D annotations. Using OV-3DET's code, we train with varying data volumes. With 10% adaptation data, OV-3DET's mAP@0.25 on SUNRGBD drops from 20.46% to 55.24%, while ours drops from 22.53% to 19.24%. ScanNet, OV-3DET's mAP@0.25 falls from 18.02% to 14.35%, and ours falls from 21.45% to 18.45% (Figure 5 (b)). We observed decrease in performance compared to using the full data set; however, our method was still able to maintain relatively high detection accuracy. This confirmed the robustness of our method and its adaptability small datasets, enabling effective 3D Object Detection even under constrained data conditions. also underscores the importance of developing models for OV3 that are capable of learning from limited data and generalizing to broader range of scenarios.
SUNRGBD
Scanfiet
ScanWet
SUNRGBO
53
24
0V13
70
21:43
26-5
10.40
20
15-3
13.30
2603
53
107%
10%
100%
33%
63
5UMPOBD Sooffl
ScarNo: SUNAGEO
Figure 5: (a) and (b) show data volume ablation results. (c) illustrates transferability ablation results.
5.4 Analysis of Transferability
Traditional 3D detectors struggle with transferability due training and testing class differences. We test ImOV3D on ScanNet and SUN RGB-D on the opposite datasets. The results, shown in Figure

---

((c), demonstrate that our model outperforms OV-3DET by 7.1% on SUN RGB-D and 7.82% on ScanNet. ImOV3D demonstrates superior transferability across domains despite the domain gap.
5.5 Analysis of Fine-tuned 2D Detector
Table 5: Comparison of fine-tuned 2D detector: Off-the-Shell vs. Fine- Tuned Detic.
SUNRGBD MAAP@0.25
ScanNet mAP@0.25
Pretraining
Adaptation
2D Off-the-shelf + 3D Adaptation
18.8
18.96
Off-the-shelf + 3D Pretraining
2D Off-the-shelff + 3D Adaptation
19.67
19.25
2D Pretraining + 3D Pretraining
2D Adaptation + 3D Adaptation
22.53
21.45
To validate the benefits of fine-tuning Detic with pseudo images, we compare the off-the-shelf Detic to the fine-tuned version. The fine-tuned Detic shows clear advantages in handling pseudo images. On the SUNRGBD dataset, the mAP@0.2 0.25 increases from 19.67% to 22.53%, and on the ScanNet dataset, it rises from 19.25% to 21.45% (see Table These experiments were conducted under the adaptation setting, illustrating the model's ability to learn from and improve detection capabilities with not entirely real data. This not only confirms the efficacy of textured image but also highlights the importance of fine-tuning models to enhance their adaptability and accuracy.
Ground Truth Point Clouds
OV-3DET
ImDV3D
BEAL SE KALLE
(8)
(b)
Figure 6: (a) KDE plots of volume ratios (Ratioy) for top 10 classes in SUNRGBD validation set. (b) Visualization comparison of OV-3DET with ours in SUNRGBD.
Conclusion and Limitation
In conclusion, this paper introduce ImOV3D, a novel framework that tackles the scarcity of annotated 3D data in OV-3Det by harnessing the extensive availability of 2D images. The framework's 's key innovation lies in its flexible modality conversion, which integrates 2D annotations into 3D space, thereby minimizing the domain gap between training and testing data. Empirical results on two common datasets confirm ImOV3D's 's superiority over existing methods, even without ground truth 3D training data, and its significant performance boost with the addition of minimal real 3D data for fine-tuning. Our method's 's success showcases the potential of leveraging 2D) images for enhancing 3D object detection, opening new avenues for future research in pseudo muuudoaa data generation and its application in 3D detection methodologies.
intitation:Athhuuse our method has demonstrated the potential of 2D images in OV-3Det tasks, especially with the proposed pseudo multimodal representation, we need dense point clouds here to ensure that the rendered images help improve performance. In the future, will explore more generalized strategies.
Acknowledge
We would like to express our gratitude to Yuanchen Ju, Wenhao Chai, Macheng Shen, and Yang Cao for their insightful discussions and contributions.
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Igge Akkaya, Florencia Lconi Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 2023.
[2] Gwangbin Bae, Ignas Budvytis, and Roberto Cipolla. Estimating and exploiting the aleatoric uncertainty in surface normal estimation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 13137-13146, 2021.
[3] Hanoona Bangalath, Muhammad Maaz, Muhammad Uzair Khattak, Salman H Khan, and Fahad Shah- baz Khan. Bridging the gap between object and image-level representations for open- apen-vocabulary detection. Advances in Neural Information Processing Systems, 55:33781-33794 2022.
[4] Shariq Faroog Bhat, Reiner BirkI, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302. 12288, 2023.
[5] Yang Cao, Yihan Zeng, Hang Xu, and Dan Xu. Coda: Collaborative novel box discovery and cross-modal alignment for open-vocabulary 3d object detection. arXiv preprint arXiv:2310.02960, 2023.
[6] Yang Cao, Yuanliang Jv. and Dan Xu. 3dgs-det: Empower 3d gaussian splatting with boundary guidance and box-focused sampling for 3d object detection. arXiv preprint arXiv:2410.01647. 2024.
[7] Yang Cao, Yihan Zeng, Hang Xu, and Dan Xu. Collaborative novel object discovery and box-guided cross-modal alignment for voca-lacy 3d object detection. arXiv preprint arXiv:2406.008.. 008300006000 2024.
[8] Boyuan Chen, Fei Xia, Brian lehter, Kanishka Rao, Keerthana Gopalakrishnan, Michael S Ryoo, Austin Stone, and Daniel Kappler. open-vocabulary queryable scene representations for real world planning. In 2023 IEEE International Conference Robotics and Automation (ICRA), pp. 11509- 11522, IEEE, 2023.
[9] Keyan Chen, Xiaolong Jiang. Yao Hu, Xu Tang, Yan Gao, Jianqi Chen, and Weidi Xie. Ovamet: Towards open- vocabulayy object attribute recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 23518-23527, 2023.
[10] Runnan Chen, Youquan Liu, Lingdong Kong, Nenglun Chen, Xinge Zhu, Yuexin Ma, Tongliang Liu, and Wenping Wang. Towards fabel-free scene understanding by vision foundation models. Advances in Neural Information Processing Systems, 36, 2024.
[11] Han-Cheol Cho, Won Young Jhoo, Wooyoung Kang, and Byungseok Roh. Open-vocabulary object detection using pseudo caption labels. arXiv preprint 2r3: 13040, 2023.
[12] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: aechly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5828-5888000 08888,99.1118 2017.
[13] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and puttern recognition, pp. 248-255. Ieee. 2009.
[14] Runyu Ding, Jihan Yang, Chuhui Xue, Wenqing Zhang, Song Bai, and Xiaojuan Qi. Pla: Language-driven open-vocabulary 3d scene understanding. In Proceedings of the IEFF/CVF Conference on Computer Vision and Patiern Recognition. pp. 7010-7019, 2023.
[15] Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open- vocabulary object detection with vision-language model. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 14084-14093, 2022.
[16] Martin Ester, Hans- Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. A density- algorithm for discovering clusters in large spatial databases with noise. In kdd, volume 96, pp. 226 231, 1996.
[17] Mingfei Gao, Chen Xing, Juan Carlos Niebles, Junnan Li, Ran Xu, Wenhao Liu, and Caiming Xiong. Open vocabulary object detection with pseudo bounding-box labels. In European Conference on Computer Vision, pp. 266-282. Springer, 2022.
[18] Qiao Gu, Alihusein Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, al. Conceptgraphs: Open- vocabulary 3d scene graphs for perception and planning. arXiv preprint 16X1v:2309.16600. 2023.
[19] Xiuye Gu, Tsung-Yr Yi Lin, Weicheng Kuo, and Yin Cui. Open vocabulaayy object detection via vision and language knowledge distillation. arXiv preprint 2xXiv:210..13921, 2021.
11

---

[20] Agrim Gupla. Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pp. 5356-5364,
2019.
[21] Deepti Hegde, Jeya Maria Jose Valanarasu, and Vishal Patel. Clip goes 3d: Leveraging prompt tuning for language grounded 3d recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2028 2038, 2023.
[22] Krishna Murthy Jatavallabbula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Alaa Maalouf, Shuang Li, Ganesh lyer, Soroush Saryazdi, Nikhil Keetha, et al. Conceptfusion: Open-set multimodal 3d mapping. arXiv preprint arXiv:2302.0722 2023.
[23] Yuanchen Ju, Kaizhe Hu, Guowei Zhang, Gu Zhang. Mingrun Jiang, and Huazhe Xu. Robo-abe: Affordance generalization beyond categories via semantic correspondence for robot manipulation. arXiv preprint arXiv:2401.07480 2024.
[24] Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 119729-19737 2023.
[25] Dahun Kim, Anelia Angelova, and Weicheng Kuo. Detection-oriented image-text pretraining for open- vocabulary detection. arXiv preprint 1100002310.0011,, 2023.
[26] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10965-10975, 2022.
[27] Ze Liu, Zheng Zhang, Yue Cao, Han Hu, and Xin Tong. Group-free 3d object detection via transformers. 2021 I iece. In CVF International Conference Computer Vision (ICCV), pp. 2929-2938, 2021.
[28] Shiyang Lu, Haonan Chang, Eric Pu Jing, Abdeslam Boularias, and Kostas Bekris. Ovir-3d: Open- vocabulary 3d instance retrieval without training on 3d data. Conference on Robot Learning, pp. 1610-1620, PMLR, 2023.
[29] Yuheng Lu, Chenfeng Xu, Xiaobao Wei, Xiaodong Xie. Masayoshi Tomizuka, Kurt Keutzer, and Shang- hang Zhang. Open- vocabulary 3d detection via image-level class and debiased cross-mode contrastive learning. arXiv preprint arXiv:2207. 00997, 2022.
[30] Yuheng Lu, Chenfeng Xu, Xiaobao Wei, Xiaodong Xic, Masayoshi Tomizuka, Kurt Keutzer, and Shang- hang Zhang, Open-vocabulary point-cloud object detection without 3d annotation. In Proceedings of the IEEL/CVF Conference Computer Vision and Puttern Recognition, pp. 1 199, 2023.
[31] Zeyu Ma, Yang Yang. Guoqing Wang. Xing Xu, Heng Tao Shen, and Mingxing Zhang. Rethinking open- world object detection in autonomous driving scenarios. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 1279-1288, 1288, 2022.
[32] Zongyang Ma, Guan Luo, Jin Gao. Liang Li, Yuxin Chen, Shaoru Wang, Congxuan Zhang, and Weiming Hu. Open-vocabulary one-stage detection with hierarchical visual-languae knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14074-14083, 2022.
[33] M Minderer, A Gritsenko, A Stone, M Neumann, D Weissenborn, A Dosovitskiy, A Mahendran, A Amab, M Dehghani, Z Shen, et al. Simple open-vocabulary object detection with vision transformers. arxiv 2022. arXiv preprint 205.v:2005.062300
[34] Ishan Misra, Rohit Girdhar, and Armand Joulin. An end-to-end transformer model for 3d object detection. In Proceedings of the IEEÉ/CVF International Conference on Computer Vision, pp. 29.2900-222 2906-2917, 2021.
[35] Benjamin Nuernberger, Eyal Ofek, Hrvoje Benko, and Andrew D Wilson. Snaptoreality: Aligning augmented reality to the real world. In Proceedings of the 2016 CHIl Conference on Human Factors in Computing Systems, pp. 1233- 1244, 2016.
[36] Songyou Peng. Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Mare Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. In Proceedings of the IEEE/CVF Conference Computer Vision and Paternn Recognition, pp. 815 824, 2023.
12

---

[37] Chau Pham, Truong Vu, and Khoi Nguyen. Lp-ovod: Open-vocabuse object detection by linear probing. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 779-788,
2024.
[38] Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas. Deep hough voting for 3d object detection in point clouds. In proceedings of the IEEF/CVF International Conference on Computer Vision, pp.
9277-9286, 2019.
[39] Charles R Qi, Xinlei Chen, Or Litany, and Leonidas Guibas. Imvotenet: Boosting 3d object detection in point clouds with image votes. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4404-4413, 2020.
[40] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8766 PMLR, 2021.
[41] Nur Muhammad Mahi Shafiullah, Chris Paxton, Lerrel Pinto, Soumith Chintala, and Arthur Szlam. Clip-fields: Weakly supervised semantic fields for robotic memory. arXiv preprint arXiv:2210.0566 2022.
[42] Cheng Shi and Sibei Yang. Edadet: Open-vocabulary object detection using early dense alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15724-15734, 2023.
[43] Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgh-d: A rgb-d scene understanding benchmark suite. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 567-576, 2015.
[44] Daniel Wagner, Gerhard Reitmayr, Alessandro Mulloni, Tom Drummond, and Dieter Schmalstieg, Real- time detection and tracking for augmented reality on mobile phones. IEEE transactions on visualization and computer graphics, 16(3):355 368, 2009.
[45] Luting Wang. Yi Liu, Penghui Du Zihan Ding, Yue Liao, Qiaosong Qi, Biaolong Chen, and Si Liu. Object-aware distillation pyramid for open-vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1118-111966 2023.
[46] Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, and Jose M Alvarez. Omnidrive: A holistic Imm-agent framework for autonomous driving with 3d perception, reasoning and planning. arXiv preprint wxXi:2205.00533,, 2024.
[47] Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. Detelip: Dictionary-enrichne visual-concept paralleled pre- gre-training for open-world detection, Advances in Neural Information Processing Systems, 35:9125-91.38, 2022.
[48] Lewei Yao, Jianhua Han, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, and Hang Xu. Detclipv2: Scalable open-vocabulary object detection pre-training via word-region alignment. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23497-23506, 2023.
[49] Ping-Chung Yu, Cheng Sun, and Min Sun. Data efficient 3d learner via knowledge transferred from 2d model. In European Conference on Computer Vision. pp. 192.198.66 Springer, 2022.
[50] Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Open-vocabulary detr with conditional matching. In European Conference on Computer Vision, pp. 106- 122. Springer, 2022.
[51] Alireza Zareian. Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang, voca-uucyyylary object detection using captions. In Proceedings of the IEEE/CVF Conference Computer Vision and Patiern Recognition. pp. 14393-14402, 2021.
[52] Dongmei Zhang, Chang Li, Ray Zhang, Shenghao Xie, Wei Xue, Xiaodong Xie, and Shanghang Zhang- Fm-ov3d: Foundation model-based cross-modal knowledge blending for open-vocabulary 3d detection. arXiv preprint arXiv:2312.1 14465, 2023.
[53] Hu Zhang, Jianhua Xu, Tao Tang, Haiyang Sun, Xin Yu, Zi Huang, and Kaicheng Yu. Opensight: A simple open-vocabulary framework for bidar-based object detection. arXiv preprint arXiv:2312.0883 2023.
[54] Junbo Zhang, Runpei Dong, and Kaisheng Ma. Clip-fo3d: Leaming free open-world world 3d scene representa- tions from 2d dense clip. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2048 2059, 2023.
13

---

[55] Lvmin Zhang, Anyi Rao. and Maneesh Agrawala. Adding conditional control to text-lo-image diffusion models. In Proceedings the IEEE/CVF International Conference Computer Vision, pp. 3836-3847. 2023.
[56] Zhuoyang Zhang, Yuhao Dong, Yunze Lit,, and Li Yi. Complete-to- paoo-prtttallaa 4d distillation for self-supervised point cloud sequence representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17661-17670 2023.
[57] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: region-based language-ing image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2022.
[58] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. Detecting twenty- thousand classes using image-level supervision. In European Conference Computer Vision, pp. 350-368. Springer, 2022.
[59] Xiangyang Zhu, Renrui Zhang, Bowei He, Ziyao Zeng, Shanghang Zhang, and Peng Gao. Pointelip v2: Adapting clip for powerful 3d open-world learning. arXiv preprint arXiv:2211.1168. 2022.
14

---

Supplementary Materials
I. Supplementary Method Details
This part supplements the methods mentioned in the main paper with detailed implementations and theoretical underpinnings.
Partial- View Removal: To obtain point cloud from 2D image, we first use monocular depth estimation model ZoeDepth [4] to generate a depth map for the image, as shown in Fig (A). Inspired by (56), utilizing partial-view depth images for data augmentation serves to bolster the robustness and generalization capabilities of machine learning models in computer vision tasks. By simulating realistic occlusions and offering diverse array of viewpoints, this approach prepares models to handle real-world variability and complexity more effectively, improving their performance across a wide range of scenarios.
In this process, we start by defining the point cloud and the camera viewpoints. The point cloud is collection of points, which is formally represented as:
P = (pI. P2. Pn
(6)
where each point is a part of the 3D object or scene we are interested in.
We consider two distinct viewpoints, A and B, characterized by their camera parameters 0 and 0g. respectively. These parameters include the position and orientation the camera that define each viewpoint.
Initially, we observe the point cloud from viewpoint A, resulting in subset of points observed from this specific viewpoint, taking into account potential occlusions and the camera's 's field of view. This subset denoted as:
PA view view(P,0.)
Next, we change our perspective to viewpoint B and observe the point cloud again, which gives us another subset of points, denoted as:
PB view(P,0g)
(8)
The key step in this process involves identifying the overlapping points that are visible from both viewpoints and B. This overlapping set represented as:
PAB PA PB
After identifying the overlapping points, we proceed to remove these points from the subset observed from viewpoint A. The final set of points, after removing the overlapping ones, is represented as:
PA = PA - PAB
(10)
Finally, this processed point cloud is rendered back into a depth image from viewpoint A. The resulting depth image, which includes holes where the overlapping points were removed, as shown in Fig 7 (B), illustrating the effect of occlusions, is obtained through the rendering process, expressed as:
I render( Par, 9A)
(11)
Normal Estimator: In this paper, we employ Normal Estimator 2 for surface normal estimation.
The process normal estimation begins with the calculation of the surface normal vector each pixel, a crucial step for understanding the orientation of surfaces in scene. This calculation is conducted using a normal estimation model that utilizes either depth information or the 3D coordinates each point in the scene. Mathematically, the surface normal vector at point is defined as 3D vector N (Nz, Nz), where Ñ represents the direction perpendicular the surface at that pixel, as shown in the equation
N NNe, Ne).
(12)
15

---

(A)
(B)
(C)
Figure 7: Illustration of (A) Depth Images, (B) Rendered Point Cloud Images after Partial- View Removal, and (C) Ground Truth 2D Images
Following the estimation of normal vectors across the scene, the next phase involves the extraction of horizontal normal vectors N, at each pixel. This selection process focuses vectors that align with the horizontal plane of the scene, essential for applications such as ground plane detection or horizon line estimation. Once horizontal normal vectors are identified, they undergo clustering process to find a consensus on the predominant horizontal surface orientation within the scene. This process yields Npred; a vector representing the normal to the horizon surface, encapsulated by the equation
Nyred Clustering(AV)
(13)
The final step in the normal estimation process is the alignment of Npred with the 7-axis of the coordinate system to ensure a standardized representation of orientation. Achieving this alignment involves calculating a rotation matrix R, which, when applied to Npred, orients the vector parallel to the Z-axis. This is crucial for maintaining consistent orientation representation across different scenes and supports applications requiring a fixed reference frame. The alignment can be mathematically represented
Natigned
R N Mpreds
(14)
where Natigned is the resulting vector aligned with the Z-axis, ensuring a uniform approach to interpreting surface orientations various applications.
3D Box Filtering Module: Similar to [53] this module's goal is to improve annotation quality by converting 2D bounding boxes into their 3D counterparts for objects within indoor scenes. This conversion process is pivotal for achieving more accurate spatial representation of objects. However, the transition from 2D 3D can introduce discrepancies between the generated 3D bounding boxes and the objects' actual dimensions. To mitigate this, we employ Large Language Models (LLMs), such as GPT-4 I. to obtain size priors for various object categories encountered in indoor environments. GPT-4[II's capability to provide reliable size estimates ensures that the adjusted 3D bounding boxes more closely align with the real-world sizes objects, thereby enhancing the precision of annotations in indoor scene analysis.
16

---

II. Unexpanded Category AP Analysis
This section provides detailed discussion on the 20 categories within the SUNRGBD and datasets that were not fully explored in the main paper.
ScanNet 12
Table 6 and Table
corresponds to Table in the main paper.
Table 8 and Table corresponds to Table the main paper.
Methods
Mean
toilet
bed
chair
sofa
dresser
table
cahinet
bookshelf'
pillow
sink
OV Net
38
5.86
17.36
25.83
0.54
28.85
0.02
0.94
0.19
0.0027
2.75
5.32
OV-3DET
5.69
23.44
22.39
7.07
24.15
0.98
0.21
0.12
0.03
0.18
1.35
OV-3DETR (4
5.3
25.34
10.12
1.11
28.2
0.001
0.05
0.029
0.0001
3.05
2.34
ImOV3D (Our))
12.64
55.86
28.82
11.72
28.16
0.453
7.44
1.23
0.05
29.12
Methods
bathtuh
refrigerator
desk
nightstand
counter
door
curtain
hox
lamp
bag
Noe 38
18.71
7.63
2.2
0.0059
0.04
0.29
1.16
0.02
5,28
0.03
OV-3DET
16,98
3.1
0.0012
0.12
0.12
4.15
7.2
2.1
OV-3DETR 24
24.68
4.24
1.6
0.0034
0.02
0.47
1.45
0,016
2.23
1.01
ImOV3D (Ours)
43.61
14.90
13.05
0.09
0.17
1.77
0.56
Table 6: Dataset During the Pretraining Stage. 'Mean' represents the average value of all the 20 categories.
Comparison of Average Precision (AP) Scores for 20 Specific Categories the ScanNet
Methods
Mean
toilet
bed
chuir
bathtub
sofa
dresser
scanner
fridge
lamp
desk
OV Not
38
5.18
25.16
20.63
1.59
14.43
24.98
0.0021
0.001
0.19
1.69
0.52
OV-3DET
30
5,47
21.92
16.94
11.18
12.74
16.81
1.674
0.89
4.12
5.60
0.88
OV-3DETR
34
5.24
33.50
8.47
2.61
26.54
17.05
0.0003
0.0001
2.29
0.73
0.05
ImOV3D (Ours)
12.61
53.45
21.61
13.65
49.00
25.20
8.49
3,23
Methods
table
stand
cabinet
counter
bin
bookshelf
pillow
microwave
sink
stool
OV-VoteNet
18
0.5
0.0004
0.0055
0,0007
0.0007
0,0008
8.66
0.01
5,1
0.11
OV-3DET 3D
1.08
0.12
0.89
0.93
0.0003
1.47
7.43
3.54
0.13
1.01
OV-3DETR 34
0.05
0.0001
0.0004
0.0001
0.0001
0.0002
9.51
0.0001
2.47
1.63
ImOV3D (Ours)
6.17
0.0011
0.0354
0,0022
12.22
0.0048
14.20
11.98
20.63
7.39
Table Comparison Average Precision (AP) Scores for 20 Specific Categories on the
SUNRGBDE3] Dataset During the Pretraining Stage. 'Mean" represents the average value of
all the 20 categories.
Methods
Mean
toilct
bed
chair
sofa.
dresser
table
cabinet
bookshelf
pillow
sink
OV-3DET 30
18.02
$7.29
42.26
27.06
31.5
8.21
14.17
2.98
5.56
23
31.6
CoDA
19.32
68.09
28.72
44.57
3.41
0.03
27.95
45.26
ImOV3D (Our))
21.45
79.23
52.07
29.25
60.20
0.47
21.67
2.21
1.37
23.50
42.02
Methods
baihtub
refrigerator
desk
nightstand
counter
door
curtain
box
lamp
bag
OV:3DET 30
56.28
10.99
19.72
0.77
0.31
9.59
10.53
3.78
2.11
2.71
CoDA 5
50.51
6.55
12.42
15.15
0.68
7.95
0.01
2.94
0.51
2.02
ImOV3D (Ours)
51.39
25.65
30.11
0.39
0.66
1.05
0.07
2.94
2.70
2.13
Table 8:
Comparison of Average Precision (AP) Scores for 20 Specific Categories on the ScanNet [2
Dataset During the Adaptation Stage. 'Mean' represents the average value of all the 20 categories.
Methods
Mean
bed
chair
bathtuh
dresser
scanner
fridge
lamp
desk
OV-3DET 30
20.46
72.64
66.13
34.8
44.74
42.10
11.52
0.29
12.57
14.64
ImOV3D (Ours)
22.53
76.70
65.36
32.21
55.77
50.01
0.20
2.94
13.77
26.20
11.54
Methods
table
stand
cabinet
counter
bin
bookshelf
pillow 10.27
microwave
sink
stool
OV-3DET 30
23.31
2.75
3.4
0.75
23.52
9.83
1.98
18.57
ImOV3D (Ours)
16.68
0.08
0.47
0.03
29.21
0.10
15.71
20.74
30.21
2.81
Table 9: Comparison of Average Precision (AP) Scores for 20 Specitic Categories on the
SUNRGBD[3 Dataset During the Pretraining Stage. 'Mean' represents the average value of
all the 20 categories.
17

---

III. Evaluating the efficiency of GPT-4 in the 3D box filtering module
In this section, we will discuss the efficiency of GPT-4 ① in the 3D box filtering module. In order to have a good comparison, we choose the SUNRGBD 4 as visualizing dataset. To clearly demonstrate the results, we select the top 10 classes with the most instances in the validation set as our visualization targets.
The volume ratio for these 10 classes is defined Ratioy
ULenar Warven's " The volume ratio provides an insightful metric to compare the performance of the 3D box filter module powered by GPT-4 [ with the ground truth (GT). A ratio close to indicates that the volume of the predicted box is very similar the true value, indicating high precision. By calculating the volume ratio Ratioy for each instance, gather statistical data and use Kernel Density Estimation (KDE) to analyze and plot the distributions of the volume ratios. Results are presented in Figure g
Ix W H
Groundtruth
bod L.2.1. Wt 16, H.O.9
bed 2.0.0 W: 1.6, H. 0.5
cabinet- C.8. W: Ht L.0
Codanel -:: 0.5, Wr: L-C, C, 1.0
chair L.S., W: D.5, 0.5
chair L.O.6 W: C.S. 0.9
1.0
122
lamp L.04,
lamp-1:
0.8
sefa- L:0. 19.
sofa L.200 Mt 0.5, H.G.9
piln--L:44, Mt n.S. H:G.3
pillow L: 0.4, W: 066 0.2
L.
Dondil
0.8
wink
0.4
0.4
Rate
Rate
Figure 8: Comparison of Volume Ratios: This image shows the efficiency of GPT-4 4 [1] using the SUNRGBD dataset by comparing volume ratios of the top 10 classes using Kernel Density Estimation (KDE).
Figure 8 fully demonstrates the effectiveness of the meansize database constructed by GPT-4 M. Subsequently, we used the constructed meansize in the 3D box filtering module filter ByDpseudos with the results shown in Figure 9
20000
35200
5500
boots
2000
Nock
Figure Results of Using the Meansize Database the 3D Box Filtering Module: This figure shows the outcome of applying the meansize database constructed by GPT-4 ①1 in the 3D box filtering module filter B3Dpseado with threshold T 0.1. The figure presents the number of 3D box of the top 50 classes with the most instances. The results highlight the efficiency and accuracy of the CPPTconstructed meansize in improving the performance the filtering process.
18

---

During the finetuning of ControlNet (55), we do not use any text prompts but instead adhere to the official parameter settings recommended by ControlNet (55). We train the model for 180 epochs, through which the model learns how to effectively map the color information from the target images IV. Detailed Experimental Implementation
IV.A Fine-tuning Model Details (Detic (58) and ControlNet (55))
This subsection delves into the model adjustment and parameter selection process during the fine-tuning phase.
Detic [58]: To finetune Detic [38] for our specific application, we first need clarify the necessity for such an adjustment. The off-the-shelf Detic model is designed and optimized for ground truth images. However, our requirement for Detic [58] significantly diverges because it is be utilized within the 2D OV Detector component of our ImOV3D project. In this component, the images we ground truth images but pseudo images generated through ControlNet (55). Such images differ visually from ground truth images, which could potentially affect the detection performance of the Detie (58) model.
Given this, we decided to finetune Detic [58] to better suit our specific scenario. The finetuning process involves replacing the annotations based on ground truth images, which the Detic [58] model originally uses, with annotations from a 2D dataset that are accurate for our ControlNet- -generttd pseudo images. This replacement is aimed enabling Detic [58] to more accurately recognize and
process objects within these generated images, considering they differ in quality and characteristics from the real images used in the model's initial training.
Several key steps are involved this finetuning process. First, we need to collect and prepare annotated data from the 2D dataset that applies to our pseudo images. Then, we use these annotations in the training process of the Detic (58) model to adapt to our unique image characteristics. Through this process, ensure that the Detic (58] model maintains efficiency and accuracy when dealing with images generated through ControlNet [55), thereby maximizing its efficacy in our ImOV3D project.
Finetuning the Detic (58] model not only improves its performance on specific images but also further validates the feasibility and flexibility of our approach. This process illustrates that by meticulously adjusting and optimizing model parameters the face of varying application scenarios, we can effectively enhance the model's performance and adaptability.
The objective of finetuning ControlNet [55] is to add color to the rendered point cloud images. Originally, point cloud data lacks color information, meaning the depth images projected from these point clouds are also devoid of color. However, in many applications including but not limited to 3D reconstruction, augmented reality (AR), and virtual reality (VR), rich color and texture information are crucial for enhancing the visual quality of the final images. This necessitates the finetuning ControlNet (55)
ControlNet [55]:
The finetuning process requires two types of data: original images and target images. In our scenario, the original images depth images obtained by rendering point cloud after partial-view removal, as shown in Fig 7 (B). While these depth images accurately represent the geometric information of the scenes, they lack color information. The target images, on the other hand, are ground truth 2D images corresponding to the point cloud data, as shown in Fig 7 (C), containing the rich color and texture information we aim incorporate.
onto the corresponding depth images. This process not only enhances the visual appearance of the rendered images, making them more realistic and rich, but also improves the model's capability to handle color and texture variations across different scenes.
Through such finetuning, ControlNet [55] can more accurately colorize depth images rendered from point clouds, preserving the geometric details from the original point cloud data while also integrating real-world color and texture characteristics. Such improvements are extremely beneficial for subsequent 3D vision applications, such as scene understanding and interaction in 3D. In summary, the process of finetuning ControlNet [55] is a meticulous attempt to adjust and optimize model performance, aiming achieve better results in handling specific tasks.
19

---

IV. Main Experiment and Baseline Experiment Details
outlines the setup and execution process of the main experiment, including comparisons with baseline experiments.
We detail and present four sets of key pretraining parameters. Table 10 showcases the training configuration for the ImOV3D method, an improved model we propose. Table 11 provides a detailed description of the training parameters for OV- DV- serving another baseline model. Table 12 focuses on OV-3DETR (34), presenting its specific training settings. Finally, Table TableinJoutiinee the training parameter details for the OV-3DET (30] model. These four tables aim clearly display the specific parameter settings of each method during the pretraining stage, offering a direct perspective for comparison to the readers.
Table 14 presents the parameter configurations of our experiments during the adaptation stage.
Table 10: Key Parameters for ImOV3D Pretraining Configuration
Parameter
Value
Description
-tower_weigee
0.3, 0.3, 0.4
Fusion weights for different input modalities Number of samples per batch
-batch_size
12
-learning_ rate
0.001
Initial rate for model training
-weight_decay
0
L2 regularization coefficient Total number of pretraining training epochs
-max_epoch
180
-1r_decay_steps
80, 120, 160
Epochs where learning rate decays
1r_deccy_raate
0.1, 0.1, 0.1
Learning rate decay factors
Table 11: Key Parameters for OV- Voteetee Pretraining Configuration
Parameter
Value
Description
-pc_only_weight
Point clouds only input
-batch_size
12
Number of samples per batch
-learning_rate
0.001
Initial rate for model training
-weight_decas
0
L2 regularization coefficient
-max_epoch
180
Total number of pretraining training epochs
-ir_decay_steps
80, 120, 160
Epochs where learning rate decays
-Ir_decay_rater
0.1, 0.1, 0.1
Learning rate decay factors
Table 12: Key Parameters for OV-3DETR Pretraining Configuration
Parameter
Value
Description
-max_epoch
180
Training duration in epochs.
-base_lr
7e-4
Initial learning rate. Samples per GPU.
-batchsize__ort Baciiieeeprr_eee gguu
12
-weight_decas
1,2 regularization.
wr
le-6
Warm-up learning rate.
-warm_lr
wwarm_lr_epochs
Duration of warm-up phase.
-final_ir
le-6
Learning rate for final phase.
20

---

Table 13: Key Parameters for OV-3DET Pretraining Configuration
Parameter
Local Phase
DTCC Phase
-max -max_epoch
200
50
-nqueries
128
128
-base_lr
4c-4
lc-4
-warm_lr_epochn
-batchsize_per__e
12
12
-final_11 1r
le-5
le-5
Table 14: Key Parameters for ImOV3D Adaptation Configuration
Parameter
Value
Description
-tower_ww nower_weighta
0.3, 0.3, 0.4
Fusion weights for different input modalities Number of samples per batch
-batch_size
12
-Learning_ran
0.0005
Initial rate for model training L2 regularization coefficient Total number of adaptation training epochs
-weight_decce
-max_epoch
100
-1r -Ir_decay_stepp
40, 80
Epochs where learning rate decays
-r_deecay_rates
0.1,
Learning rate decay factors
21

---

V. Visualization Comparison during Pretraining Stage
The visualization colors have specific meanings, and we're providing ImOV3D the prompt: "Please help me locate the /category/". In the image, bed is represented by red lines, lamp by light green, a chair by pink, a table by blue, sofa by purple, pillow by orange, and cabinet by dark green.
Ground Truth Images
Ground Truth Point Clouds
OV-3DET
ImOv3D(Ours)
During the pretraining stage, the visual comparison tested on the SUNRGBD 3 dataset includes four columns: the first column shows the ground truth images, the second column displays the ground truth point clouds, the third column presents the detection results OV-3DET (3), and the fourth column reveals the detection outcomes by IMOV3D.
Figure 10:
22

---

Ground Truth Images
Ground Truth Point Clouds
OV-3DET
ImOV3D(Ours)
During the pretraining stage, the visual comparison tested on the ScanNet 12 dataset includes four columns: the first column shows the ground truth images, the second column displays the ground truth point clouds, the third column presents the detection results by OV-3DET and the fourth column reveals the detection outcomes by ImOV3D.
Figure 11:
23

---

VI. Explanation of the formula for the Rotation Correction Module
In this section, we explore Equation[Tin depth, utilizing Rodrigues' rotation formula. It calculates the rotation matrix R E SO(3) for rotation by 0 (less than 180%) around the axis defined by the unit vector á (Rz; Ry, n2). The Rodrigues' rotation formula matrix is defined as:
R= I+ (sin 0)N + (1 cos N)
Where is the identity matrix, and the skew-symmetric matrix N for the unit vector át is constructed as:
T2
fy
N
0
-My
To align unit vector Nprd with unit vector Zpred, start from the definitions of the inner product and cross product:
Npreat Zepred
N Pred 7prodd cos 0, |Npeed Zerrdl sin
Thus, we have:
1 Npred X Zpred
t sin f
Accordingly, define:
n2
0
-2
fy
del (sin N)N
K
0
-2
ny
fl
0
VVy
Ur
Based on Rodrigues' rotation formula, we have:
COS 0
== 1=I+K K
K²
sin² 0
= K 12 1 Npred Zpred
||0|
24

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We conduct comprehensive experiments, and the results included in our paper support the main claims made in the abstract and introduction well.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We have addressed the limitations of our work, including assumptions, scope, performance factors, computational efficiency, and privacy and fairness considerations, to ensure a balanced view of our research.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
25

---

Answer: [Yes]
Justification: For each theoretical result in our paper, we have provided a complete set of assumptions and a complete (and correct) proof. All theorems, formulas, and proofs are numbered and cross-referenced for clarity and consistency. All assumptions stated or referenced in the statement of any theorem. The proofs are presented in the main paper, and we have included brief proof sketches to provide intuition. Additionally, any theorems and lemmas upon which the proofs depend are properly cited, ensuring the rigor and reliability of our theoretical results.
explicitly
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have uploaded the code and fully disclosed all the necessary information required reproduce the main experimental results, ensuring the reproducibility of the main claims and/or conclusions of the paper. We recognize the importance of enabling other researchers to replicate and verify our experimental findings and have taken appropriate measures, including providing detailed descriptions of our experimental setup and the necessary code, facilitate transparency and reprodueibility of our research.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (c.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
26

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have provided open access the data and code, along with clear in- structions in the supplementary material to replicate the main experimental results. This includes details on data access, preparation, and the exact commands needed execute the experiments. Anonymous versions of data and code have been released to maintain submission integrity.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.ca
www.sssiinnooooioiiioo for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wwtp:/ccppccccgudeeeeeeeeeeeeeee
wwwiiisssooooooooooooo for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Our paper specifies all training and testing details, such as data partitioning, hyperparameter selection, and the type of optimizer used, which are essential for understand- ing our results. The experimental setup presented within the core of the paper with a level of detail sufficient for reviewers to comprehend and make sense of the outcomes. Complete details are provided alongside the code, either in the appendix or as supplementary material.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
27

---

The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report enror bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Our paper provides appropriate error bars and statistical significance informa- tion for the experiments, clearly explaining the variability and the method of calculation. We have also specified the assumptions made. An error bar or the variance value included in all the results, either in table or a figure.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Our paper provides sufficient information regarding the computational re- sources used for each experiment, including the type of computational work (CPU GPU), memory requirements, and execution time. This information is essential for replicating the experiments and is detailed ensure clarity and reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:p////aarpp...oop/a/////teae
28

---

Answer: [Yes]
Justification: Our research fully complies with the NeurlPS ethical guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our paper discusses the broader applications of the work, we have conducted comprehensive assessment of the societal implications that the technology might bring.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Our paper does not present any data or models that carry high risk of misuse, such as pre-trained language models, image generators, scraped datasets. The research conducted does not involve risks that would necessitate the implementation of security measures to prevent abuse.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
29

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: our paper, we have properly credited the creators and original owners of all used assets, including clear mention and respect for their licenses and terms of use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We have focused on the research presented without the need to create share new assets that would necessitate additional documentation or user consent processes.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourced experiments or research with human subjects, hence there no need to provide the complete text of instructions for participants, screenshots, detailed information regarding compensation.
30

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve research with human subjectsthe research conducted does not pose any potential risks to participants that would necessitate disclosure or ethical review.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
31


Reviews:
[{"review_id": "er8MsSW4oK", "reviewer": "Reviewer_MLwT", "summary": "This work addresses the problem of the lack of 3D data and attempts to train open-vocabulary point-based 3D detection models solely with 2D images to avoid using 3D data and annotations. It proposes a pseudo data generation pipeline for this purpose, which generally follows a paradigm of estimating depth, projecting 2D to 3D, and rendering back from 3D to 2D to obtain 2D-3D paired pseudo data. Experiments show that models trained with the generated data achieve better results than previous methods.", "questions": "See the weakness section.", "limitations": "See the weakness section.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation of this paper is reasonable.\n2. The proposed method is needed in this field and indeed contributes to addressing the 3D data issue.\n3. The experimental results, to some extent, show that the proposed method is effective.", "weaknesses": "My major concerns lie with the unclear motivation and implementation of some components.\n\n1. Intuitively, using a fixed camera intrinsic causes inaccurate projection. This work does not propose specific designs for addressing this problem nor discuss the limitations of using fixed intrinsics.\n2. The reasons for using Formula (1) are not explained.\n3. The method for lifting 2D bounding boxes to 3D, as described from L161-L163, is unclear. Did the authors use the minimum and maximum depth of the four corners of the 2D bounding boxes to generate the frustum? How is the orientation of the 3D bounding boxes determined? How is the gap addressed between the pseudo 3D bounding boxes, which are oriented, and the 3D ground truth bounding boxes in ScanNet, which are axis-aligned?\n4. For the pre-training stage, why not use the original 2D images directly for training, considering that there is still another adaptation training stage to handle the domain gap between images from real point clouds and pseudo point clouds? In other words, why not use the adaptation training stage to address the gap between real images and images from real point clouds, thereby skipping the process of generating pseudo images from pseudo point clouds?\n5.  As the key contribution of this work is the data generation pipeline, there should be more qualitative results for each step to demonstrate how this pipeline truly works. In the rebuttal phase, I suggest the authors show the results (e.g., intermediate results in Figure 2, point clouds with pseudo 3D bounding boxes can be shown using Open3D) using the first image of scene0000_00 in ScanNet and the first image of scene0012_01 in ScanNet as examples.\n\nThe data generation pipeline itself is typical and lacks novelty (e.g., similar work like SpatialGPT exists). It is difficult to fully understand the implementations and evaluate the performance of the pipeline solely through the manuscript. Nevertheless, this work involves significant engineering effort and tricks and contributes to the 3D perception community. If the code is open-sourced, I suggest accepting the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**：\\\nIn depth estimation research, using fixed camera intrinsics is common practice [1][2]. While we've followed this approach, we recognize it may lead to inaccurate bounding box sizes. To address this, we've implemented a scale filter using GPT-4, adjusting bounding boxes based on empirically determined object sizes from LLMs. This correction has significantly improved performance. As shown in Table 3, the first row presents results without bounding box correction, and the second row shows improved performance with the filtering strategy: mAP0.25 increased by 1.65% on the SUNRGBD dataset and by 1.27% on the ScanNet dataset. Thank you for your valuable feedback; we will include a discussion on this limitation in the paper.\\\n**Q2**：\\\nThe motivation for using this formula is explained in the main text from L143 to 150. We explore Equation (1) in depth within paper, utilizing Rodrigues' rotation formula [3]. It calculates the rotation matrix $R$ for a rotation by $\\theta$ (less than $180^\\circ$) around the axis defined by the unit vector $\\hat{n}=(n_x,n_y,n_z)$. The formula is defined as:\n$$R=I+(\\sin\\theta)N+(1-\\cos\\theta)N^2$$\nWhere $I$ is the identity matrix, and the skew-symmetric matrix $N$ for the unit vector $\\hat{n}$ is constructed as:\n$$N=\\begin{bmatrix}0&-n_z&n_y;&n_z&0&-n_x;&-n_y&n_x&0\\end{bmatrix}$$\nTo align unit vector $N_{pred}$ with unit vector $Z_{axis}$, we start from the definitions of the inner product $N_{pred} \\cdot Z_{axis} = \\cos\\theta$ and cross product $v = N_{pred} \\times Z_{axis} \\Rightarrow v = \\sin\\theta\\hat{n}, |v| = \\sin\\theta$.\nAccordingly, define:\n$$K\\stackrel{\\mathrm{def}}{=}(\\sin\\theta)N= \\begin{bmatrix}0&-v_z&v_y;&v_z&0&-v_x;&-v_y&v_x&0\\end{bmatrix}$$\nBased on rotation formula, we have:\n$$R=I+K+\\frac{1-\\cos\\theta}{\\sin^2\\theta}K^2 $$\n$$=I+K+K^2\\frac{1-N_{pred}\\cdot Z_{axis}}{||v||^2} $$\n**Q3**：\\\n【**Generating a Visual Frustum**】 \\\nWe utilize the four corner points of a 2D bounding box, along with the minimum and maximum depth values from the depth image, to create a visual frustum. Specifically, we first convert the 2D image coordinates into 3D spatial coordinates using the intrinsic matrix. Then, based on the depth information, we calculate the actual positions of these corner points in 3D space, thereby generating the visual frustum. Within the visual frustum, we extract the corresponding point cloud data. To ensure the stability and efficiency of the calculations, we perform downsampling on the point cloud data. Subsequently, we apply the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) clustering algorithm to the extracted point cloud data to remove outliers and noise, retaining only the main point cloud clusters.\\\n【**3D Bounding Box Orientation**】\\\nBy applying Principal Component Analysis (PCA) to the main point cloud clusters, we calculate the main orientation angle (yaw) of the point cloud. Specifically: PCA analysis is conducted on the X-Y plane of the point cloud data to obtain the principal component direction vector. The main orientation angle (yaw) is calculated based on the principal component direction vector. The point cloud is rotated using a rotation matrix to align the main direction with the X-axis. The range of the bounding box after rotation alignment is calculated. The bounding box is then rotated back to its original direction to obtain the final 3D bounding box position and orientation. We construct the final 3D bounding box description based on the calculated center point coordinates, dimensions (width, depth, and height), and orientation angle (yaw).\\\n【**Regarding the Orientation of Bounding Boxes in the ScanNet Test Dataset**】\\\nThe official bounding boxes provided by the ScanNet dataset are axis-aligned, in our research, we have adopted the baseline method (OV3DET and CoDA), following the common practices established by previous studies, using Principal Component Analysis (PCA) to determine the orientation of the bounding boxes. In all our experiments, we uniformly used Oriented Bounding Boxes for testing, having standardized the representation of bounding boxes, thus eliminating any discrepancies in bounding box orientation. At lines L162-163, we have provided a description pertinent to the issue raised. We will elaborate more on this in the paper to ensure clarity. \\\n**Q4**：\\\nAs mentioned in the global reply.\\\n**Q5**：\\\nThank you for your suggestion, we have followed your request and displayed the visualization of each step in Fig2 and Fig3 of the global response PDF.\\\n**Q6**：\\\n【**About lack novelty**】\\\nOur data generation pipeline is meticulously designed, and we do not agree that similar work implies a lack of novelty in our approach. The works you mentioned, such as Spatial-RGPT and Spatial VLM, are considered concurrent under the NeurIPS Policy, which treats works published within two months of submission as being from the same period. Our task design and complete pipeline differ significantly, particularly in our focus on 3D detection and detailed descriptions of 3D bounding boxes, including orientation and size. During the rebuttal phase, we will provide more detailed visualizations of the pipeline in the global PDF, Fig 2 and 3, to better illustrate the entire process. We are committed to open-sourcing our code to enable others to explore further applications based on our work, fostering development and innovation within the community.\n\n【Dear Reviewer】We value your feedback and look forward to discussing any additional issues in the next communication phase.\\\n**References** \\\n[1]Pan Ji, Runze Li, Bir Bhanu, Yi Xu, \"MonoIndoor: Towards Good Practice of Self-Supervised Monocular Depth Estimation for Indoor Environments\"\\\n[2]Vitor Guizilini, Igor Vasiljevic, Dian Chen, Rares Ambrus, Adrien Gaidon, \"Towards Zero-Shot Scale-Aware Monocular Depth Estimation\"\\\n[3]Richard M. Murray, Zexiang Li, S. Shankar Sastry, pp. 26-28,\"A Mathematical Introduction to Robotic Manipulation\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dBwsXB8VFr", "reviewer": "Reviewer_teog", "summary": "The paper introduces ImOV3D, a novel framework for open-vocabulary 3D object detection (OV-3Det) that learns exclusively from 2D images. The method addresses the scarcity of annotated 3D data by leveraging the wealth of annotations in 2D images. ImOV3D employs a pseudo-multimodal representation that bridges the gap between 2D images and 3D point clouds, enabling effective knowledge transfer. The framework converts 2D images to pseudo-3D point clouds using monocular depth estimation and vice versa, integrating 2D semantic information with 3D spatial characteristics. Extensive experiments on SUNRGBD and ScanNet datasets demonstrate significant performance improvements over existing methods.", "questions": "Please see the weakness.", "limitations": "Please see the weakness.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.The approach of using only 2D images to train a 3D object detector is highly innovative and addresses a critical bottleneck in the field.\n\n2.The framework’s ability to convert 2D images into 3D point clouds and back into 2D images provides a robust method for integrating multimodal data.\n\n3.The paper includes extensive experiments and ablation studies on benchmark datasets, showcasing significant improvements over state-of-the-art methods.\n\n4.The framework achieves impressive results without ground truth 3D data, and even better performance with minimal 3D data for fine-tuning.", "weaknesses": "Real-World Performance: How does ImOV3D perform when applied to real-world 3D data beyond the benchmark datasets used in the paper?\n\nAdaptability: Can the model be easily adapted or fine-tuned for different types of 3D environments, such as outdoor scenes or highly dynamic environments?\n\nResource Requirements: What are the computational resources required for training and inference? Can the method be efficiently scaled for use in large-scale applications? What is the latency?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**：\\\nWhile our method, ImOV3D, has been primarily designed and evaluated for indoor scenes, we recognize the importance of assessing its performance on real-world 3D data beyond the benchmark datasets mentioned in our paper. To address this, we conducted experiments on the KITTI dataset, which consists of outdoor scenes, to evaluate how well our method generalizes to different datasets.\n##### Table 1: Results from the Pretraining Stage Comparison of Open Vocabulary 3D Object Detection Performance on KITTI Moderate Difficulty Level\n\n| **Method**   | **Car** | **Pedestrian** | **Cyclist** | **mAP\\@0.25** |\n|--------------|---------|----------------|-------------|--------------|\nOV-VoteNet | 11.32 | 8.64 | 8.08 | 9.35 |\nOV-3DETR | 13.68 | 12.08 | 8.56 | 11.44 | \nOV-3DET | 12.92 | 6.24 | 8.52 | 9.22 |\nImOV3D | 20.42 | 11.24 | 14.96 | 15.54 |\n\nThe results indicate that our method, while showing strong performance on indoor datasets, also exhibits a reasonable level of robustness and adaptability when confronted with outdoor scenes. Compared to other baselines, our mAP\\@0.25 is 4.1% higher. The pretraining images encompass both indoor and outdoor scenes, hence allowing us to naturally generalize to outdoor scenes.\n\n**Q2**：\\\nWe fine-tuned our model using real point cloud data, and the results show that our method outperforms OV3DET by 5.5% in the mAP\\@0.25, indicating a significant enhancement in performance after fine-tuning (compared to the data in Table 1 of Q1). However, the relatively limited categories in outdoor scenes restrict the full potential of the open-vocabulary detector. Moreover, the differences between indoor and outdoor data also affect the outcomes. More refined adjustments and designs tailored for outdoor scenarios, our results could further improvement.\n##### Table 2: Results from the Adaptation Stage Comparison of 3D Object Detection Performance on KITTI Moderate Difficulty Level\n\n| **Method**   | **Car** | **Pedestrian** | **Cyclist** | **mAP\\@0.25** |\n|--------------|---------|----------------|-------------|--------------|\nOV-3DET | 42.65    | 15.71    | 18.20    | 25.52  |\nImOV3D  | 45.51    | 19.53    | 28.02    | 31.02  |\n\n\n**Q3**：\\\n【***Computational Resources***】 In our research project, the model's pre-training stage utilized 8 NVIDIA GeForce RTX 3090 GPUs, taking 72 hours, while the adaptation stage required only 12 hours, with both stages demanding approximately 16GB of GPU memory per GPU with a batch size of 12. 0.2s per scene on average on the scannet and sunrgbd dataset. \\\n【***Scalability and Latency***】Our method is designed with scalability, suitable for large-scale applications. Its core strength lies in the flexibility of its backbone network, which can be seamlessly replaced with the latest efficient architectures, thereby further enhancing efficiency with future technological advancements.\n\n【Dear Reviewer】After reviewing our rebuttal response, do you have any further questions or areas that require additional clarification? We place great importance on your opinions and hope to continue the discussion and address any remaining concerns you may have in the upcoming communication session.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BdFBkP4iNr", "reviewer": "Reviewer_TsJm", "summary": "This paper presents a novel LiDAR-based open-vocabulary 3D detection model that relies solely on 2D images for training, without using any 3D annotations. During the training phase, a pre-trained monocular depth estimation model generates depth maps from 2D images, which are then projected into pseudo point clouds. ControlNet is used to render 2D pseudo images from these pseudo point clouds. The 3D detection model takes both pseudo 3D point clouds and pseudo images as input to predict 3D bounding boxes. During inference, the model uses ground truth point clouds and rendered 2D images to predict the 3D boxes. Additionally, the authors employ GPT to filter the pseudo annotations. The results demonstrate that this model achieves state-of-the-art performance on the SUNRGBD and ScanNet datasets. The authors have also released their code in the supplementary materials.", "questions": "Please the authors address my questions presented in the weaknesses.", "limitations": "Not obvious limitation associated with broader societal impacts.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed approach is novel as it only requires 2D images to generate pseudo point clouds and pseudo images for training the 3D detection model, which includes auto-generating pseudo 3D ground truth. \n2. The model outperforms existing SOTA models on two datasets.\n3. The paper is well-written with a detailed methodology, making it easy to understand the proposed method. \n4. The authors provide the code, ensuring reproducibility.", "weaknesses": "The model achieves great performance in open-vocabulary 3D detection, but there are several concerns that need to be addressed:\n\n1. Given that the inference input is only the point cloud, why is the training input limited to images? The training data also includes point clouds; why not utilize these as well?\n2. 3D Pseudo Label Generation: The pseudo point cloud is noisy, making it challenging to generate accurate 3D pseudo ground truth from it. The paper lacks details on how the 3D boxes are fitted to the point cloud and the accuracy of the pseudo ground truth. Providing these details is crucial.\n3. Complex Pipeline: The overall pipeline is complex. Could the image generation step be removed and the original image used instead? During inference, could both image and LiDAR data be used, as in OV-3DET?\n4. Training Data Overlap: The paper assumes that the training data does not include LiDAR. Is there any overlap between the depth model training data and the detection dataset?\n5. Minor issue: Does not cite ControlNet in the main paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**：\\\nWe focus on 2D images because real-world 3D point cloud data is not only limited in quantity but also has relatively few annotations. At the same time, we have observed that 2D image data is not only abundant in quantity but also rich in annotation information. Based on these observations, we wish to explore the potential of 2D images to enhance 3D understanding performance under the condition of having only 2D images.\n\nIn the adaptation stage, we do indeed use real 3D point cloud data, but the main purpose is to reduce the differences between the model's performance across different data domains, that is, to minimize the domain gap. The reason we choose not to use point clouds and images simultaneously during the training phase is that the existing RGB-D paired datasets are small in scale. We do not want the model to be optimized only for these limited datasets during the adaptation stage, thereby restricting the model's generalization ability.\n\nTherefore, our approach aims to improve the performance of the 3D detection model by making full use of the rich 2D image data and utilizing the limited 3D point cloud data in the adaptation stage to optimize the model. This ensures that the model maintains efficient and accurate performance when facing real-world data, while also possessing good generalization capabilities.\n\n**Q2**：\\\n【***3D Pseudo Annotation Generation***】\\\nFirstly, we determine the boundaries of the frustum based on the 2D bounding boxes, and then extract all points within the frustum from the 3D point cloud. The extracted point cloud may contain background points and outliers. To remove these points, we employ a clustering algorithm to analyze the point cloud, gathering points that belong to the same object together. Through the clustering results, we can identify and remove background points and outliers that do not belong to the target object, thus obtaining cleaner point cloud data. After removing non-target points, we calculate the 3D pseudo bounding box based on the remaining points. This includes determining the center position, size, and orientation of the pseudo box. Typically, the center position can be determined by the centroid of the point cloud, the size can be estimated by the boundary of the point cloud, and the orientation can be determined by the object's main axis or normal. Therefore, we can generate corresponding pseudo labels, including the position, size, and orientation information of the box. \\\n【***The Accuracy of the Pseudo Labels***】\\\nSince the depth images Dmetric obtained from monocular depth estimation may contain noise, this can affect the accuracy of the 3D bounding boxes. To address this issue, we use a 3D bounding box filtering module to filter out inaccurate 3D bounding boxes. We have constructed a median object size database using GPT-4, by asking for the average length, width, and height of specific categories, defining a filtering criterion for comparing the size L, W, H of each object in the scene with the median size. If the ratio of the object's dimensions to the median size is within a range of the threshold T, the object is retained. We filter out bounding boxes that do not meet the size criteria before training. During inference, we remove categories that are semantically similar but have different sizes to prevent errors such as misidentifying a book as a bookshelf. From Table 3 in the paper, the quantitative results of this part can be seen. We will elaborate this more clearly in the main paper. \\\n**Q3**：\\\n【***Answer questions about the complex pipeline***】\\\nUnderstanding 3D scenes using only images is an incredibly challenging problem. That's why we've designed this component and the entire pipeline. Although it may appear complex at first glance, every part of it is essential. We haven't included any components to make it redundant or to add unnecessary complexity. Instead, this seemingly complex pipeline is carefully crafted to include everything necessary to effectively address the task. \\\n【***Regarding the possibility of removing image generation step***】\\\nPlease review the global reply.\\\n【***Inference process***】\\\nThe OV3DET operates on a point cloud only basis during the inference process, which is not what question mentioned as being able to input both image and point cloud simultaneously. On the contrary, our pipeline can incorporate images during the inference process. However, Tables 3 and 4 in the PDF indicate that using point clouds and images during inference leads to a decline in results. The reason is the domain gap between the rendered images from training and the real images from inference. \\\n**Q4：**\\\nIn our paper, the training data for the depth estimation model comes from a mix of 12 datasets (e.g. NYU DepthV2,KITTI etc), excluding SUNRGBD and Scannet, which are used for detection. This ensures there is no overlap in the training data.\\\n**Q5**：\\\nOur apologies, we will cite \"Controlnet\" in the main paper subsequently.\n\n【Dear Reviewer】After reviewing our rebuttal response, do you have any further questions or areas that require additional clarification? We place great importance on your opinions and hope to continue the discussion and address any remaining concerns you may have in the upcoming communication session.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aJT3NDHEQT", "reviewer": "Reviewer_HQsS", "summary": "The paper aims to develop an open-vocabulary 3D detector with the help of 2D data. The key idea is to lift 2D images to 3D point clouds by using metric monocular depth estimation models, combined with estimating the extrinsics and intrinsics. The 2D bounding boxes are lifted to 3D as well, and filtered based on some constraints like their usual object sizes that are provided by GPT-4. Additionally, the 3D pointclouds are rendered to 2D images and re-texturerized using control-net. Finally, the paper trains a detector on the lifted 2D image and pointcloud data to get a base detector in \"pre-training\" stage. Then, the pseudo labels are generated on real 3D point clouds and projected to 2D RGB images, and trained further for a few epochs. The paper compared against existing baselines like OV-3DET and outperform them.", "questions": "- Some more explanation on the model architecture and losses would help a lot", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- open vocabulary 3d detection is an important problem and the paper makes progress on that front\n- the proposed pipeline of lifting 2d images to 3d images and 3d point clouds to 2d is interesting.\n- i appreciate the through implementation details throughout the paper and the supplementary materials.", "weaknesses": "- the paper claims (even in the title) to make a 3D open vocabulary object detector ONLY from 2D images. However, the results indicate that such a detector is significantly worse than detectors that uses real point cloud data. I think the \"only 2D images\" can be misleading and the authors should consider removing/modifying it.\n- While the writing for the most part is very clear, I am very unclear on the model architecture. The description seems to be missing, and from the figure it seems like the 2D images are processed by an open-vocabulary detector and 3D point clouds by a 3D detector and somehow their predictions are merged to generate the final output. The description of losses could also really benefit a lot from a more detailed description. \n- As already acknowledged in the limitations, the proposed pipelines adds significantly more complexities over OV-3DET while improving the performance of their point cloud version by about 2% (although the increase of the 2D only version is significant).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**：\\\nWe gladly accept the suggestion and are considering modifying it to “*How far can 2D images drive 3D open-vocabulary object detection?* ” We appreciate your advice, but our motivation is to explore how can we fully exploit information from 2D images and how far they can drive the performance of 3D understanding both in an adaptation-free setting and an adaptation setting. We are also open to any suggestions.\\\n**Q2**：\\\nFor a better understanding, please refer to Figure 1 in the global pdf response. \\\n【2D and 3D Fusion Details】\\\n(1) Seed point generation and feature extraction\\\nIn the 3D branch, K seed points are initialized from the 3D point cloud, each defined by its coordinates in 3D space (Kx3). These seed points are further enriched by feature extraction, where each point, in addition to its coordinates, also acquires additional feature representations $F_{pc} \\in \\mathbb{R}^{K \\times F}$. \\\n(2) Lifting 2D information to 3D\\\nUsing a 2D OV detector to identify objects on RGB images generates 2D bounding boxes and semantic labels. These 2D information are transformed into 3D rays through the camera matrix, with these rays originating from the seed points and pointing towards the center of the objects in 3D space. This process elevates the geometric cues from the 2D image into 3D space, aiding in the precise localization of objects. These lifted image features can be represented as $F_{img} \\in \\mathbb{R}^{K \\times (3+F')}$.\\\n(3) 2D and 3D Feature Fusion\\\nDuring the feature fusion phase, the point cloud features $F_{pc}$ are concatenated with the image features $F_{img}$ to form the joint feature representation $F_{joint} \\in \\mathbb{R}^{K \\times (3+F+F')}$. As a result, each seed point contains not only the geometric information in 3D space but also the semantic information integrated from the 2D image.\\\n(4) Multi-Tower Architecture\\\nWe employ a multi-tower architecture to process the point cloud features and 2D image features separately. Each tower focuses on handling one type of input feature and balances the contributions of different modalities through a gradient fusion strategy. The img tower has a weight of 0.3, the point tower has a weight of 0.3, and the joint tower has a weight of 0.4, consistent with the backbone ImVoteNet.\\\n【Loss Details】\\\nFinally, the loss function can be expressed as:\n$$\n\\small\nL_{\\text{total}} = L_{\\text{loc}} +  \\sum_{i} W_i \\times \\text{CrossEntropy}(\\text{Cls-header}(F_{i}) \\cdot F_{\\text{text}}) \n$$\n\nwhere $i$ represents different features, such as $\\text{pc}$, $\\text{img}$, $\\text{joint}$. $W_i$ is the weight corresponding to feature $i$. $L_{\\text{loc}}$ represents the original localization loss function used in ImVoteNet. $F_{\\text{text}}$ denotes the feature extracted by the text encoder in CLIP.\\\n**Q3**：\\\nOur research paper demonstrates the immense potential of using a 2D-only setting. Through our approach, we primarily aim to address the question of how much 2D images themselves can enhance the performance of OV 3D object detection. In our work, this is not just a paper that solves existing problems but also a scientific exploration. We believe that the 2D-only version setting, as an important experimental conclusion, shows people that 2D images can also provide abundant information for 3D understanding. The current increase in the adaptation setting exceeds the improvements made by previous works, such as CoDA. Although our method increases the complexity of training, our ablation study has proven that every step in the training stage is crucial. However, in the inference stage, our scheme is no more complex than the existing OV3DET. In actual use, the complexity of our scheme does not affect the user experience. The latency remains comparable to OV3DET, and the additional memory cost is even smaller. To promote the reproducibility of research, we will also fully open-source our code for everyone to reproduce and further study.\n\n【Dear Reviewer】After reviewing our rebuttal response, do you have any further questions or areas that require additional clarification? We place great importance on your opinions and hope to continue the discussion and address any remaining concerns you may have in the upcoming communication session.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ORfCOMAntv", "reviewer": "Reviewer_m5vv", "summary": "This paper addresses the challenge in Open-vocabulary 3D object detection (OV-3Det), specifically the modality gap between training images and testing point clouds, which hinders effective integration of 2D knowledge into OV-3Det. The main contribution of the paper is a novel method to generate pseudo multimodal data, enabling the training of a 3D encoder to extract 3D features that are coupled with a frozen 2D detector for 3D detection.", "questions": "What about the generalization ability? For example, how does the model perform on outdoor datasets?\n\nThe experiment needs more details. During the inference stage on the ScanNet dataset, do you use only the point cloud information without using the image information?", "limitations": "The authors have adequately addressed the limitations.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The provided pseudo-multimodal data generation method used to train the 3D detector, when coupled with the 2D detector, achieves a notable improvement in object detection on the SUNRGBD and ScanNet datasets.", "weaknesses": "The framework in Fig. 2 is confusing. The authors claim that their model's input is only point clouds, but the framework suggests that both images and point clouds are needed for inference. Additionally, the generalization ability should be discussed, particularly regarding performance on outdoor datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**：\\\n【***Pre-training Stage***】While our method, ImOV3D, has been primarily designed and evaluated for indoor scenes, we recognize the importance of assessing its performance on real-world 3D data beyond the benchmark datasets mentioned in our paper. To address this, we conducted experiments on the KITTI dataset, which consists of outdoor scenes, to evaluate how well our method generalizes to different datasets.\n##### Table 1: Results from the Pre-training Stage Comparison of Open Vocabulary 3D Object Detection Performance on KITTI Moderate Difficulty Level\n\n| **Method**   | **Car** | **Pedestrian** | **Cyclist** | **mAP\\@0.25** |\n|--------------|---------|----------------|-------------|--------------|\nOV-VoteNet | 11.32 | 8.64 | 8.08 | 9.35 |\nOV-3DETR | 13.68 | 12.08 | 8.56 | 11.44 | \nOV-3DET | 12.92 | 6.24 | 8.52 | 9.22 |\nImOV3D | 20.42 | 11.24 | 14.96 | 15.54 |\n\nDuring pre-training stage, the results indicate that our method, while showing strong performance on indoor datasets, also exhibits a reasonable level of robustness and adaptability when confronted with outdoor scenes. Compared to other baselines, our mAP\\@0.25 is 4.1% higher. The pretraining images encompass both indoor and outdoor scenes, hence allowing us to naturally generalize to outdoor scenes. \\\n【***Adaptation Stage***】We fine-tuned our model using real point cloud data, and the results show that our method outperforms OV3DET by 5.5% in the mAP\\@0.25, indicating a significant enhancement in performance after fine-tuning (compared to the data in Table 1). However, the relatively limited categories in outdoor scenes restrict the full potential of the open-vocabulary detector. Moreover, the differences between indoor and outdoor data also affect the outcomes. More refined adjustments and designs tailored for outdoor scenarios, our results could further improvement.\n##### Table 2: Results from the Adaptation Stage Comparison of  Open Vocabulary 3D Object Detection Performance on KITTI Moderate Difficulty Level\n\n| **Method**   | **Car** | **Pedestrian** | **Cyclist** | **mAP\\@0.25** |\n|--------------|---------|----------------|-------------|--------------|\nOV-3DET | 42.65    | 15.71    | 18.20    | 25.52  |\nImOV3D  | 45.51    | 19.53    | 28.02    | 31.02  |\n\n**Q2**：L59-61 \\\nIn response to the question regarding the data used during the inference phase, we confirm that in our experiments on the ScanNet and SUNRGBD datasets, our method strictly used only point cloud data for inference and did not utilize image information. Within our framework, although it may appear that images and point clouds are used concurrently during the inference phase, we would like to clarify that the images presented are actually pseudo images generated by our rendering module based on the input point cloud data. While these pseudo images are visually similar to real images, they do not contain real image data inputs. We have adopted the  previous baseline method (OV3DET and CoDA), it guarantees that our inference is exclusively based on point cloud data, aligning with the methodological rigor of using a single modality for input. You can see more detailed intermediate results of the pipeline visualized in Fig 2 and 3 of the global pdf response.\\\n\n【Dear Reviewer】\nAfter reviewing our rebuttal response, do you have any further questions or areas that require additional clarification? We place great importance on your opinions and hope to continue the discussion and address any remaining concerns you may have in the upcoming communication session.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are deeply grateful for the professional reviews and valuable feedback from all the reviewers. Reviewer MLwT acknowledged the motivation behind our research and its contribution to addressing the 3D data issue; Reviewer teog praised the innovation of our approach and the robust method for integrating multimodal data; Reviewer TsJm appreciated the novelty of our method that generates pseudo point clouds and images using only 2D images; Reviewer HQsS affirmed the detailed implementation and supplementary materials we provided; Reviewer m5vv noted the significant improvements achieved by our method on the SUNRGBD and ScanNet datasets. We value every piece of feedback and have responded to each reviewer's questions individually. More additional experiments were also conducted during the rebuttal phase to support our proposed method, as suggested by the reviewers (given in the PDF file).

In response to the question raised by **Reviewer MLwT Q4** and **Reviewer TsJm Q3** regarding "***whether the image generation step can be removed and original images can be used instead***," we provide the following reply:

（1）Our OV 3D detector utilizes only point cloud input during both the pre-training stage and the adaptation stage. These point clouds contain only geometric information and do not include color information. We need and can only use pseudo point clouds to train a general point cloud renderer that is applicable to various scenarios. This way, we can generate images with only point cloud inputs, thus leveraging the capability of multimodal detection.\
（2）For such a point cloud renderer, it is extremely challenging to narrow the gap between the images rendered from real point clouds and real images. In contrast, the difference between the images rendered from real point clouds and pseudo point clouds is not significant. Therefore, we choose to use pseudo images in the pre-training stage. This way, during the adaptation training and inference stages, the image branch can migrate well since the quality gap between images rendered from real point clouds and those used in training is small.\
（3）If we use real images during the pre-training stage, in a zero-shot setting, the large gap between real images and images rendered from real point clouds will lead to a failure in migration. Even if a small amount of RGB-D data is used for adaptation, it is still difficult to bridge the gap between real images and rendered images, causing the 2D branch to fail. \
（4）In the global response of the PDF: The first row of Table 1 and Table 2 shows the results of consistently using the renderer during pretraining, adaptation, and inference stages. This indicates that maintaining consistent data representation throughout the process can yield optimal performance.

The second row of Table 1 and Table 2 in the PDF's global response reflects the decline in mAP\@0.25 on both datasets when real images are used during training and the renderer is used during inference. This reveals a significant difference between real images and the pseudo images generated by the renderer, a discrepancy that affects the model's ability to transfer knowledge.

Similarly, the second row of Table 3 and Table 4 also shows a trend of performance decline when the renderer-generated pseudo images are used during training and real images are used during inference. This further confirms the negative impact of using different data representations during training and inference stages.

These results suggest that in order to maintain the stability and consistency of the model across different stages, we cannot use different image sources during training and inference. The pseudo images generated by the renderer are indispensable throughout the process because they provide a representation close enough to real images, ensuring the model can learn and transfer knowledge effectively. Although a limited amount of real data is introduced during the adaptation phase to reduce the gap between real and pseudo images, this approach still fails to surpass the performance achieved by using the renderer in all stages. This underscores the importance of the rendering step throughout the process and that it cannot be omitted.

【Dear Reviewer】
After reviewing our rebuttal response, do you have any further questions or areas that require additional clarification? We place great importance on your opinions and hope to continue the discussion and address any remaining concerns you may have in the upcoming communication session.

Meta-review: The paper aims to develop an open-vocabulary 3D detector with the help of 2D data. The key idea is to lift 2D images to 3D point clouds by using metric monocular depth estimation models, combined with estimating the extrinsics and intrinsics. Finally, the paper trains a detector on the lifted 2D image and point-cloud data to get a base detector in "pre-training" stage. Then, the pseudo labels are generated on real 3D point clouds and projected to 2D RGB images, and trained further for a few epochs. The paper compared against existing baselines like OV-3DET and outperform them. The initial key concerns are the clarity in many components, more real-world 3D dataset experiments are needed, overall pipeline complexity. In the rebuttal, the authors give more details of the key components and show additional experiments on KITTI dataset. The final ratings are two borderline accept, one weak accept, and one accept. Hence, the AC recommend accept.
Decision: Accept (poster)